> ## Documentation Index > Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt > Use this file to discover all available pages before exploring further. # Multi-Agent Systems > Instrument multi-agent systems — per-sub-agent datasets, orchestration-aware training data, and independent version tracking. DecimalAI supports multi-agent architectures out of the box. When your orchestrator delegates to sub-agents, DecimalAI automatically discovers the agent hierarchy, tracks each sub-agent's version independently, and lets you build per-sub-agent training datasets. ## How It Works ```mermaid theme={null} %%{init: {'theme':'base','themeVariables':{'primaryColor':'#f5f5f4','primaryBorderColor':'#a8a29e','primaryTextColor':'#44403c','lineColor':'#a8a29e'}}}%% flowchart LR subgraph sys [Your Multi-Agent System] direction TB O[travel-planner
orchestrator] O --> FA[flight_agent] O --> HA[hotel_agent] O --> IA[itinerary_agent] end subgraph dai [DecimalAI Platform] direction TB S[Agents sidebar
auto-discovered hierarchy] S --> M[Independent manifest
per sub-agent] S --> V[Own version timeline] S --> P[Per-agent datasets] end O -->|traces| S ``` The flowchart above shows the *topology* — who delegates to whom. The sequence diagram below shows the *order* in which the orchestrator hands off to each sub-agent at runtime, and where each handoff span lands in the trace: ```mermaid theme={null} %%{init: {'theme':'base','themeVariables':{'primaryColor':'#f5f5f4','primaryBorderColor':'#a8a29e','primaryTextColor':'#44403c','lineColor':'#a8a29e'}}}%% sequenceDiagram participant U as User participant O as travel-planner
(orchestrator) participant F as flight_agent participant H as hotel_agent participant I as itinerary_agent U->>O: Plan a trip to Tokyo O->>F: handoff:travel-planner->flight_agent F-->>O: Found 3 flights O->>H: handoff:travel-planner->hotel_agent H-->>O: Found hotels in Tokyo O->>I: handoff:travel-planner->itinerary_agent I-->>O: Complete itinerary O-->>U: Here's your trip ``` *** ## Setup The SDK auto-detects sub-agents from handoffs — no extra configuration needed: ```python theme={null} from agents import Agent, Runner # Define sub-agents flight_agent = Agent( name="flight_agent", instructions="You search for and book flights.", tools=[search_flights, book_flight], model="gpt-4o", ) hotel_agent = Agent( name="hotel_agent", instructions="You search for and book hotels.", tools=[search_hotels, book_hotel], model="gpt-4o", ) itinerary_agent = Agent( name="itinerary_agent", instructions="You combine flights and hotels into a complete itinerary.", tools=[format_itinerary], model="gpt-4o", ) # Orchestrator with handoffs to sub-agents orchestrator = Agent( name="travel-planner", instructions="You coordinate travel planning by delegating to specialists.", handoffs=[flight_agent, hotel_agent, itinerary_agent], model="gpt-4o", ) # Install DecimalAI — handoffs are auto-detected as sub-agents import decimalai decimalai.init(api_key="dai_sk_...") from decimalai.openai_agents import install install(agent=orchestrator) ``` When you pass the orchestrator `Agent` to `install()`, the SDK introspects `agent.handoffs` and registers each handoff target as a `subagent` component in the manifest. No manual registration needed. For custom multi-agent frameworks, register sub-agents manually: ```python theme={null} import decimalai decimalai.init(api_key="dai_sk_...") decimalai.register_manifest( agent_name="travel-planner", tools=[ {"name": "delegate", "schema": {"type": "object", "properties": {"agent": {"type": "string"}}}}, ], prompts={"system": "You coordinate travel planning."}, models={"default": {"provider": "openai", "model": "gpt-4o"}}, subagents=[ {"name": "flight_agent"}, {"name": "hotel_agent"}, {"name": "itinerary_agent"}, ], ) ``` *** ## What Gets Captured When the orchestrator runs and delegates to sub-agents, the SDK captures the full trace with per-call agent attribution: | Field | What It Contains | Set By | | ----------------------------- | ------------------------------- | -------------------------- | | `agent_name` on each LLM call | Which sub-agent made the call | Auto from agent spans | | `handoff` spans | Delegation events (from → to) | Auto from handoff spans | | `subagents` in manifest | Discovered sub-agent references | Auto from `agent.handoffs` | A single trace contains LLM calls from all sub-agents, each tagged with `agent_name`: ```json theme={null} { "trace_id": "abc-123", "agent_name": "travel-planner", "llm_calls": [ {"agent_name": "travel-planner", "model": "gpt-4o", "output": "I'll search flights first..."}, {"agent_name": "flight_agent", "model": "gpt-4o", "tool_calls": [{"name": "search_flights", "args": {"from": "SFO", "to": "NRT"}}]}, {"agent_name": "flight_agent", "model": "gpt-4o", "output": "Found 3 flights..."}, {"agent_name": "hotel_agent", "model": "gpt-4o", "tool_calls": [{"name": "search_hotels", "args": {"city": "Tokyo"}}]}, {"agent_name": "itinerary_agent", "model": "gpt-4o", "output": "Here's your complete itinerary..."} ], "spans": [ {"type": "agent", "name": "travel-planner"}, {"type": "handoff", "name": "handoff:travel-planner->flight_agent"}, {"type": "agent", "name": "flight_agent"}, {"type": "handoff", "name": "handoff:travel-planner->hotel_agent"}, {"type": "agent", "name": "hotel_agent"}, {"type": "handoff", "name": "handoff:travel-planner->itinerary_agent"}, {"type": "agent", "name": "itinerary_agent"} ] } ``` *** ## Dashboard ### Agent Sidebar The agents page shows the hierarchy — orchestrators are listed with their sub-agents nested below, marked with a → icon: * **travel-planner** — 150 traces, manifest v3 * → **flight\_agent** — discovered from handoffs * → **hotel\_agent** — discovered from handoffs * → **itinerary\_agent** — discovered from handoffs Click any sub-agent to see its traces, manifest timeline, and build per-agent datasets. ### Topology Graph The agent detail page shows a **topology graph** — a visual map of which agents delegate to which, with handoff edges and call counts. Multi-agent trace view with orchestrator and nested sub-agent calls *** ## Per-Sub-Agent Datasets The key benefit: **build training datasets for each sub-agent independently.** When you select `flight_agent` and build a dataset, only `flight_agent`'s LLM calls are extracted — even though those calls live inside the orchestrator's trace: ```python theme={null} # Via the API: build a dataset targeting just the flight agent curl -X POST https://api.decimal.ai/api/v1/datasets \ -H "Authorization: Bearer $API_KEY" \ -d '{ "name": "flight-agent-sft", "training_target_agent": "flight_agent", "dataset_type": "sft" }' ``` ### Orchestration-Aware SFT Export The platform produces two SFT export formats for multi-agent traces: Standard SFT format with `[HANDOFF]` markers between agent turns and `name` fields on assistant messages: ```json theme={null} {"messages": [ {"role": "system", "content": "You coordinate travel planning."}, {"role": "user", "content": "Plan a trip to Tokyo"}, {"role": "assistant", "content": null, "tool_calls": [...], "name": "travel-planner"}, {"role": "tool", "tool_call_id": "tc_1", "content": "Delegating to flight_agent"}, {"role": "system", "content": "[HANDOFF: flight_agent]"}, {"role": "assistant", "content": null, "tool_calls": [...], "name": "flight_agent"}, {"role": "tool", "tool_call_id": "tc_2", "content": "Found 3 flights..."}, {"role": "system", "content": "[HANDOFF: hotel_agent]"}, {"role": "assistant", "content": "Here is your complete itinerary...", "name": "itinerary_agent"} ]} ``` Best for: Training individual sub-agents with full conversation context. Orchestrator-focused format with explicit `[DELEGATION]` markers that include the instruction given to each sub-agent: ```json theme={null} {"messages": [ {"role": "system", "content": "You coordinate travel planning."}, {"role": "user", "content": "Plan a trip to Tokyo"}, {"role": "system", "content": "[DELEGATION to flight_agent]: Search for flights from SFO to NRT"}, {"role": "assistant", "content": null, "tool_calls": [...], "name": "flight_agent"}, {"role": "system", "content": "[DELEGATION to hotel_agent]: Find hotels in Tokyo"}, {"role": "assistant", "content": null, "tool_calls": [...], "name": "hotel_agent"}, {"role": "assistant", "content": "Here is your complete itinerary...", "name": "itinerary_agent"} ]} ``` Best for: Training the orchestrator to make better delegation decisions. *** ## Version Drift Detection Each sub-agent has its own version timeline. When a sub-agent changes, **only its own data is affected** — other sub-agents' data remains untouched. | What Changed | Impact Scope | Severity | | -------------------------------------------- | ------------------------------- | ------------ | | `flight_agent` minor version bump | Only `flight_agent` data | REPAIRABLE | | `flight_agent` major version (tools changed) | Only `flight_agent` data | INCOMPATIBLE | | `hotel_agent` updated | No impact on `flight_agent` | None | | Orchestrator prompt changed | All traces through orchestrator | Per-policy | When a change touches a sub-agent's traces, the **Impact Report** assigns each affected trace one of four compatibility verdicts: **keep** (reuse the trace as-is for training), **repair** (patch it to match the new manifest), **replay** (re-run it against the current agent to regenerate outputs), or **drop** (exclude it — too divergent to reuse). Impact severity (HIGH / MEDIUM / LOW) answers "was this trace structurally touched?"; the verdict answers "what to do with it for training." ### Cross-Sub-Agent Dependencies If one sub-agent's output feeds into another (e.g., `itinerary_agent` depends on `flight_agent`'s output), DecimalAI can detect when an upstream change breaks a downstream consumer: ``` [WARNING] flight_agent output contract changed └── itinerary_agent depends_on: [flight_agent] └── Flagged as INCOMPATIBLE — downstream data may be affected ``` Dependencies are declared in the sub-agent's schema via `register_manifest()`: ```python theme={null} decimalai.register_manifest( agent_name="itinerary_agent", tools=[{"name": "format_itinerary", "schema": {...}}], subagents=[ {"name": "flight_agent", "depends_on": True}, {"name": "hotel_agent", "depends_on": True}, ], ) ``` When `flight_agent`'s output contract changes, `itinerary_agent` is automatically flagged. *** ## Supported Frameworks | Framework | Auto-Detection | How Sub-Agents Are Found | | ----------------- | -------------- | ---------------------------------------------------------------------------- | | **OpenAI Agents** | ✅ Full | `agent.handoffs` introspected at `install()` time + handoff spans at runtime | | **LangChain** | Partial | Sub-agent names captured from `agent_name` field on LLM calls if set | | **OTel / CrewAI** | Partial | From span attributes if framework emits agent identity | | **Explicit** | ✅ Full | `register_manifest(subagents=[...])` | For the best experience with multi-agent systems, use the OpenAI Agents SDK with `install(agent=orchestrator)`. This gives full handoff introspection, tool schema extraction, and automatic sub-agent discovery. *** ## Best Practices 1. **Name your agents clearly.** Use descriptive names like `flight_agent`, not `agent_1`. These names appear in the dashboard, datasets, and SFT export. 2. **Pass the orchestrator to `install()`.** This gives the SDK the full agent graph upfront, enabling immediate manifest registration with all sub-agents. 3. **Build datasets per sub-agent.** Don't mix all sub-agents into one dataset — each agent has different skills and should be fine-tuned independently. 4. **Start with the flat SFT format.** The `[HANDOFF]` marker format works with standard OpenAI fine-tuning. Use the orchestration format only when specifically training delegation behavior. 5. **Declare dependencies.** If sub-agents consume each other's output, use `depends_on` in `register_manifest()` so DecimalAI can alert you to cascade breakage. ## Next Steps Orchestrators, sub-agents, delegation vs handoff, drift detection. REST reference for listing traces by agent (used for sub-agent activity). How sub-agent components appear in the manifest diff. Building per-sub-agent training datasets.