> ## Documentation Index
> Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# End-to-End Training Pipeline
> Tutorial: From production traces to a fine-tuned model in one workflow.
This tutorial walks through DecimalAI's end-to-end training workflow: **trace → evaluate → build dataset → fine-tune**. By the end, you'll have a fine-tuned model trained on your agent's best production outputs.
This isn't a one-shot pipeline — it's a flywheel. Each fine-tuned model you deploy produces new traces, which feed the next round of evaluation and training:
```mermaid theme={null}
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#f5f5f4','primaryBorderColor':'#a8a29e','primaryTextColor':'#44403c','lineColor':'#a8a29e'}}}%%
flowchart LR
A[Instrument] --> B[Evaluate]
B --> C[Build dataset]
C --> D[Fine-tune]
D --> E[Deploy]
E -->|new traces| B
```
## Prerequisites
* DecimalAI SDK installed (`pip install decimalai[evals]`)
* An API key (`DECIMAL_API_KEY`)
* An agent producing traces (see [Quickstart](/quickstart))
* An OpenAI or Together.AI API key for fine-tuning
***
Make sure your agent is instrumented and sending traces:
```python theme={null}
import decimalai
decimalai.init(api_key="dai_sk_...", agent_name="support-agent")
from decimalai.langchain import install
install()
# Your agent runs as normal — traces are captured automatically
```
After running your agent on production traffic for a period, you'll have traces in the dashboard.
Attach evaluators to score output quality:
```python theme={null}
import decimalai
decimalai.init(api_key="dai_sk_...", agent_name="support-agent")
from decimalai.langchain import install
from decimalai.evals import Relevance, Toxicity, eval, TraceData
# Pre-built evaluators
evals = [Relevance(), Toxicity()]
# Custom evaluator
@eval(name="answered_question")
def check_answered(trace: TraceData) -> bool:
return len(trace.output) > 50 and "?" not in trace.output[-20:]
# Register with auto-scoring
install(evals=[*evals, check_answered])
```
Traces are now scored automatically. Check the **Evaluate** page in the dashboard to see pass rates and trends.
In the dashboard, navigate to **Evaluate**:
* **Pass Rate**: What percentage of traces are passing your evaluators
* **Score Distribution**: Histogram of scores across all traces
* **Evaluator Breakdown**: Which evaluators catch the most failures
Use the verdict filter to isolate failing traces and understand what's going wrong.
Navigate to **Datasets** in the sidebar, then click **"Build Dataset"**:
1. **Select agent**: Choose `support-agent`
2. **Filter by manifest**: Use the latest manifest version (ensures current agent config)
3. **Filter by eval verdict**: Select only `pass` verdicts
4. **Choose format**: SFT (supervised fine-tuning)
5. **Click Build**
DecimalAI converts multi-turn agent traces into the chat completion format:
```json theme={null}
{
"messages": [
{"role": "system", "content": "You are a support agent..."},
{"role": "user", "content": "How do I reset my password?"},
{"role": "assistant", "content": null, "tool_calls": [{"function": {"name": "search_docs", "arguments": "{\"query\": \"password reset\"}"}}]},
{"role": "tool", "content": "{\"results\": [\"Go to Settings > Security...\"]}"},
{"role": "assistant", "content": "To reset your password, go to Settings > Security..."}
]
}
```
Each multi-turn conversation becomes one training example. Tool calls and results are preserved so the fine-tuned model learns when and how to use tools.
From the dataset detail page, click **"Train"**:
1. **Select provider**: OpenAI, Together.AI, or Gemini
2. **Enter credentials**: API key for the training provider
3. **Configure**: Choose base model, training epochs
4. **Launch**
The platform submits the job and polls for completion. Training metrics (loss, validation) are stored for review.
**Supported Providers:**
| Provider | Models |
| ----------- | ------------------------------------------------------- |
| OpenAI | GPT-4o, GPT-4.1-mini, GPT-4.1-nano |
| Together.AI | Llama 4, Llama 3.3/3.1, Qwen 3, DeepSeek R1/V3, Mistral |
| Gemini | Gemini 2.5 Flash, Gemini 2.5 Pro |
Prefer to train locally or with your own infrastructure? Pull the dataset via SDK or CLI:
```python theme={null}
import decimalai
decimalai.init()
# Pull to a local file
result = decimalai.pull_dataset("ds_abc123", "./training_data.jsonl", version="latest")
print(f"Wrote {result['row_count']} rows")
```
```bash theme={null}
# CLI equivalent
decimalai datasets pull ds_abc123 -o ./training_data.jsonl
```
**Push to HuggingFace Hub** for use with Axolotl, Unsloth, or TRL:
```python theme={null}
# Push to HF Hub — instantly usable by open-source trainers
decimalai.push_to_hub("ds_abc123", "my-org/support-agent-sft")
```
```bash theme={null}
# CLI equivalent
decimalai datasets push-to-hub ds_abc123 my-org/support-agent-sft
```
Now use the data in any training framework:
```yaml theme={null}
datasets:
- path: my-org/support-agent-sft
type: chat_template
```
```python theme={null}
from datasets import load_dataset
ds = load_dataset("my-org/support-agent-sft")
# Use with FastLanguageModel...
```
```python theme={null}
from datasets import load_dataset
from trl import SFTTrainer
ds = load_dataset("my-org/support-agent-sft")
trainer = SFTTrainer(model=model, train_dataset=ds, ...)
```
```python theme={null}
# Skip file entirely — load directly as HF Dataset
ds = decimalai.load_hf_dataset("ds_abc123")
# → Dataset({features: ['messages'], num_rows: 500})
```
Update your agent to use the fine-tuned model. DecimalAI will:
1. **Detect the manifest change** (model changed) and register a new version
2. **Generate a compatibility report** for existing traces
3. Continue evaluating new traces from the fine-tuned model
4. Build the next dataset from improved outputs
This creates a **continuous improvement loop**: better model → better traces → better training data → even better model.
***
## You've done it
Instrumented an agent and collected production traces
Scored traces with evaluators and a manifest compatibility check
Built a versioned SFT dataset filtered to keep + pass traces
Exported to HuggingFace and fine-tuned a model
Closed the loop — deployed the fine-tuned model, which produces traces for the next iteration
## What Makes This Unique
Most platforms stop at evaluation. DecimalAI connects:
* **Manifest compatibility** ensures training data matches your current agent config
* **Eval scoring** ensures only high-quality outputs enter training data
* **Automatic format conversion** handles the complex multi-turn, tool-using conversation structure
* **HuggingFace Hub integration** means one-click compatibility with every open-source trainer
* **The loop repeats** — each fine-tuned model feeds the next iteration
## Next Steps
Filter strategies, version pinning, and export formats in depth.
Regenerate training data by replaying historical inputs against the new model.
Configure quality gates so only high-signal traces enter datasets.
How compatibility is computed when you change the agent.