End-to-End Training Pipeline

This tutorial walks through DecimalAI’s end-to-end training workflow: trace → evaluate → build dataset → fine-tune. By the end, you’ll have a fine-tuned model trained on your agent’s best production outputs. This isn’t a one-shot pipeline — it’s a flywheel. Each fine-tuned model you deploy produces new traces, which feed the next round of evaluation and training:

Prerequisites

DecimalAI SDK installed (pip install "decimalai[evals]")
An API key (DECIMAL_API_KEY)
An agent producing traces (see Quickstart)
An OpenAI or Together.AI API key for fine-tuning

Instrument Your Agent

Make sure your agent is instrumented and sending traces:

import decimalai
decimalai.init(api_key="dai_sk_...", agent_name="support-agent")

from decimalai.langchain import install
install()

# Your agent runs as normal — traces are captured automatically

After running your agent on production traffic for a period, you’ll have traces in the dashboard.

You should now see: traces accumulating on the Traces page as your agent runs. If nothing appears, check startup logs — a framework flag whose package isn’t installed logs a warning and continues untraced (see the silent no-ops list at the bottom of this page).

Evaluate Traces

Attach evaluators to score output quality:

import decimalai
decimalai.init(api_key="dai_sk_...", agent_name="support-agent")

from decimalai.langchain import install
from decimalai.evals import Relevance, Toxicity, eval, TraceData

# Pre-built evaluators
evals = [Relevance(), Toxicity()]

# Custom evaluator
@eval(name="answered_question")
def check_answered(trace: TraceData) -> bool:
    return len(trace.output) > 50 and "?" not in trace.output[-20:]

# Register with auto-scoring
install(evals=[*evals, check_answered])

Traces are now scored automatically. Check the Evaluate page in the dashboard to see pass rates and trends.

You should now see: new traces arriving with eval scores attached — pass rates and per-evaluator verdicts on the Evaluate page. Scores apply to traces recorded after install(evals=...) ran; earlier traces stay unscored.

Review the Eval Dashboard

In the dashboard, navigate to Evaluate:

Pass Rate: What percentage of traces are passing your evaluators
Score Distribution: Histogram of scores across all traces
Evaluator Breakdown: Which evaluators catch the most failures

Use the verdict filter to isolate failing traces and understand what’s going wrong.

Build a Dataset

Navigate to Datasets in the sidebar, then click “Build Dataset”:

Select agent: Choose support-agent
Filter by manifest: Use the latest manifest version (ensures current agent config)
Filter by eval verdict: Select only pass verdicts
Choose format: SFT (supervised fine-tuning)
Click Build

DecimalAI converts multi-turn agent traces into the chat completion format:

{
  "messages": [
    {"role": "system", "content": "You are a support agent..."},
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": null, "tool_calls": [{"function": {"name": "search_docs", "arguments": "{\"query\": \"password reset\"}"}}]},
    {"role": "tool", "content": "{\"results\": [\"Go to Settings > Security...\"]}"},
    {"role": "assistant", "content": "To reset your password, go to Settings > Security..."}
  ]
}

Each multi-turn conversation becomes one training example. Tool calls and results are preserved so the fine-tuned model learns when and how to use tools.

You should now see: a new dataset version with a non-zero row count on its detail page. A 0-row build means your filters matched nothing — most often the eval-verdict filter with no scored traces yet (step 2), or a manifest filter pinned to a version your traffic predates.

Launch Fine-Tuning

From the dataset detail page, click “Train”:

Select provider: OpenAI, Together.AI, or Gemini
Enter credentials: API key for the training provider
Configure: Choose base model, training epochs
Launch

The platform submits the job and polls for completion. Training metrics (loss, validation) are stored for review.Supported Providers:

Provider	Models
OpenAI	GPT-4o, GPT-4.1-mini, GPT-4.1-nano
Together.AI	Llama 4, Llama 3.3/3.1, Qwen 3, DeepSeek R1/V3, Mistral
Gemini	Gemini 3.5 Flash, Gemini 2.5 Pro

Alternative: Pull Data for External Training

Prefer to train locally or with your own infrastructure? Pull the dataset via SDK or CLI:

import decimalai
decimalai.init()

# Pull to a local file
result = decimalai.pull_dataset("ds_abc123", "./training_data.jsonl", version="latest")
print(f"Wrote {result['row_count']} rows")

# CLI equivalent
decimalai datasets pull ds_abc123 -o ./training_data.jsonl

Push to HuggingFace Hub for use with Axolotl, Unsloth, or TRL:

# Push to HF Hub — instantly usable by open-source trainers
decimalai.push_to_hub("ds_abc123", "my-org/support-agent-sft")

# CLI equivalent
decimalai datasets push-to-hub ds_abc123 my-org/support-agent-sft

Now use the data in any training framework:

Axolotl
Unsloth
TRL
In-Memory (No File)

datasets:
  - path: my-org/support-agent-sft
    type: chat_template

from datasets import load_dataset
ds = load_dataset("my-org/support-agent-sft")
# Use with FastLanguageModel...

from datasets import load_dataset
from trl import SFTTrainer

ds = load_dataset("my-org/support-agent-sft")
trainer = SFTTrainer(model=model, train_dataset=ds, ...)

# Skip file entirely — load directly as HF Dataset
ds = decimalai.load_hf_dataset("ds_abc123")
# → Dataset({features: ['messages'], num_rows: 500})

Deploy and Iterate

Update your agent to use the fine-tuned model. DecimalAI will:

Detect the manifest change (model changed) and register a new version
Generate a compatibility report for existing traces
Continue evaluating new traces from the fine-tuned model
Build the next dataset from improved outputs

This creates a continuous improvement loop: better model → better traces → better training data → even better model.

If a step produced nothing

The pipeline degrades quietly rather than crashing — an uninstrumented agent, an unscored trace, or an empty dataset build all look like success until you check the counts. The usual causes: The SDK is deliberately fail-open: a misconfiguration degrades quietly instead of crashing your agent. These are the six places that bites, in the order people hit them.

1. A registry skill is never offered to your agent

Browsing or previewing a skill on the public registry doesn’t make it routable — you have to adopt it into your org first. Either way works — Use (router.use("name") or the Use button) creates a linked pointer that tracks the author’s updates, and Fork (router.fork("name"), or router.install("name") to also write it to disk, or the Fork button) creates an editable copy you own. The Skill Router offers both. If you use per-agent assignments, also assign it to the requesting agent. Until you Use or Fork it, the skill never appears in your agent’s menu, with no warning anywhere.

2. Only menu rows reach the model — never the skill body

By default build_prompt_fragment injects one-line menu rows (name + description). The skill’s actual instructions reach the model only if you pass inject_body=True to the SkillRouter (smart-routed queries), or enable the load_skill tool so the model can pull bodies on demand. Menu rows count as offered — usage panels show rung-labeled counts for them, and activation isn’t measurable for bare prompt-injection usage. If a skill “isn’t working,” check whether its body ever actually reached the model.

3. The load_skill tool never appears

The on-demand body loader is opt-in: pass enable_skill_loader=True to install() on the openai_agents or pydantic_ai adapters. On the anthropic adapter there is no tool loop — enable_skill_loader=True there injects the skill menu into system (the offered rung only); add init(inject_skill_body=True) if you want full bodies delivered, and note enable_load_skill_tool is accepted but dormant. Kill-switch: init(load_skill_tool=False) or DECIMALAI_LOAD_SKILL_TOOL=0.

4. An integration flag whose package is missing

decimalai.init(langchain=True) (and every other framework flag) logs a warning and continues untraced when the framework package isn’t importable — your agent runs, nothing is recorded. The same applies to DECIMAL_AUTO_TRACE (auto-init warns and skips) and to the raw-provider flags (openai=True etc.), which soft-skip when the matching OpenInference instrumentor is absent. Fix: for a framework flag, install the matching extra, e.g. pip install "decimalai[langchain]"; for a raw-provider flag, install the instrumentor package the startup warning names, e.g. pip install openinference-instrumentation-openai (the [openai] extra covers only the provider SDK, not the instrumentor). Then re-check startup logs.

5. Skill injection skips callable instructions and prebuilt prompts

Adapters only inject skills into prompt shapes they can safely rewrite. On openai_agents, an agent whose instructions is a user-supplied callable is left untouched; on langchain, a prebuilt PromptValue (or any unrecognized prompt shape) passes through unchanged. The agent runs normally — with zero skills injected and no error raised.

6. The impact report says 0 traces (or 'first run')

The regression check needs two things before it can say anything real: a baseline manifest (recorded automatically on the Action’s first run, or by decimalai.init() running in production) and ingested traces to measure blast radius against. A fresh workspace legitimately reports “first run — no baseline” and then near-zero affected traces. That’s honesty, not breakage — reports gain weight over days as production trace volume accumulates.

You’ve done it

Instrumented an agent and collected production traces

Scored traces with evaluators and a manifest compatibility check

Built a versioned SFT dataset filtered to keep + pass traces

Exported to HuggingFace and fine-tuned a model

Closed the loop — deployed the fine-tuned model, which produces traces for the next iteration

What Makes This Unique

Most platforms stop at evaluation. DecimalAI connects:

Manifest compatibility ensures training data matches your current agent config
Eval scoring ensures only high-quality outputs enter training data
Automatic format conversion handles the complex multi-turn, tool-using conversation structure
HuggingFace Hub integration means one-click compatibility with every open-source trainer
The loop repeats — each fine-tuned model feeds the next iteration

Next Steps

Datasets Guide

Filter strategies, version pinning, and export formats in depth.

Replay Guide

Regenerate training data by replaying historical inputs against the new model.

Evaluations

Configure quality gates so only high-signal traces enter datasets.

Manifests

How compatibility is computed when you change the agent.

Get Started

Foundation

Ship Safely

Improve with skills

Trust & Safety

Data & Training

Operate

Core Concepts

Platform

End-to-End Training Pipeline

Prerequisites

If a step produced nothing

You’ve done it

What Makes This Unique

Next Steps

Datasets Guide

Replay Guide

Evaluations

Manifests

​Prerequisites

​If a step produced nothing

​You’ve done it

​What Makes This Unique

​Next Steps

Datasets Guide

Replay Guide

Evaluations

Manifests

Prerequisites

If a step produced nothing

You’ve done it

What Makes This Unique

Next Steps