Datasets & Training

A dataset turns your agent’s traces into training rows. You build a version from the traces you choose, then download it or start a fine-tuning job from it. The fine-tuned model produces new traces, and those feed the next dataset.

Every plan can build and export. Hosted fine-tuning starts on Core, so on Free you export and train elsewhere. Pricing has the monthly row and launch limits.

What a row is made of

A row is assembled from a trace’s LLM calls: the messages sent to the model and what came back. A trace with no LLM call adds nothing, however many spans it recorded. A framework integration captures those calls for you. If you log by hand:

What you log	Where it lands
`log_llm_call(input=[...])`	The `system` and `user` messages, taken from the trace’s first call
`log_llm_call(output=...)`	An `assistant` message
`log_tool_call(...)`	A step on the timeline and a tool on the manifest, not a message
Model, system prompt, tool names	The agent’s manifest, registered automatically

A tool message appears only when the LLM call itself reports tool calls. Framework integrations record that, and a manual log_llm_call can’t.

Building Datasets

From the Dashboard

Open the form

On Datasets, click + New Dataset.

Name it and choose a type

Give it a Dataset Name and a Dataset Type: Fine-tuning (SFT), Evaluation, Preference or Evergreen. The type says what the dataset is for. A build writes the same chat rows for all four.

Choose the agent and manifest

Choose the Target Agent. Target Manifest then appears, set to the active version, with each version’s trace count. A bar shows how many traces are keep, repair, replay and drop. If it’s empty, Compute compatibility fills it in.

Build

Click Create & Build. Version 1 is built and the dataset opens. With no manifest selected the button reads Create & Configure, and you get an empty dataset to build later.

The manifest is the filter. It takes every trace recorded under that manifest, plus older traces whose verdict for it is keep or repair, and only the LLM calls made by that manifest’s agent. There is no quality filter in this dialog.

keep goes in as recorded. repair goes in with its repaired tool calls if a repair toward this manifest has run, and as recorded if not. replay and drop stay out.

Build another version

The dataset’s page has a Build new version form. Each build adds a numbered version.

Agent appears only when the dataset has no target agent.
Version is a manifest, or All versions, which builds from every version.
Min Eval Score keeps traces scoring at least this, from 0 to 1. A trace with no score is left out.

From selected traces

On an agent’s Traces page, tick the traces you want and click New Dataset in the bar that appears. It creates a dataset and builds it from exactly those traces.

From the API

import os

import httpx

base = os.environ.get("DECIMAL_BASE_URL", "https://api.decimal.ai")
headers = {"Authorization": f"Bearer {os.environ['DECIMAL_API_KEY']}"}

dataset = httpx.post(f"{base}/api/v1/datasets", headers=headers, json={
    "name": "support-agent SFT",
    "dataset_type": "sft",
    "training_target_agent": "support-agent",
}).json()

build = httpx.post(
    f"{base}/api/v1/datasets/{dataset['dataset_id']}/build",
    headers=headers,
    json={"min_eval_score": 0.7},
    timeout=60,
).json()
print(build["version_status"], build["row_count"])

Filtering

The build body takes:

Field	What it does
`trace_ids`	Exactly these traces. Wins over the next two
`manifest_id`	The same selection as Target Manifest. Wins over `agent_name`
`agent_name`	That agent’s LLM calls in every trace, including calls inside another agent’s trace
`min_eval_score`	Traces whose score is at least this, from 0 to 1
`allowed_verdicts`	A list. `keep`, `repair`, `replay` and `drop` match the compatibility verdict; `pass`, `fail` and `review` match the eval verdict. Any match counts
`example_type`	`full_trajectory` (default): one row per trace. `step`: one text-only row per LLM call

An empty body uses the dataset’s own manifest, then its target agent. With neither, it reads every LLM call in your organization and ignores the two quality filters.

Scores you push aren’t what the filters read. decimalai.score() records a score on the trace’s Evaluations tab, but min_eval_score and the eval verdicts read the trace’s overall eval_score and eval_verdict. Only a server-side evaluator sets those. To gate on your own scores, send the traces that pass as trace_ids.

good = [trace_id for trace_id, score in my_scores.items() if score >= 0.5]

httpx.post(f"{base}/api/v1/datasets/{dataset['dataset_id']}/build", headers=headers, json={"trace_ids": good}, timeout=60)

The CLI’s decimalai datasets build --verdict sends a key this route doesn’t read, so it builds unfiltered. Use allowed_verdicts instead.

If a build comes back empty

A build with no rows still returns HTTP 200, with "status": "failed" and "row_count": 0. It leaves a failed version as a record, and the current version stays where it was. Check these in order:

The traces have no LLM calls. Open one and look at LLM calls in the strip at the top. If it’s 0, the integration never captured a model call: see the silent no-ops.
The agent name doesn’t match. support-agent, Support Agent and [Demo] support-agent are three different agents.
A quality filter reads a score nothing wrote. See the warning above.
The target is a handoff target on the OpenAI Agents SDK. Those calls are filed under the root agent. See What Gets Captured.

SFT Format

SFT (supervised fine-tuning) teaches the model to produce the assistant turns. By default one trace becomes one row, a whole conversation in the chat format fine-tuning APIs expect:

{
  "messages": [
    {"role": "system", "content": "You are a support agent."},
    {"role": "user", "content": "How do I reset my password?"},
    {"role": "assistant", "content": null, "tool_calls": [
      {"id": "call_1", "type": "function", "function": {"name": "search_docs", "arguments": "{\"query\": \"password reset\"}"}}
    ]},
    {"role": "tool", "tool_call_id": "call_1", "content": "Go to Settings > Security."},
    {"role": "assistant", "content": "Go to Settings > Security and choose Reset password."}
  ],
  "tools": [
    {"type": "function", "function": {"name": "search_docs", "parameters": {"type": "object"}}}
  ]
}

system and user come from the trace’s first LLM call. Each call then adds an assistant turn.
tool takes its content from the matching tool span, or (no result captured) if there is none.
tools comes from the manifest, on versions built with one. A manifest that knows only a tool’s name gives an incomplete definition, so drop this key if your trainer validates it.

Each row is also assigned to train, val or test at 80/10/10, which Quality reports. The JSONL has no split field and a hosted job trains on every row, so make your own split if you need a held-out set. A row holding calls from more than one agent carries handoff markers.

DPO Format

DPO (direct preference optimization) trains on pairs: a preferred answer and a worse one for the same prompt. Pairs come from a completed replay batch, not from a build:

{
  "prompt": "How do I reset my password?",
  "chosen": "Go to Settings > Security and choose Reset password.",
  "rejected": "I'm not sure, maybe check the FAQ?",
  "metadata": {"reason": "replay_preferred", "eval_score": 0.9}
}

The replay task’s score decides the order. At 0.6 or above the replayed answer is chosen, at 0.4 or below the original is, and in between the pair is skipped. score_threshold (default 0.1) sets that gap. metadata also names both traces and the batch. Download them with Export DPO Pairs on a completed batch, or from GET /api/v1/replay/batches/{batch_id}/export-dpo.

Dataset Versioning

Pick a version on the dataset’s page, and its rows open under SFT rows preview. Each row expands to its messages, with View raw JSON. Beside the version:

Quality shows rows, messages per row, tool calls, and the eval score, split and verdict breakdowns.
↓ JSONL downloads it.
Fine-Tune opens the training panel.

The API adds:

Route, under `/api/v1/datasets/{id}/versions/{version_id}`	What it does
`GET /compare/{other_version_id}`	Counts of added, removed and unchanged rows, with samples
`POST /validate`	A check for CI. Takes `min_row_count` and `required_verdicts`, returns `gate_passed` and `issues`
`PATCH /quality`	Records a review: `pending_review`, `approved`, `rejected` or `needs_revision`. `rejected` archives the version and returns its rows to your monthly count

Export

Download from the dashboard, or with the SDK:

import decimalai

decimalai.init()

result = decimalai.pull_dataset("ds_abc123", "./training_data.jsonl")
print(f"Wrote {result['row_count']} rows to {result['file_path']}")

version= takes "latest" (the default: the newest build with rows), "v3" or a full UUID. The REST route is GET /api/v1/datasets/{id}/versions/{version_id}/export?format=jsonl. Commands are on the CLI page, and parameters on the SDK page.

Export JSONL. The SDK and CLI accept a .parquet path, but the hosted API answers format=parquet with a 501, so the call raises. Download JSONL and convert it.

HuggingFace Hub Integration

A pushed dataset works with Axolotl, Unsloth, TRL and anything else that reads load_dataset(). It needs pip install huggingface_hub datasets.

import decimalai

decimalai.init()

result = decimalai.push_to_hub("ds_abc123", "my-org/support-agent-sft")
print(f"Pushed to {result['repo_url']}")

The repo is private unless you pass private=False, and the token comes from token= or HF_TOKEN. Load it with load_dataset("my-org/support-agent-sft"), or point Axolotl at it:

datasets:
  - path: my-org/support-agent-sft
    type: chat_template

To skip the push and the file, decimalai.load_hf_dataset("ds_abc123") returns a datasets.Dataset in memory.

Fine-Tuning

A hosted job uploads a version’s rows to your provider, under your own account, and starts a fine-tune there.

Supported Providers

Add a key under Settings → Provider keys (BYOK): Configure, then Save Credentials. An organization owner or admin can do this.

Provider	Base models	What it needs
OpenAI	`gpt-4o-mini-2024-07-18`, `gpt-4o-2024-08-06`, `gpt-4.1-mini-2025-04-14`, `gpt-4.1-nano-2025-04-14`	API Key
Gemini (Vertex AI)	`gemini-3.5-flash`, `gemini-2.5-pro`	API Key, Project ID, GCS Bucket
Raw / Webhook	Any name you type	Nothing, or a Webhook URL to receive the rows. The job reads `succeeded` at once and no model comes back

Together.AI is the least tested launch route. You can now save a key and start a launch, but we haven’t run one through to a finished model. If it fails, export the JSONL and upload it to Together yourself.

Launching a Job

Open the panel

On the dataset’s page, select a version, click Fine-Tune, then + Launch Job.

Fill in the dialog

Choose the Provider, type the Base Model and set Epochs (default 3). The model field is free text and starts on an OpenAI model, so retype it when you switch provider.

Launch

Click Launch.

Each job moves through uploading and training to succeeded, failed or cancelled. The panel refreshes every 15 seconds while it’s open, and a slower background check covers the rest. A finished job shows the model’s ID with Copy, a failed one the provider’s error, and a running one Cancel, along with any metrics your provider reports. From the API, POST /api/v1/training/jobs takes dataset_id, dataset_version_id, provider, base_model and optional hyperparameters, such as {"n_epochs": 3}. GET /api/v1/training/jobs/{id} reads a job, and POST to /poll or /cancel on that path checks or stops it. A launch counts against your plan once it succeeds, and over the limit you get a 429. An organization can run five jobs at once.

Deploy and iterate

Point your agent at the fine-tuned model. The model is part of the manifest, so the next run registers a new manifest version. Compute compatibility on that version to see which older traces still fit, then build the next dataset against it. Replay regenerates the ones that don’t.

Next Steps

Evaluations

Set up the evaluators that quality filters read.

SDK reference

pull_dataset, push_to_hub and load_hf_dataset.

Start here

Tutorial: a support agent

Use skills

Manage and measure

Publish a skill

More

Datasets & Training

What a row is made of

Building Datasets

From the Dashboard

Build another version

From selected traces

From the API

Filtering

If a build comes back empty

SFT Format

DPO Format

Dataset Versioning

Export

HuggingFace Hub Integration

Fine-Tuning

Supported Providers

Launching a Job

Deploy and iterate

Next Steps

Evaluations

SDK reference

​What a row is made of

​Building Datasets

​From the Dashboard

​Build another version

​From selected traces

​From the API

​Filtering

​If a build comes back empty

​SFT Format

​DPO Format

​Dataset Versioning

​Export

​HuggingFace Hub Integration

​Fine-Tuning

​Supported Providers

​Launching a Job

​Deploy and iterate

​Next Steps

Evaluations

SDK reference

What a row is made of

Building Datasets

From the Dashboard

Build another version

From selected traces

From the API

Filtering

If a build comes back empty

SFT Format

DPO Format

Dataset Versioning

Export

HuggingFace Hub Integration

Fine-Tuning

Supported Providers

Launching a Job

Deploy and iterate

Next Steps