> ## Documentation Index
> Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets API

> Build versioned SFT/DPO training datasets from manifest-classified, eval-scored traces.

A dataset is a curated collection of training examples built from filtered production traces. The key innovation: combining manifest compatibility + eval scores means your training data is both **current** (recorded against the active agent config) and **high-quality** (passed evaluation). Each dataset version locks the manifest and filter set used to build it, so builds are reproducible.

## Lifecycle

```mermaid theme={null}
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#f5f5f4','primaryBorderColor':'#a8a29e','primaryTextColor':'#44403c','lineColor':'#a8a29e'}}}%%
flowchart LR
    A[create dataset] --> B[build version N]
    B --> C[classify traces<br/>keep / repair / replay / drop]
    B --> D[ready]
    D --> E[quality report]
    D --> F[export<br/>JSONL / Parquet]
    F --> G[push to HF Hub<br/>pull as Dataset]
    A -.->|manifest change<br/>or new traces| H[build version N+1]
    H --> D
```

## Common patterns

<CardGroup cols={2}>
  <Card title="Build from keep + pass traces" icon="filter" href="/api-reference/common-endpoints">
    `POST /datasets/{id}/build` with `allowed_verdicts=["keep"]` and `min_eval_score=0.7` is the canonical SFT recipe.
  </Card>

  <Card title="Export to JSONL or Parquet" icon="file-export" href="/api-reference/common-endpoints">
    `GET /datasets/{id}/versions/{v}/export?format=jsonl` returns the rows ready for OpenAI fine-tuning or HuggingFace.
  </Card>

  <Card title="Pull as HuggingFace Dataset" icon="cloud-arrow-down">
    `decimalai.load_hf_dataset(...)` returns a materialized `datasets.Dataset` object. Compatible with every open-source trainer. (Use `decimalai.pull_dataset(dataset_id, path)` instead to download to a local JSONL/Parquet file.)
  </Card>

  <Card title="Compare versions" icon="code-branch" href="/guides/datasets">
    Versions are immutable. To see what's changed between v1 and v2, fetch both and diff the row counts + source breakdowns.
  </Card>
</CardGroup>

## Formats

| Format  | Full Name                      | Use Case                                                                                                             |
| ------- | ------------------------------ | -------------------------------------------------------------------------------------------------------------------- |
| **SFT** | Supervised Fine-Tuning         | One input→output row per LLM call. Imitation learning. Most common.                                                  |
| **DPO** | Direct Preference Optimization | One row per (input, chosen, rejected) triple. Replay-driven — when v2 outperforms v1 on the same input, v2 = chosen. |

## Quick start

```python theme={null}
import httpx

# Build a new SFT version
resp = httpx.post(
    "https://api.decimal.ai/api/v1/datasets/ds_abc123/build",
    headers={"Authorization": "Bearer dai_sk_..."},
    json={"allowed_verdicts": ["keep"], "min_eval_score": 0.7},
)
version_id = resp.json()["version_id"]

# Load as a HuggingFace Dataset (in memory — plugs straight into TRL/Axolotl/Unsloth)
import decimalai
decimalai.init()  # reads DECIMAL_API_KEY; or init(api_key="dai_sk_...")
ds = decimalai.load_hf_dataset("ds_abc123", version=version_id)
print(f"{len(ds)} rows ready for training")

# Or download to a local file instead — returns a summary dict, not a Dataset
result = decimalai.pull_dataset("ds_abc123", "./train.jsonl", version=version_id)
print(f"Wrote {result['row_count']} rows to {result['file_path']}")
```

## Related

* [Datasets Guide](/guides/datasets) — filter strategies and recipes
* [Training Pipeline Tutorial](/tutorials/training-pipeline) — end-to-end: trace → eval → fine-tune
* [Skills & Data Pipeline](/concepts/skills-and-data) — SFT vs DPO, repair, replay