> ## Documentation Index
> Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Datasets & Training

> Pull versioned datasets to JSONL or Parquet, push to HuggingFace Hub, or load directly as a HuggingFace Dataset object.

The datasets API materializes a versioned DecimalAI dataset into a form your training stack can consume. Three patterns, in increasing convenience:

1. **`pull_dataset`** — writes a local file (JSONL or Parquet). Works with any tool.
2. **`push_to_hub`** — pushes to HuggingFace Hub. Instantly loadable by Axolotl, Unsloth, TRL, etc.
3. **`load_hf_dataset`** — returns a `datasets.Dataset` directly, no file needed.

See the [Datasets guide](/guides/datasets) for versioning and verdict-based filtering.

***

## `decimalai.pull_dataset()`

Download a versioned dataset to a local file.

```python theme={null}
# Pull latest version as JSONL
result = decimalai.pull_dataset("ds_abc123", "./training_data.jsonl")
print(f"Wrote {result['row_count']} rows to {result['file_path']}")

# Pull specific version as Parquet
result = decimalai.pull_dataset("ds_abc123", "./data.parquet", version="v2")
```

<ParamField path="dataset_id" type="str" required>
  The dataset ID.
</ParamField>

<ParamField path="path" type="str" required>
  Local file path. Format is inferred from extension (`.jsonl` or `.parquet`).
</ParamField>

<ParamField path="version" type="str" default="latest">
  Version specifier: `None`/`"latest"`, `"v3"`/`"3"`, or a full UUID.
</ParamField>

<ParamField path="format" type="str" default="auto">
  Override format: `"jsonl"` or `"parquet"`. Defaults to auto-detect from file extension.
</ParamField>

**Returns:** `{"row_count": 500, "file_path": "./data.jsonl", "bytes_written": 12345, "format": "jsonl"}`

***

## `decimalai.push_to_hub()`

Push a dataset to HuggingFace Hub. Makes the dataset instantly loadable by Axolotl, Unsloth, TRL, and any tool supporting `load_dataset()`.

```python theme={null}
result = decimalai.push_to_hub(
    "ds_abc123",
    "my-org/support-agent-sft",
    version="latest",
)
print(f"Pushed to {result['repo_url']}")

# Now usable in training:
# from datasets import load_dataset
# ds = load_dataset("my-org/support-agent-sft")
```

<ParamField path="dataset_id" type="str" required>
  The DecimalAI dataset ID.
</ParamField>

<ParamField path="repo_id" type="str" required>
  HuggingFace repo in `"org/dataset-name"` format.
</ParamField>

<ParamField path="version" type="str" default="latest">
  Version specifier: `None`/`"latest"`, `"v3"`/`"3"`, or UUID.
</ParamField>

<ParamField path="token" type="str" default="HF_TOKEN env">
  HuggingFace API token. Falls back to `HF_TOKEN` env var or cached login.
</ParamField>

<ParamField path="private" type="bool" default="True">
  Create a private repo.
</ParamField>

<ParamField path="split" type="str" default="train">
  Dataset split name.
</ParamField>

**Returns:** `{"repo_url": "...", "repo_id": "...", "row_count": 500, "version_id": "...", "split": "train"}`

<Note>
  Requires `pip install huggingface_hub datasets`. These are optional dependencies.
</Note>

***

## `decimalai.load_hf_dataset()`

Load a dataset directly as a HuggingFace `Dataset` object — no intermediate file needed.

```python theme={null}
ds = decimalai.load_hf_dataset("ds_abc123", version="v2")
# Dataset({features: ['messages'], num_rows: 500})

# Plug directly into TRL
from trl import SFTTrainer
trainer = SFTTrainer(model=model, train_dataset=ds, ...)
```

<ParamField path="dataset_id" type="str" required>
  The DecimalAI dataset ID.
</ParamField>

<ParamField path="version" type="str" default="latest">
  Version specifier.
</ParamField>

**Returns:** A `datasets.Dataset` object.

***

## What's next

<CardGroup cols={2}>
  <Card title="Datasets guide" icon="book" href="/guides/datasets">
    Verdict filtering, versioning, and how to build training-ready splits.
  </Card>

  <Card title="Training tutorial" icon="graduation-cap" href="/tutorials/training-pipeline">
    End-to-end SFT recipe using a DecimalAI dataset.
  </Card>
</CardGroup>