> ## Documentation Index
> Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Runs, checks and datasets

> CLI reference for what your agent produced and what a change does to it: traces, eval scores, evaluators, manifests, regression and compatibility checks, repair, replay and datasets.

Every command here also takes `--api-key` and `--base-url` ([Global options](https://docs.decimal.ai/sdk/cli.md#global-options)), so the tables leave them out.

> [!NOTE]
>
> List commands print shortened ids: the first 8 characters for traces and evaluators, the first 12 for manifests and datasets. A command that takes an id needs the whole id: it's in the page's URL in the dashboard, or for evaluators, in `GET /api/v1/evaluators`.

## Traces

[Read a run](https://docs.decimal.ai/guides/tracing.md) explains what a trace holds.

### `decimalai traces list`

List recent traces.

```bash theme={null}
decimalai traces list --limit 20 --status success
```

| Option | Default | Description |
| - | - | - |
| `--limit` | 20 | Max results to return, up to 100 |
| `--status` | all | Filter by `success`, `error` or `degraded` |

### `decimalai traces show`

Show full detail for a specific trace as JSON.

```bash theme={null}
decimalai traces show trc_abc123
```

### `decimalai traces stats`

Show trace statistics for your workspace as JSON.

```bash theme={null}
decimalai traces stats --agent-name my-agent
```

| Option | Default | Description |
| - | - | - |
| `--agent-name` | all | Filter by agent name |

### `decimalai traces import`

Import traces from a JSON or JSONL file.

```bash theme={null}
decimalai traces import traces.json
decimalai traces import traces.jsonl --agent-name my-agent
```

| Option | Default | Description |
| - | - | - |
| `--format` | auto | Import format: `auto` (by file extension), `json`, `jsonl` |
| `--agent-name` | none | Agent these traces belong to. Required for JSONL unless every row carries its own `agent_name`. The JSON format carries it inside each trace |

## Evaluations

### `decimalai eval push`

Push evaluation scores to a trace. To score from code, see [External score import](https://docs.decimal.ai/guides/evaluations.md#external-score-import).

```bash theme={null}
decimalai eval push trc_abc123 --score quality=0.9 --score relevance=0.85
```

| Option | Required | Description |
| - | - | - |
| `--score` | Yes (repeatable) | Score in `name=value` format |
| `--source` | No (default: `cli`) | Source label for the scores |

## Evaluators

An evaluator scores traces on DecimalAI: a pre-built template, or an LLM judge with your own rubric. [Score production traces](https://docs.decimal.ai/guides/evaluations.md#score-production-traces) explains how they run.

### `decimalai evaluators add`

Add an evaluator to one agent, or to the whole workspace.

```bash theme={null}
# From a template
decimalai evaluators add --template helpfulness --agent support-agent

# Your own LLM judge
decimalai evaluators add \
  --name tone_check \
  --type llm_judge \
  --rubric 'Rate the professionalism of: {output}' \
  --agent support-agent
```

| Option | Default | Description |
| - | - | - |
| `--agent` | workspace-wide | Attach to this agent |
| `--template` | none | A name from [`evaluators templates`](#decimalai-evaluators-templates) |
| `--name` | the template's | Evaluator name. Required without `--template`, and unique per agent |
| `--type` | the template's | `llm_judge` or `deterministic`. Required without `--template` |
| `--category` | `quality` | `quality`, `safety`, `rag`, `agentic` or `custom`. Sent with `--template` too, so it replaces the template's own category |
| `--rubric` | the template's | The judge's prompt, with `{input}` and `{output}` placeholders. Required for your own `llm_judge` |
| `--threshold` | the template's, else 0.5 | Pass threshold, 0.0 to 1.0 |
| `--display-name`, `--description` | the template's | Human-readable name, and a one-line description |

`--type deterministic` needs `--template`: the API can't run your code. A custom deterministic check is [a Python function in your agent](https://docs.decimal.ai/guides/evaluations.md#write-a-check).

### `decimalai evaluators list`

List configured evaluators: enabled (`●`) or disabled (`○`), type, agent and shortened id.

```bash theme={null}
decimalai evaluators list
decimalai evaluators list --agent support-agent
```

| Option | Default | Description |
| - | - | - |
| `--agent` | none | Also list this agent's evaluators. Without it, only workspace-wide evaluators are listed |

### `decimalai evaluators remove`

Remove an evaluator. It needs the full id and the admin role.

```bash theme={null}
decimalai evaluators remove 3f2a9c1e-7b44-4c1d-9a55-0e8d6f1b2c3d
```

### `decimalai evaluators templates`

List the pre-built templates, with each one's name, type, category and description. Pass the name to `evaluators add --template`.

```bash theme={null}
decimalai evaluators templates
```

## Manifests

A manifest is a snapshot of an agent's configuration. [Manifests](https://docs.decimal.ai/guides/manifests.md) explains the model.

### `decimalai manifests list`

List manifests, with each one's shortened id, agent and version label.

```bash theme={null}
decimalai manifests list --agent-name my-agent
```

| Option | Default | Description |
| - | - | - |
| `--agent-name` | all | Filter by agent name |
| `--limit` | 20 | Max results |

## Regression and compatibility checks

### `decimalai regression-check`

Compare a candidate manifest with the agent's current baseline, and report which recent production traces it would break. Use it locally, or in CI other than GitHub Actions: [Running outside CI](https://docs.decimal.ai/guides/regression-check.md#running-outside-ci).

```bash theme={null}
decimalai regression-check --agent-name support-agent
decimalai regression-check --agent-name support-agent --candidate-manifest-id mfst_xyz --fail-on medium
decimalai regression-check --agent-name support-agent --dry-run
```

| Option | Default | Description |
| - | - | - |
| `--agent-name` | (required) | The agent's name, as your code names it |
| `--candidate-manifest-id` | read from a file | The manifest to check. Without it, read from `$GITHUB_OUTPUT`, then `./decimal_manifest_id.txt`, which `decimalai.flush_manifest_for_ci()` writes |
| `--fail-on` | `high` | When to exit 1: `high`, `medium` (medium or high), or `none` |
| `--format` | `terminal` | `terminal`, `json` or `github` (annotations for a pull request) |
| `--trace-window-days` | 30 | How far back to look for affected traces |
| `--dry-run` | off | Compute the report without saving it or using your plan's quota |

| Exit | Meaning |
| - | - |
| 0 | The verdict is below `--fail-on`, or `--fail-on none` |
| 1 | The verdict meets `--fail-on` |
| 2 | The request failed, or no candidate manifest id was found |

An `unverified` verdict, a change with no traces to measure it against, is held to `--fail-on` at the severity of the manifest change. A `first_run` verdict, with no baseline yet, exits 0.

### `decimalai compat-check`

Compare an agent's latest two manifests and count its traces by what to do with them for training: keep, repair, replay or drop ([what each means](https://docs.decimal.ai/guides/manifests.md#compatibility-scoring)).

```bash theme={null}
decimalai compat-check --agent-name my-agent
decimalai compat-check --agent-name my-agent --format json
```

| Option | Default | Description |
| - | - | - |
| `--agent-name` | (required) | The agent to check |
| `--format` | `table` | `table`, `json` (includes `old_manifest_id` and `new_manifest_id`) or `github` (annotations for a pull request) |
| `--recompute` | off | Run a fresh analysis instead of returning a cached report |

It's informational, so it won't block CI: it exits 0 whatever it finds, and also when the agent has fewer than two manifests. It exits 1 only when the request fails.

## Repair

Repair fixes the traces marked **repair** without re-running your agent. [Repair](https://docs.decimal.ai/guides/manifests.md#repair-fix-traces-mechanically) explains the rules. Both commands take the two manifests' full ids, which `compat-check --format json` prints.

### `decimalai repair preview`

Show the repair rules for moving traces from one manifest to another, each with its index, and how many traces are eligible.

```bash theme={null}
decimalai repair preview --old-manifest-id "$OLD_ID" --new-manifest-id "$NEW_ID"
```

| Option | Default | Description |
| - | - | - |
| `--old-manifest-id` | (required) | The manifest the traces were captured against |
| `--new-manifest-id` | (required) | The manifest to migrate the traces to |
| `--sample-size` | 5 | How many traces to preview, 1 to 50. The samples appear in `--format json` |
| `--format` | `table` | `table` or `json` |

### `decimalai repair apply`

Apply all eligible rules, or only the ones you name. It prints the repair batch's id and status, and how many traces it repaired.

```bash theme={null}
decimalai repair apply --old-manifest-id "$OLD_ID" --new-manifest-id "$NEW_ID"
decimalai repair apply --old-manifest-id "$OLD_ID" --new-manifest-id "$NEW_ID" --rule-index 0 --rule-index 2
```

| Option | Default | Description |
| - | - | - |
| `--old-manifest-id`, `--new-manifest-id` | (required) | As for `repair preview` |
| `--rule-index` | all rules | Apply only this rule, by its index in `repair preview`. Repeatable |
| `--format` | `table` | `table` or `json` |

## Replay

### `decimalai replay run`

Execute a replay batch: re-run its pending tasks through your updated agent and submit the results. [Replay](https://docs.decimal.ai/guides/replay.md) explains the workflow.

```bash theme={null}
decimalai replay run batch_abc123 --agent-fn my_app.agent:run
```

| Argument / Option | Required | Description |
| - | - | - |
| `BATCH_ID` | Yes | The replay batch to execute |
| `--agent-fn` | Yes | Python path to your agent function (`module:function`) |
| `--dry-run` | No | Show how many tasks would run, without running them |
| `--no-skip-failed` | No | Stop on first failure instead of continuing |

Output:

```
✓ Loaded agent function: my_app.agent:run
  ✓ [1/50] abc12345: keep
  ✗ [3/50] ghi11111: agent error: connection timed out
  ...
========================================
Replay Summary
========================================
  Total:      50
  Completed:  45
  Passed:     42
  Failed:     3
  Skipped:    2
  Pass rate:  84%
```

The CLI exits with code 1 if any replays fail, so you can use it in CI. A task whose original trace has no input is skipped.

## Datasets

[Fine-tuning and datasets](https://docs.decimal.ai/guides/datasets.md) explains the formats and how to build one from the dashboard.

### `decimalai datasets list`

List the datasets in the workspace with row counts and version info.

```bash theme={null}
decimalai datasets list
```

| Option | Default | Description |
| - | - | - |
| `--limit` | 20 | Max results |

`--help` also lists `--agent-name`. The command accepts it and doesn't apply it.

### `decimalai datasets show`

Show version history for a specific dataset.

```bash theme={null}
decimalai datasets show ds_abc123
```

### `decimalai datasets pull`

Pull a dataset version to a local file.

```bash theme={null}
decimalai datasets pull ds_abc123 -o ./training_data.jsonl
decimalai datasets pull ds_abc123 -o ./data.jsonl --version v2
decimalai datasets pull ds_abc123 -o ./data.parquet
```

| Option | Default | Description |
| - | - | - |
| `--output / -o` | (required) | Output file path. Format inferred from extension. |
| `--version / -v` | `latest` | Version: `latest`, `v3`, `3`, or UUID |
| `--format` | auto | Override: `jsonl` or `parquet` |

### `decimalai datasets export`

Export a dataset version to stdout (for piping) or a file.

```bash theme={null}
decimalai datasets export ds_abc123 | head -5
decimalai datasets export ds_abc123 --version v2 -o training.jsonl
```

| Option | Default | Description |
| - | - | - |
| `--version / -v` | `latest` | Version specifier |
| `--format` | `jsonl` | Format: `jsonl` or `parquet`. Parquet needs `-o` |
| `--output / -o` | stdout | Output file path |

### `decimalai datasets build`

Build a new dataset version from traces. It prints the new version's id and row count.

```bash theme={null}
decimalai datasets build ds_abc123
```

`--help` also lists `--verdict`, but the server doesn't apply it, so the build isn't filtered. To filter by verdict, use [`allowed_verdicts`](https://docs.decimal.ai/guides/datasets#from-the-api) through the API.

### `decimalai datasets push-to-hub`

Push a dataset to HuggingFace Hub, so any tool that supports `load_dataset()` can load it. [HuggingFace Hub Integration](https://docs.decimal.ai/guides/datasets.md#huggingface-hub-integration) shows how to train on it.

```bash theme={null}
decimalai datasets push-to-hub ds_abc123 my-org/support-agent-sft
decimalai datasets push-to-hub ds_abc123 my-org/my-dataset --version v2 --public
```

| Argument / Option | Default | Description |
| - | - | - |
| `DATASET_ID`, `REPO_ID` | (required) | The dataset ID, and the HuggingFace repo: `org/dataset-name` |
| `--version / -v` | `latest` | Version specifier |
| `--token` | `$HF_TOKEN` | HuggingFace API token |
| `--public / --private` | `--private` | Repo visibility |
| `--split` | `train` | Dataset split name |

> [!NOTE]
>
> Requires `pip install huggingface_hub datasets`.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.