Runs, checks and datasets

Every command here also takes --api-key and --base-url (Global options), so the tables leave them out.

List commands print shortened ids: the first 8 characters for traces and evaluators, the first 12 for manifests and datasets. A command that takes an id needs the whole id: it’s in the page’s URL in the dashboard, or for evaluators, in GET /api/v1/evaluators.

Traces

Read a run explains what a trace holds.

`decimalai traces list`

List recent traces.

decimalai traces list --limit 20 --status success

Option	Default	Description
`--limit`	20	Max results to return, up to 100
`--status`	all	Filter by `success`, `error` or `degraded`

`decimalai traces show`

Show full detail for a specific trace as JSON.

decimalai traces show trc_abc123

`decimalai traces stats`

Show trace statistics for your workspace as JSON.

decimalai traces stats --agent-name my-agent

Option	Default	Description
`--agent-name`	all	Filter by agent name

`decimalai traces import`

Import traces from a JSON or JSONL file.

decimalai traces import traces.json
decimalai traces import traces.jsonl --agent-name my-agent

Option	Default	Description
`--format`	auto	Import format: `auto` (by file extension), `json`, `jsonl`
`--agent-name`	none	Agent these traces belong to. Required for JSONL unless every row carries its own `agent_name`. The JSON format carries it inside each trace

Evaluations

`decimalai eval push`

Push evaluation scores to a trace. To score from code, see External score import.

decimalai eval push trc_abc123 --score quality=0.9 --score relevance=0.85

Option	Required	Description
`--score`	Yes (repeatable)	Score in `name=value` format
`--source`	No (default: `cli`)	Source label for the scores

Evaluators

An evaluator scores traces on DecimalAI: a pre-built template, or an LLM judge with your own rubric. Score production traces explains how they run.

`decimalai evaluators add`

Add an evaluator to one agent, or to the whole workspace.

# From a template
decimalai evaluators add --template helpfulness --agent support-agent

# Your own LLM judge
decimalai evaluators add \
  --name tone_check \
  --type llm_judge \
  --rubric 'Rate the professionalism of: {output}' \
  --agent support-agent

Option	Default	Description
`--agent`	workspace-wide	Attach to this agent
`--template`	none	A name from `evaluators templates`
`--name`	the template’s	Evaluator name. Required without `--template`, and unique per agent
`--type`	the template’s	`llm_judge` or `deterministic`. Required without `--template`
`--category`	`quality`	`quality`, `safety`, `rag`, `agentic` or `custom`. Sent with `--template` too, so it replaces the template’s own category
`--rubric`	the template’s	The judge’s prompt, with `{input}` and `{output}` placeholders. Required for your own `llm_judge`
`--threshold`	the template’s, else 0.5	Pass threshold, 0.0 to 1.0
`--display-name`, `--description`	the template’s	Human-readable name, and a one-line description

--type deterministic needs --template: the API can’t run your code. A custom deterministic check is a Python function in your agent.

`decimalai evaluators list`

List configured evaluators: enabled (●) or disabled (○), type, agent and shortened id.

decimalai evaluators list
decimalai evaluators list --agent support-agent

Option	Default	Description
`--agent`	none	Also list this agent’s evaluators. Without it, only workspace-wide evaluators are listed

`decimalai evaluators remove`

Remove an evaluator. It needs the full id and the admin role.

decimalai evaluators remove 3f2a9c1e-7b44-4c1d-9a55-0e8d6f1b2c3d

`decimalai evaluators templates`

List the pre-built templates, with each one’s name, type, category and description. Pass the name to evaluators add --template.

decimalai evaluators templates

Manifests

A manifest is a snapshot of an agent’s configuration. Manifests explains the model.

`decimalai manifests list`

List manifests, with each one’s shortened id, agent and version label.

decimalai manifests list --agent-name my-agent

Option	Default	Description
`--agent-name`	all	Filter by agent name
`--limit`	20	Max results

Regression and compatibility checks

`decimalai regression-check`

Compare a candidate manifest with the agent’s current baseline, and report which recent production traces it would break. Use it locally, or in CI other than GitHub Actions: Running outside CI.

decimalai regression-check --agent-name support-agent
decimalai regression-check --agent-name support-agent --candidate-manifest-id mfst_xyz --fail-on medium
decimalai regression-check --agent-name support-agent --dry-run

Option	Default	Description
`--agent-name`	(required)	The agent’s name, as your code names it
`--candidate-manifest-id`	read from a file	The manifest to check. Without it, read from `$GITHUB_OUTPUT`, then `./decimal_manifest_id.txt`, which `decimalai.flush_manifest_for_ci()` writes
`--fail-on`	`high`	When to exit 1: `high`, `medium` (medium or high), or `none`
`--format`	`terminal`	`terminal`, `json` or `github` (annotations for a pull request)
`--trace-window-days`	30	How far back to look for affected traces
`--dry-run`	off	Compute the report without saving it or using your plan’s quota

Exit	Meaning
0	The verdict is below `--fail-on`, or `--fail-on none`
1	The verdict meets `--fail-on`
2	The request failed, or no candidate manifest id was found

An unverified verdict, a change with no traces to measure it against, is held to --fail-on at the severity of the manifest change. A first_run verdict, with no baseline yet, exits 0.

`decimalai compat-check`

Compare an agent’s latest two manifests and count its traces by what to do with them for training: keep, repair, replay or drop (what each means).

decimalai compat-check --agent-name my-agent
decimalai compat-check --agent-name my-agent --format json

Option	Default	Description
`--agent-name`	(required)	The agent to check
`--format`	`table`	`table`, `json` (includes `old_manifest_id` and `new_manifest_id`) or `github` (annotations for a pull request)
`--recompute`	off	Run a fresh analysis instead of returning a cached report

It’s informational, so it won’t block CI: it exits 0 whatever it finds, and also when the agent has fewer than two manifests. It exits 1 only when the request fails.

Repair

Repair fixes the traces marked repair without re-running your agent. Repair explains the rules. Both commands take the two manifests’ full ids, which compat-check --format json prints.

`decimalai repair preview`

Show the repair rules for moving traces from one manifest to another, each with its index, and how many traces are eligible.

decimalai repair preview --old-manifest-id "$OLD_ID" --new-manifest-id "$NEW_ID"

Option	Default	Description
`--old-manifest-id`	(required)	The manifest the traces were captured against
`--new-manifest-id`	(required)	The manifest to migrate the traces to
`--sample-size`	5	How many traces to preview, 1 to 50. The samples appear in `--format json`
`--format`	`table`	`table` or `json`

`decimalai repair apply`

Apply all eligible rules, or only the ones you name. It prints the repair batch’s id and status, and how many traces it repaired.

decimalai repair apply --old-manifest-id "$OLD_ID" --new-manifest-id "$NEW_ID"
decimalai repair apply --old-manifest-id "$OLD_ID" --new-manifest-id "$NEW_ID" --rule-index 0 --rule-index 2

Option	Default	Description
`--old-manifest-id`, `--new-manifest-id`	(required)	As for `repair preview`
`--rule-index`	all rules	Apply only this rule, by its index in `repair preview`. Repeatable
`--format`	`table`	`table` or `json`

Replay

`decimalai replay run`

Execute a replay batch: re-run its pending tasks through your updated agent and submit the results. Replay explains the workflow.

decimalai replay run batch_abc123 --agent-fn my_app.agent:run

Argument / Option	Required	Description
`BATCH_ID`	Yes	The replay batch to execute
`--agent-fn`	Yes	Python path to your agent function (`module:function`)
`--dry-run`	No	Show how many tasks would run, without running them
`--no-skip-failed`	No	Stop on first failure instead of continuing

Output:

✓ Loaded agent function: my_app.agent:run
  ✓ [1/50] abc12345: keep
  ✗ [3/50] ghi11111: agent error: connection timed out
  ...
========================================
Replay Summary
========================================
  Total:      50
  Completed:  45
  Passed:     42
  Failed:     3
  Skipped:    2
  Pass rate:  84%

The CLI exits with code 1 if any replays fail, so you can use it in CI. A task whose original trace has no input is skipped.

Datasets

Fine-tuning and datasets explains the formats and how to build one from the dashboard.

`decimalai datasets list`

List the datasets in the workspace with row counts and version info.

decimalai datasets list

Option	Default	Description
`--limit`	20	Max results

--help also lists --agent-name. The command accepts it and doesn’t apply it.

`decimalai datasets show`

Show version history for a specific dataset.

decimalai datasets show ds_abc123

`decimalai datasets pull`

Pull a dataset version to a local file.

decimalai datasets pull ds_abc123 -o ./training_data.jsonl
decimalai datasets pull ds_abc123 -o ./data.jsonl --version v2
decimalai datasets pull ds_abc123 -o ./data.parquet

Option	Default	Description
`--output / -o`	(required)	Output file path. Format inferred from extension.
`--version / -v`	`latest`	Version: `latest`, `v3`, `3`, or UUID
`--format`	auto	Override: `jsonl` or `parquet`

`decimalai datasets export`

Export a dataset version to stdout (for piping) or a file.

decimalai datasets export ds_abc123 | head -5
decimalai datasets export ds_abc123 --version v2 -o training.jsonl

Option	Default	Description
`--version / -v`	`latest`	Version specifier
`--format`	`jsonl`	Format: `jsonl` or `parquet`. Parquet needs `-o`
`--output / -o`	stdout	Output file path

`decimalai datasets build`

Build a new dataset version from traces. It prints the new version’s id and row count.

decimalai datasets build ds_abc123

--help also lists --verdict, but the server doesn’t apply it, so the build isn’t filtered. To filter by verdict, use allowed_verdicts through the API.

`decimalai datasets push-to-hub`

Push a dataset to HuggingFace Hub, so any tool that supports load_dataset() can load it. HuggingFace Hub Integration shows how to train on it.

decimalai datasets push-to-hub ds_abc123 my-org/support-agent-sft
decimalai datasets push-to-hub ds_abc123 my-org/my-dataset --version v2 --public

Argument / Option	Default	Description
`DATASET_ID`, `REPO_ID`	(required)	The dataset ID, and the HuggingFace repo: `org/dataset-name`
`--version / -v`	`latest`	Version specifier
`--token`	`$HF_TOKEN`	HuggingFace API token
`--public / --private`	`--private`	Repo visibility
`--split`	`train`	Dataset split name

Requires pip install huggingface_hub datasets.

Python SDK

Skills in your repo

CLI

REST

Runs, checks and datasets

Traces

`decimalai traces list`

`decimalai traces show`

`decimalai traces stats`

`decimalai traces import`

Evaluations

`decimalai eval push`

Evaluators

`decimalai evaluators add`

`decimalai evaluators list`

`decimalai evaluators remove`

`decimalai evaluators templates`

Manifests

`decimalai manifests list`

Regression and compatibility checks

`decimalai regression-check`

`decimalai compat-check`

Repair

`decimalai repair preview`

`decimalai repair apply`

Replay

`decimalai replay run`

Datasets

`decimalai datasets list`

`decimalai datasets show`

`decimalai datasets pull`

`decimalai datasets export`

`decimalai datasets build`

`decimalai datasets push-to-hub`

​Traces

​decimalai traces list

​decimalai traces show

​decimalai traces stats

​decimalai traces import

​Evaluations

​decimalai eval push

​Evaluators

​decimalai evaluators add

​decimalai evaluators list

​decimalai evaluators remove

​decimalai evaluators templates

​Manifests

​decimalai manifests list

​Regression and compatibility checks

​decimalai regression-check

​decimalai compat-check

​Repair

​decimalai repair preview

​decimalai repair apply

​Replay

​decimalai replay run

​Datasets

​decimalai datasets list

​decimalai datasets show

​decimalai datasets pull

​decimalai datasets export

​decimalai datasets build

​decimalai datasets push-to-hub

Traces

`decimalai traces list`

`decimalai traces show`

`decimalai traces stats`

`decimalai traces import`

Evaluations

`decimalai eval push`

Evaluators

`decimalai evaluators add`

`decimalai evaluators list`

`decimalai evaluators remove`

`decimalai evaluators templates`

Manifests

`decimalai manifests list`

Regression and compatibility checks

`decimalai regression-check`

`decimalai compat-check`

Repair

`decimalai repair preview`

`decimalai repair apply`

Replay

`decimalai replay run`

Datasets

`decimalai datasets list`

`decimalai datasets show`

`decimalai datasets pull`

`decimalai datasets export`

`decimalai datasets build`

`decimalai datasets push-to-hub`