Replay

When your agent changes, some past traces go stale. Replay takes their inputs, runs them through your updated agent, and links each new trace to its original so DecimalAI can score the pair.

How It Works

DecimalAI never runs your agent. You do, on your own infrastructure:

Pull the inputs. By default, these come from traces with a replay or drop verdict.
Run your agent on each one, with tracing on, so each run records a new trace.
Link or submit each new trace. DecimalAI scores it against the original.

An input comes whole from the run’s recorded model calls while retention keeps them, about 60 days. After that it’s the trace’s stored preview, cut at 200 characters, so a longer prompt replays cut short.

A batch’s target manifest labels the results. It doesn’t change what runs, so point your agent function at the code you mean to test.

When to Replay

Change	Replay?	Why
Added a tool	Usually not	Existing traces still fit
Rewrote the prompt	Yes	Outputs may differ
Removed a tool	Yes	Traces that used it are stale
Fixed a bug inside a tool	By choice	The manifest may not change, so no trace gets marked for you

Repair vs Replay: For a schema change such as a renamed parameter, Repair rewrites stored data and costs nothing. Use replay when the agent’s behavior changed.

In the Dashboard

Select your agent, then open More → Replay, which opens Regression checks with its Replay section expanded. The dashboard shows and exports replays. To run one, use the calls below.

Export Prompts downloads the inputs as JSONL.
Replay Batches lists each batch with its status and counts. Open one to see its tasks.
Prepare SDK replay, on Code changes, opens this section for one version pair, with Export prompts for this pair and Copy scoped setup.

SDK

replay.run() does the whole loop. Your function takes an input string, and its runs must be traced under this agent name:

import decimalai
from decimalai import replay

decimalai.init(api_key="dai_sk_...")

@decimalai.trace(agent_name="support-agent")
def run_agent(user_input: str) -> str:
    return call_my_agent(user_input)

results = replay.run(agent_fn=run_agent, agent_name="support-agent", limit=50)
print(f"{results.passed} of {results.completed} passed ({results.pass_rate:.0%})")

To run the loop yourself, pull with get_prompts() and connect each new trace with link():

from decimalai import replay

for prompt in replay.get_prompts("support-agent", verdict="replay", limit=50):
    new_trace_id = run_and_get_trace_id(prompt.user_input)
    result = replay.link(prompt.trace_id, new_trace_id)
    print(prompt.trace_id, result["eval_verdict"], result["eval_score"])

replay.run_sessions() and get_sessions() do this for conversations, passing earlier turns as history. To work from a list of traces, create a batch. These calls return plain dicts:

import decimalai

stale = decimalai.get_replay_prompts("support-agent", verdict="replay")
print(stale["total"], "stale prompts")

batch = decimalai.create_replay_batch(
    source_manifest_id=stale["prompts"][0]["manifest_id"],
    target_manifest_id="TARGET_MANIFEST_ID",
    trace_ids=[p["trace_id"] for p in stale["prompts"]],
)

status = decimalai.get_replay_batch(batch["batch_id"])
print(status["status"], f"{status['completed_tasks']}/{status['total_tasks']}")

task = status["tasks"][0]
print(task["task_input"]["user_input"])
decimalai.submit_replay_result(task_id=task["id"], replayed_trace_id=new_trace_id)

Watch the field names: a batch is batch_id when created but id when read back. A prompt’s text is user_input, and a task’s is task_input["user_input"]. decimalai replay run BATCH_ID --agent-fn my_app.agent:run runs a batch’s pending tasks, and --dry-run previews them. It can’t create a batch. See Runs and data.

REST

curl "https://api.decimal.ai/api/v1/replay/export?agent_name=support-agent&verdict=replay&format=json" \
  -H "Authorization: Bearer $DECIMAL_API_KEY"

curl -X POST https://api.decimal.ai/api/v1/replay/batches \
  -H "Authorization: Bearer $DECIMAL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"target_manifest_id": "TARGET_MANIFEST_ID", "trace_ids": ["TRACE_ID"]}'

Route under `/api/v1/replay`	What it does
`GET /export`	Inputs as JSONL, or `format=json`. Needs `agent_name`. Optional: `verdict`, `limit`, `source_manifest_id` with `target_manifest_id`
`GET /export/sessions`	The same, grouped into conversations
`POST /batches`	One task per trace. `source_manifest_id` is optional
`GET /batches/{batch_id}`	Status (`pending`, `in_progress`, `completed`), counts and tasks
`GET /tasks/{task_id}`	One task, with its `replayability`
`POST /tasks/{task_id}/submit`	A task’s new trace. Scored for you unless you send `eval_score` and `eval_verdict`
`POST /link`	Links two traces without a batch of your own
`GET /batches/{batch_id}/export-dpo`	Preference pairs

Pass both manifest IDs to export to get the verdicts your policy set for that pair. Without them, export, get_prompts() and replay.run() read each trace’s agent-wide status, which can differ. replayability is a hint from the input’s text: not_replayable for no input or a mention of uploads or files, partially_replayable for words like “today”, otherwise fully_replayable. decimalai replay run skips a task only when it has no input.

What Happens After Replay

Each new trace is a normal trace you can read. DecimalAI also scores it against the original, with four checks:

completion: the run ended with status success.
output_format: it has an output and no error.
tool_compliance: it used only expected tools. Replay sets no expected list, so this passes.
similarity: it used the same set of tools, with 0.3 to 3 times as many LLM calls.

The score is the share that passed. A run that didn’t complete is fail. Otherwise a score of 0.75 or more is pass, 0.5 or more is review, and anything lower is fail. Batch progress counts these as equivalent and regressed. These checks compare how the run went, not what it said. To judge the answer, use evaluators.

DPO Pair Generation

A replay can become training data for preference-based fine-tuning (DPO). On a completed batch, choose Export DPO Pairs, or call the route:

{
  "prompt": "How do I reset my password?",
  "chosen": "Go to Settings, then Security, then Reset Password.",
  "rejected": "I don't have access to account settings."
}

A score of 0.6 or more makes the replay chosen. A score of 0.4 or less makes the original chosen. Anything between is skipped. Set the margin with score_threshold (default 0.1). Batches filled by link() or replay.run() stay in_progress, so the button doesn’t appear for them. Use the route.

Start here

Tutorial: a support agent

Use skills

Manage and measure

Publish a skill

More

How It Works

When to Replay

In the Dashboard

SDK

REST

What Happens After Replay

DPO Pair Generation

Next Steps

Datasets

Manifests

​How It Works

​When to Replay

​In the Dashboard

​SDK

​REST

​What Happens After Replay

​DPO Pair Generation

​Next Steps

Datasets

Manifests

How It Works

When to Replay

In the Dashboard

SDK

REST

What Happens After Replay

DPO Pair Generation

Next Steps