Evaluations

An evaluation answers one question about a run: did the agent do what it should? DecimalAI uses the same checks in two places:

	Before a change ships	On production traffic
What runs	A set of cases you wrote, through your agent	Your users’ real conversations
How	`decimalai.evaluate()`	Checks on every trace, LLM judges, server-side auto-scoring, scores from other tools
What it tells you	Whether a new prompt version or a skill change helps, compared with what the agent serves now	How the agent is doing, run by run
What it changes	Nothing: production keeps serving what it served	Nothing: scores are added to traces

Both write checks the same way, as @eval functions of a trace.

decimalai.evaluate() accepts only OpenAI Agents SDK agents whose instructions come from decimalai.openai_agents.agent_config(). Any other agent raises a TypeError. On another framework, score runs that already happened with batch_eval(), and use decimalai.agent_config() to make one run on another setup.

Test a change before it ships: `decimalai.evaluate()`

decimalai.evaluate() runs each case in a set through your agent once, runs your checks on each run’s trace, and reports the counts. Run it on the setup the agent serves today, then on a different prompt version or without a skill, and compare the two. Neither run changes what production serves. It needs DECIMAL_API_KEY. 5. Measure it walks through it on the tutorial’s order-support agent.

import decimalai
from decimalai.evals import TraceData, eval


@eval
def looked_up_the_order(trace: TraceData):
    order = trace.case.get("order_id")
    if order is None:
        return None  # this case names no order, so the check doesn't apply
    return any(call.name == "lookup_order" and call.args.get("order_id") == order
               for call in trace.tool_calls)


served = decimalai.evaluate(agent, "cases.jsonl", evals=[looked_up_the_order])
v1 = decimalai.evaluate(agent, "cases.jsonl", evals=[looked_up_the_order], prompt_version=1)
print(v1.compare(served))

Cases are a .jsonl file, a list of dicts, or a decimalai.Dataset. Each needs an input, the message. A check reads the whole row, your own fields included, as trace.case.
The setup. prompt_version=4 runs every case on that saved prompt version. exclude_skills=["refund-policy"] leaves those skills out: they aren’t listed, and load_skill refuses them.
The report. print(report) shows each check’s counts, then the failed cases with links to their traces. A check that returned None for a case isn’t counted for it, so 6/6 means six cases it applied to. A check that raised is an error on that case, never a skip.
The comparison. candidate.compare(baseline) puts two reports of the same cases side by side, matched by id, with the cases whose outcome changed. It counts and doesn’t call a difference an effect: a few cases run once each can flip with nothing changed.

Every argument is in the SDK reference. Each case is an ordinary traced run, marked as an evaluation run. It shows on the agent’s Traces tab with your checks’ results, and stays out of the agent’s Home, Versions and experiment numbers. Your server evaluators still score it, and it counts in the Evaluation page’s pass rates. evaluate() fails closed. If DecimalAI can’t give the runs the setup you asked for (a prompt version or skill the agent doesn’t have, no API key, an outage), it raises decimalai.EvaluationNotApplied, naming what the agent has, rather than running on the served setup. Tracing has to be on: calling it inside an open agents.trace() raises, and a case run with OpenAI Agents tracing disabled is reported as not checked.

One run on another setup

The same overrides work on a single run, without a set of cases:

OpenAI Agents SDK
Your own model calls (decimalai.agent_config)

with support.override(prompt_version=4, exclude_skills=["refund-policy"]) as trial:
    result = await Runner.run(agent, "My mug arrived broken")
print(trial.trace_url)   # that run's trace; trial.trace_id is its id

support = decimalai.agent_config("support", fallback=BASE_PROMPT)
run = support.run(message, prompt_version=4, exclude_skills=["refund-policy"])
# system=run.system, tools=[*my_tools, run.tool_spec]; on a load_skill call: run.load_skill(name)
print(run.trace_id)

These are evaluation runs too, and fail closed the same way. The framework-specific agent_config()s for the OpenAI SDK, the Anthropic API and the others don’t take these overrides.

Write a check

eval is two different things. decimalai.evals.eval is the decorator that defines a check. The top-level decimalai.eval(...) records an already-computed score on a trace (see External score import).

A check is a function of one trace. The same function works in evaluate() and on production traces:

from decimalai.evals import eval, TraceData

@eval(name="has_citation")
def check_citation(trace: TraceData) -> bool:
    """Returns True if the output includes a citation."""
    return "[source:" in trace.output

What a check returns becomes its score:

Return type	Pass logic	Example
`bool`	`True` = pass (score 1.0), `False` = fail (score 0.0)	`return "[source:" in trace.output`
`float`	0.0–1.0 score, pass at 0.5 or above	`return 0.85`
`dict`	Several scores: each key becomes a separate score	`return {"clarity": 0.8, "accuracy": 0.9}`
`EvalResult`	Full control over score, pass/fail, and reason	`return EvalResult(score=0.9, passed=True, reason="Professional tone")`
`None`	The check doesn’t apply to this run: no score	`return None`

Sampling rate

On production traffic, @eval(name="expensive_llm_review", sampling_rate=0.1) runs an expensive check on 10% of traces. evaluate() ignores this and runs every check on every case.

TraceData fields

Field	Type	Description
`trace.input`	`str`	User input / query
`trace.output`	`str`	The agent’s whole final answer, up to 20,000 characters, from the OpenAI Agents SDK integration; other traces carry its first 200 characters
`trace.tool_calls`	`list[ToolCallView]`	Tool calls, each with `name`, `args` and `result`
`trace.llm_calls`	`list[LlmCallView]`	LLM calls, each with its model and token counts
`trace.skills_offered`	`list[str]`	The skills listed to the model on this run: Offered
`trace.skills_loaded`	`list[str]`	The skills the model loaded with `load_skill`: Loaded
`trace.active_skills`	`list[str]`	Loaded skills whose version came back with their text, and any your code declared. Empty when nothing reports a load
`trace.case`	`dict`	Under `evaluate()`, the case’s row from your cases; empty otherwise
`trace.status`, `trace.latency_ms`, `trace.total_tokens`		Whether the run succeeded (`"success"` or `"error"`), how long it took, and its tokens
`trace.metadata`	`dict`	Custom metadata

The LLM judges DecimalAI runs on the server read the same answer, up to its first 12,000 characters; a longer one is shown cut, with a note saying so.

Score production traces

A check on production traffic is one of three kinds: deterministic (a rule, instant and free), an LLM judge (a model scores the output), or custom (your own @eval function).

Checks attach through decimalai.langchain.instrument(), not init(). The evals= and builtin_evals= arguments are LangChain-only. On other frameworks, attach evaluators on the server, run batch_eval(), or push scores with decimalai.eval().

import decimalai
from decimalai.langchain import instrument

decimalai.init(api_key="dai_sk_...")
instrument(agent_name="my-bot", evals=[check_citation])

Pre-built evaluators

DecimalAI ships 10 ready-to-use evaluators in two tiers.

Deterministic evaluators (free)

Evaluator	What it checks
`json_valid`	Output is valid JSON
`contains(patterns)`	Output contains required substrings, as in `contains(["source:", "http"])`
`not_contains(patterns)`	Output doesn’t contain banned strings
`regex_match(pattern)`	Output matches a regex pattern
`length_check(min, max)`	Word or character count is within bounds, as in `length_check(min_words=10, max_words=500)`

LLM-as-judge evaluators

Evaluator	What it scores	Input needed
`Relevance()`	Does the output address the input?	input + output
`Factuality()`	Is the output grounded in facts?	input + output + context
`Faithfulness()`	Is output faithful to tool results?	output + tool_calls
`Toxicity()`	Is the content safe and non-harmful?	output
`Conciseness()`	Is the output appropriately concise?	input + output

import decimalai
from decimalai.evals import Relevance, Toxicity, json_valid
from decimalai.langchain import instrument

decimalai.init(api_key="dai_sk_...")
instrument(
    agent_name="my-rag-bot",
    evals=[json_valid, Relevance(), Toxicity(model="claude-haiku-5-5")],  # the default judge is gpt-5.4-mini
)

The LLM judges score production traces, not evaluate() runs. A judge in your process needs [evals], which can’t be installed beside the OpenAI Agents SDK. A judge on DecimalAI scores a trace DecimalAI already has, and evaluate() sends each case’s trace only after its checks have run. Under evaluate(), write checks as plain Python.

Where an LLM judge runs

	In your process	On DecimalAI: `Relevance(use_server=True)`
The key	Yours, through litellm: `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` and so on	DecimalAI’s
The judge model	Any model litellm can call	DecimalAI’s
The limit	None from DecimalAI	Your plan’s monthly LLM-judge calls. Over it, the call returns an error
What DecimalAI sees	Neither the judge’s prompts nor its responses	The trace it already has (`POST /api/v1/traces/{id}/evaluate`, editor role)

A judge in your process needs the extra:

pip install "decimalai[evals]>=0.27"

[evals] brings litellm, which pins openai<3, so it can’t share an environment with [openai], [openai-agents] or [pydantic-ai], and it isn’t part of [all]. Keep the >=0.27 floor: without it, pip resolves that conflict by quietly installing an old decimalai.

Built-in checks (every trace, on the server)

DecimalAI runs 5 basic checks on every trace it receives, from any SDK or a bare POST /api/v1/traces, with nothing to configure and no cost:

Check	Scores
`completion`	`1.0` if the run succeeded
`has_output`	`1.0` if there is a final output
`tool_compliance`	`1.0` unless a tool call failed to return. If a trace can’t show what a call returned, that call counts against this only when the run failed
`latency`	Falls from `1.0` at 0 seconds to `0.0` at 10
`token_efficiency`	Falls from `1.0` at 0 tokens to `0.0` at 5,000

A trace shows them as Built-in. If an SDK sends its own scores under these five names, the server’s replace them. They count toward a trace’s verdict only until a real grade arrives (see Verdicts).

Auto-scoring

To score an agent’s traffic on the server, attach evaluators to it: Add evaluator on the agent’s Evals tab, an empty cell of the coverage matrix, or the API below. Each enabled evaluator runs on every trace the agent sends, as it arrives. An evaluator with no agent runs on every agent’s traces. An evaluator starts from a template. GET /api/v1/evaluators/templates lists them:

Kind	Templates	Cost
LLM judge	`relevance`, `correctness`, `helpfulness`, `conciseness`, `toxicity`, `pii_detection`, `factuality`, `faithfulness`, `context_precision`, `tool_selection`, `step_efficiency`	Your plan’s monthly LLM-judge calls
Deterministic	`completion` (a non-empty, successful response), `skill_output_check` (the output has the sections and patterns you require, and a minimum length)	Free

When the allowance is used up, LLM-judge evaluators are skipped on incoming traces until the next month; nothing fails. Evaluators also score evaluate()’s evaluation runs, and LLM judges on them count against the allowance.

curl -X POST -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  https://api.decimal.ai/api/v1/evaluators \
  -d '{"agent_name": "my-bot", "template_id": "relevance", "eval_type": "llm_judge", "enabled": true}'

For a deterministic one, send "eval_type": "deterministic" with a template_id, "name" and "category": "quality". GET /api/v1/evaluators?agent_name=my-bot lists what runs on an agent’s traces. There is no server-side JSON-validity check: use the SDK’s json_valid in your process. To score one trace on demand, POST /api/v1/traces/{id}/evaluate runs the free deterministic checks. Send {"mode": "llm_judge"} for a judge, with "evaluator": "relevance" to name one. Judge runs count against the allowance.

Eval dashboard

Each agent’s Evals tab shows its scores and failures. The Evaluation page, at /evaluate, shows every agent at once: open it from the command palette’s Evals entry.

The DecimalAI Evaluation page with its stat cards, coverage matrix and recent failures. — The Evaluation page.

An empty cell of its coverage matrix is a category nobody is checking on that agent, and it offers to add one.

Score sources

An eval score is one check’s result on one trace: its name, a score between 0 and 1, whether it passed, its source and its category. quality scores ask whether the output is good. compatibility scores ask whether the trace still works with the agent’s current setup. source says who produced a score:

`source`	Where it comes from
`built_in`	The five built-in checks
`deterministic`, `llm_judge`	A server evaluator, or `POST /traces/{id}/evaluate`
`custom`, `builtin`	Checks that ran in your process and were sent with the trace. `custom` is also the default for `decimalai.eval()`
`sdk`	A score sent with a trace that names no source of its own
`evaluate`, `batch_eval`	`decimalai.evaluate()` and `batch_eval()`
`deepeval-import`, `langsmith-import`	`push_deepeval_results()` and `push_langsmith_scores()`
`compat_engine`	A trace compared with a manifest. Its scores have the category `compatibility`
Any other name	The `source` you pushed: see REST API

A trace keeps one score per name and source, so pushing the same pair again updates it. GET /api/v1/traces/{id}/eval-breakdown, or decimalai.get_eval_breakdown(trace_id), returns a trace’s scores by source (source_groups), with eval_verdict, quality_avg, compat_avg and decision_reasons. The score routes are on Common endpoints.

Verdicts

Scores add up to a verdict on each trace. A trace can carry two, on separate axes:

The eval verdict, pass / fail / review: is this output good? A trace with none yet is shown as Unevaluated.
The training verdict, keep / repair / replay / drop: what to do with this trace as training data.

How the eval verdict is set:

When	Pass	Fail	Review
As a trace arrives, from the scores the SDK sent and from your server evaluators	Every score passed	Not all passed, and the average is below 0.4	Otherwise
When scores are pushed or imported later, on a trace with no verdict yet	Average 0.75 or above	Average below 0.4	Otherwise

So a trace arriving with five passing scores and one failing score gets review, or fail if their average is below 0.4. The five built-in checks don’t set a verdict as a trace arrives, and later they count toward the average only when the trace has no other score, so they can’t outvote a real grade. The training verdict takes the first of these that matches:

A compatibility score at 0 or below: drop
A compatibility score below 0.2: replay
A quality average below 0.4: drop
A compatibility score below 0.5: repair
Anything else: keep

So low quality overrides a repair, because a poor output isn’t worth fixing, and a trace has to clear both quality and compatibility to be kept. POST /api/v1/traces/{id}/decision works out a trace’s training verdict from its scores and saves it as compat_status. POST /api/v1/traces/batch-decision does the same for trace_ids or every trace of a manifest_id, and runs nothing again. To set one by hand, PATCH /api/v1/traces/{id}/compatibility takes target_manifest_id and override_verdict. Verdicts feed Datasets, where allowed_verdicts limits a dataset to traces with given verdicts (for example ["keep"] or ["pass"]), and the pass rates and failure lists on the dashboard.

Batch evaluation

Run evaluators across traces that already exist. This works on any framework’s traces:

from decimalai.evals import batch_eval, Relevance, Toxicity

results = batch_eval(
    trace_ids=["abc", "def", "ghi"],
    evals=[Relevance(), Toxicity()],
    max_workers=4,
)

print(results["summary"])
# {"relevance": {"passed": 2, "failed": 1, "total": 3}, "toxicity": {"passed": 3, "failed": 0, "total": 3}}

External score import

Push scores from other tools (DeepEval, LangSmith, your own pipeline). They appear beside built-in scores and feed the same verdicts.

SDK convenience methods

import decimalai

decimalai.eval(trace_id="abc-123", name="correctness", score=0.91, reason="Factually accurate")

decimalai.push_deepeval_results() and decimalai.push_langsmith_scores() turn those tools’ results into scores: see External Integration Helpers.

REST API

curl -X POST "https://api.decimal.ai/api/v1/traces/abc-123/eval-scores" \
  -H "Authorization: Bearer $DECIMAL_API_KEY" -H "Content-Type: application/json" \
  -d '{"source": "custom", "scores": [{"name": "faithfulness", "score": 0.92}]}'

source is required and applies to the whole batch. The names deepeval, langsmith, braintrust and ragas are reserved and rejected with a 422 here: use custom, or a name of your own like my-pipeline. Without passed, a score passes at 0.5 or above.

LangSmith scores

There is no inbound webhook to point LangSmith at. Call push_langsmith_scores() once its online eval run finishes. Each key becomes the metric name and comment the reason.

Start here

Tutorial: a support agent

Use skills

Manage and measure

Publish a skill

More

Test a change before it ships: `decimalai.evaluate()`

One run on another setup

Write a check

Sampling rate

TraceData fields

Score production traces

Pre-built evaluators

Deterministic evaluators (free)

LLM-as-judge evaluators

Where an LLM judge runs

Built-in checks (every trace, on the server)

Auto-scoring

Eval dashboard

Score sources

Verdicts

Batch evaluation

External score import

SDK convenience methods

REST API

LangSmith scores

Next Steps

5. Measure it

SDK reference

​Test a change before it ships: decimalai.evaluate()

​One run on another setup

​Write a check

​Sampling rate

​TraceData fields

​Score production traces

​Pre-built evaluators

​Deterministic evaluators (free)

​LLM-as-judge evaluators

​Where an LLM judge runs

​Built-in checks (every trace, on the server)

​Auto-scoring

​Eval dashboard

​Score sources

​Verdicts

​Batch evaluation

​External score import

​SDK convenience methods

​REST API

​LangSmith scores

​Next Steps

5. Measure it

SDK reference

Test a change before it ships: `decimalai.evaluate()`

One run on another setup

Write a check

Sampling rate

TraceData fields

Score production traces

Pre-built evaluators

Deterministic evaluators (free)

LLM-as-judge evaluators

Where an LLM judge runs

Built-in checks (every trace, on the server)

Auto-scoring

Eval dashboard

Score sources

Verdicts

Batch evaluation

External score import

SDK convenience methods

REST API

LangSmith scores

Next Steps