Evaluations

The evaluations API has four layers: test (evaluate, before a change ships), push (eval, score, batch_eval, external helpers), read (get_eval_breakdown), and define (the @eval decorator). See the Evaluations guide for when to use which.

`decimalai.evaluate()`

Run each case through an agent once, run the checks on each run’s trace, and report. 5. Measure it is a walkthrough.

import decimalai

served = decimalai.evaluate(agent, "cases.jsonl", evals=CHECKS)
v1 = decimalai.evaluate(agent, "cases.jsonl", evals=CHECKS, prompt_version=1)
print(v1.compare(served))

Parameter	Type	Default	Description
`agent`	`agents.Agent`	required	An OpenAI Agents SDK agent whose instructions come from `decimalai.openai_agents.agent_config()`
`cases`	`str \| Path \| list[dict] \| Dataset`	required	A `.jsonl` file (one JSON object per line), a list of dicts, or a `decimalai.Dataset`. Each case needs `input`; `id` names it; `context` goes to `Runner.run(context=...)`; other fields are read by checks as `trace.case`
`evals`	`list`	required	The `@eval` checks. A plain function of one `TraceData` is wrapped
`prompt_version`	`int \| None`	`None`	Run every case on this saved prompt version instead of the served one
`exclude_skills`	`list[str] \| None`	`None`	Leave these skills out: not listed, and `load_skill` refuses them
`context`	`Callable[[dict], Any] \| None`	`None`	`case -> object` for a typed `Runner.run(context=...)`
`label`	`str \| None`	`None`	This setup’s name in reports. Default: `served`, `prompt v4`, `without refund-policy`
`concurrency`	`int`	`4`	How many cases run at once

Returns an EvalReport:

Member	Description
`print(report)`	Each check’s counts, then the failed cases (with reasons and trace links), checks that raised, and cases that did not run
`report.compare(baseline)`	A `Comparison` of the same cases (matched by `id`): each check’s counts side by side, and the cases whose outcome changed. `print()` it
`report.summary`	`{check: {"passed", "failed", "skipped", "errors"}}`
`report.cases`	One `CaseResult` per case: `id`, `input`, `output`, `trace_id`, `trace_url`, `results` (check → `EvalResult`, or `None` when skipped), `check_errors`, `error`, `prompt_version`
`report.prompt_version`	The saved prompt version the runs used, read off their traces
`report.to_dict()`	All of the above, JSON-ready

Each case is an evaluation run: traced with the checks’ results attached, and kept out of the agent’s own numbers. A check that returns None skips that case; one that raises is reported as an error on it. Sampling (@eval(sampling_rate=...)) is ignored: every check runs on every case. Raises decimalai.EvaluationNotApplied when no case could run on the setup asked for (an unknown prompt version or skill name, no API key, a DecimalAI backend from before evaluation runs); a single case that could not is reported on that case. Raises RuntimeError inside a running event loop: there, await decimalai.evaluate_async(...), which takes the same arguments. To run a single run on another setup, without a set of cases, use support.override(prompt_version=..., exclude_skills=...) around Runner.run (OpenAI Agents SDK), or support.run(message, prompt_version=..., exclude_skills=...) on decimalai.agent_config(). See the Evaluations guide.

`decimalai.eval()`

Push a single eval score to a trace.

decimalai.eval(
    trace_id="abc-123",
    name="factual_accuracy",
    score=0.85,
    reason="4/5 facts verified against source docs",
    source="custom",
)

str

required

The trace to attach the score to.

str

required

Metric name (e.g., "factual_accuracy", "relevance").

float

required

Score value between 0.0 and 1.0.

str

default:"custom"

Eval source identifier. Appears grouped in dashboard.

str

Human-readable display name (e.g., "My RAG Eval").

bool

default:"score >= 0.5"

Binary pass/fail override.

str

Human-readable explanation of the score.

str

default:"quality"

"quality" or "compatibility".

`decimalai.score()`

Shorthand for pushing a single eval score.

decimalai.score("abc-123", "coherence", 0.92)
decimalai.score("abc-123", "factual", 0.75, reason="One hallucination detected")

`decimalai.get_eval_breakdown()`

Get the full eval breakdown for a trace, grouped by source.

bd = decimalai.get_eval_breakdown("abc-123")

print(f"Verdict: {bd['eval_verdict']}")       # "keep", "drop", "replay"
print(f"Quality avg: {bd['quality_avg']}")     # 0.85
print(f"Compat avg: {bd['compat_avg']}")       # 1.0

for group in bd["source_groups"]:
    print(f"  {group['source']}: {group['scores']}")

`decimalai.batch_eval()`

Apply @eval evaluators across a list of traces.

from decimalai import batch_eval
from decimalai.evals import eval, TraceData

@eval(name="has_citation")
def my_evaluator(trace: TraceData) -> bool:
    return "[source:" in trace.output

results = batch_eval(
    trace_ids=["abc-123", "def-456"],
    evals=[my_evaluator],
)
print(f"Evaluated {results['traces_evaluated']} traces")

trace.output is the trace’s whole final answer, up to 20,000 characters, for traces the OpenAI Agents SDK integration sent; other traces carry its first 200 characters.

External Integration Helpers

Import scores from third-party evaluation frameworks:

DeepEval
LangSmith
Custom Scores

from decimalai import push_deepeval_results

# After running DeepEval evaluations
push_deepeval_results(
    deepeval_results,  # DeepEval TestResults object
    trace_id_field="input",  # test-case field holding the trace id
)

from decimalai import push_langsmith_scores

push_langsmith_scores(
    trace_id="abc-123",
    run_scores=langsmith_feedback,  # LangSmith feedback list
)

from decimalai import push_custom_scores

push_custom_scores(
    trace_id="abc-123",
    source="my-eval-pipeline",
    scores=[
        {"name": "relevance", "score": 0.9},
        {"name": "safety", "score": 1.0, "passed": True},
    ],
)

Custom Evaluators (`@eval`)

Define evaluators that run client-side before trace upload:

from decimalai.evals import eval, TraceData, EvalResult

@eval(name="has_citation", category="quality")
def check_citation(trace: TraceData) -> bool:
    """Returns True if the output contains a citation."""
    return "[source:" in trace.output

@eval(name="response_length", category="quality")
def check_length(trace: TraceData) -> EvalResult:
    """Check response length is reasonable."""
    length = len(trace.output)
    return EvalResult(
        score=min(length / 500, 1.0),
        passed=50 < length < 2000,
        reason=f"Length: {length} chars",
    )

# Register evals with your framework.
# The evals= kwarg lives on the LangChain adapter's instrument():
from decimalai.langchain import instrument
instrument(agent_name="my-agent", evals=[check_citation, check_length])

The @eval decorator also accepts category="llm_judge" (for judge-style scorers), sampling_rate (a 0.0–1.0 fraction of traces to evaluate), and version (a string tag for the evaluator definition). A check may also return None when a trace isn’t its concern: no score is recorded. The same checks are what decimalai.evaluate() takes. See the Evaluations guide for built-in evaluators and sampling configuration.

What’s next

Replay

Re-run traces after a manifest change and re-score them.

Datasets

Datasets filter on the eval verdict — keep traces become training data.

Python SDK

Skills in your repo

CLI

REST

`decimalai.evaluate()`

`decimalai.eval()`

`decimalai.score()`

`decimalai.get_eval_breakdown()`

`decimalai.batch_eval()`

External Integration Helpers

Custom Evaluators (`@eval`)

What’s next

Replay

Datasets

​decimalai.evaluate()

​decimalai.eval()

​decimalai.score()

​decimalai.get_eval_breakdown()

​decimalai.batch_eval()

​External Integration Helpers

​Custom Evaluators (@eval)

​What’s next

Replay

Datasets

`decimalai.evaluate()`

`decimalai.eval()`

`decimalai.score()`

`decimalai.get_eval_breakdown()`

`decimalai.batch_eval()`

External Integration Helpers

Custom Evaluators (`@eval`)

What’s next