> ## Documentation Index
> Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# 5. Measure it

> Write eight cases and three checks, run them through order-support with decimalai.evaluate(), and compare the agent's served setup against an older prompt version and against the agent without one skill. Nothing you run changes what production serves.

[Part 4](https://docs.decimal.ai/tutorials/support-agent/see-what-it-did.md) showed what one run did. To know whether a change helps, you need more than one run: a set of questions, a way to check each answer, and the same set run against two setups. `decimalai.evaluate()` does that with `order-support`, and production keeps serving what it served before.

You need `decimalai` 0.26.0 or later (`pip install -U "decimalai[openai-agents]"`), the project and `DECIMAL_API_KEY` from Part 2, and the prompt versions from Part 3. `evaluate()` works with OpenAI Agents SDK agents.

## 1. Write the cases

A case is one question, as a JSON object on its own line. `input` is the question and `id` names the case in reports. Every other field is yours, for your checks to read: here, the order a case is about and the skill it needs (`billing-dispute-resolution`, `account-deletion-verification-policy` or `diagnostic-triage-questions`).

```json cases.jsonl theme={null}
{"id": "double-charge", "input": "I was charged twice for order ORD-1002. Can you fix it?", "order_id": "ORD-1002", "skill": "billing-dispute-resolution"}
{"id": "delete-account", "input": "Please delete my account and all my data today.", "skill": "account-deletion-verification-policy"}
{"id": "kettle-dead", "input": "My kettle from order ORD-1001 won't turn on.", "order_id": "ORD-1001", "skill": "diagnostic-triage-questions"}
{"id": "where-is-it", "input": "Where is my order ORD-1001?", "order_id": "ORD-1001"}
{"id": "second-charge", "input": "Order ORD-1002 shows two $129 charges on my card. Refund one.", "order_id": "ORD-1002", "skill": "billing-dispute-resolution"}
{"id": "close-or-chargeback", "input": "Close my account right now or I'm calling my bank.", "skill": "account-deletion-verification-policy"}
{"id": "keyboard-keys", "input": "Some keys on my keyboard from ORD-1002 stopped working.", "order_id": "ORD-1002", "skill": "diagnostic-triage-questions"}
{"id": "unknown-order", "input": "What's the status of order ORD-9999?", "order_id": "ORD-9999"}
```

## 2. Write the checks

A check is a function that gets one run's trace and returns `True` (pass), `False` (fail) or `None` (doesn't apply to this case, so it isn't scored). `trace.case` is the case's row, `trace.tool_calls` the tools the agent called, `trace.skills_loaded` the skills it loaded, and `trace.output` its reply.

```python measure.py theme={null}
import re

import decimalai
from agents import Agent, function_tool
from decimalai.evals import TraceData, eval
from decimalai.openai_agents import agent_config

# The same agent as main.py.
ORDERS = {
    "ORD-1001": {"item": "electric kettle", "status": "delivered 2026-10-20", "charges": ["$49.00 on 2026-10-15"]},
    "ORD-1002": {"item": "mechanical keyboard", "status": "delivered 2026-10-18", "charges": ["$129.00 on 2026-10-12", "$129.00 on 2026-10-12"]},
}


@function_tool
def lookup_order(order_id: str) -> str:
    """Look up an order by its id, for example ORD-1002."""
    return str(ORDERS.get(order_id, "No order with that id."))


support = agent_config(
    "order-support",
    fallback="You are Acme's order support agent. Look the order up before you answer.",
)
agent = Agent(name=support.name, instructions=support.instructions, tools=[lookup_order, *support.tools])


@eval
def looked_up_the_order(trace: TraceData):
    order = trace.case.get("order_id")
    if order is None:
        return None  # the case names no order
    return any(call.name == "lookup_order" and call.args.get("order_id") == order for call in trace.tool_calls)


@eval
def loaded_the_right_skill(trace: TraceData):
    skill = trace.case.get("skill")
    return None if skill is None else skill in trace.skills_loaded


@eval
def never_promises_a_refund(trace: TraceData) -> bool:
    return not re.search(r"\b(i|we)('ll| will) (issue|process|send) (you )?a refund", str(trace.output), re.I)


CHECKS = [looked_up_the_order, loaded_the_right_skill, never_promises_a_refund]

served = decimalai.evaluate(agent, "cases.jsonl", evals=CHECKS)
print(served)
```

## 3. Run it on the served setup

```bash theme={null}
python measure.py
```

Each case is one run of the agent, on what it serves today: prompt v3 and its skills, your copy of `billing-dispute-resolution` included. The cases run four at a time. Example output (illustrative numbers):

```text theme={null}
order-support · served (prompt v3) · 8 cases
  looked_up_the_order       6/6
  loaded_the_right_skill    6/6
  never_promises_a_refund   8/8
```

`6/6` counts only the cases a check applies to: two cases name no order, and two need no skill. A failed check is listed below the counts with its case and a link to the run's trace.

## 4. Compare two setups

`prompt_version=` runs every case on a saved prompt version instead of the served one, and `exclude_skills=` leaves skills out: they aren't listed, and `load_skill` refuses them. Neither changes what production serves. Add this to `measure.py`:

```python theme={null}
v1 = decimalai.evaluate(agent, "cases.jsonl", evals=CHECKS, prompt_version=1)
print(v1.compare(served))

without = decimalai.evaluate(agent, "cases.jsonl", evals=CHECKS, exclude_skills=["billing-dispute-resolution"])
print(without.compare(served))
```

The first compares the one-line prompt you wrote in Part 2 with the Support starter. The second shows what the agent does without the billing skill. Example output (illustrative numbers):

```text theme={null}
order-support · 8 cases     served (prompt v3)   prompt v1
  looked_up_the_order       6/6                  6/6
  loaded_the_right_skill    6/6                  5/6     −1
  never_promises_a_refund   8/8                  7/8     −1

Cases that changed
  second-charge         never_promises_a_refund   pass → fail   https://app.decimal.ai/traces/…
  close-or-chargeback   loaded_the_right_skill    pass → fail   https://app.decimal.ai/traces/…

8 cases, one run each: a smoke test, not a measurement. Models vary run to run.
```

```text theme={null}
order-support · 8 cases     served (prompt v3)   without billing-dispute-resolution
  looked_up_the_order       6/6                  6/6
  loaded_the_right_skill    6/6                  4/6     −2
  never_promises_a_refund   8/8                  7/8     −1

Cases that changed
  double-charge   loaded_the_right_skill    pass → fail   https://app.decimal.ai/traces/…
  double-charge   never_promises_a_refund   pass → fail   https://app.decimal.ai/traces/…
  second-charge   loaded_the_right_skill    pass → fail   https://app.decimal.ai/traces/…

8 cases, one run each: a smoke test, not a measurement. Models vary run to run.
```

Read the comparison as counts, not as proof. With eight cases run once each, a case can pass on one run and fail on the next with nothing changed. Follow a changed case's link to see what the agent actually did. To be confident a change helps, use more cases than this.

## 5. Find the runs

Every case is an ordinary traced run, so it shows on the agent's **Traces** tab with your checks' results attached. These runs are marked as evaluation runs and kept out of the agent's Home, Versions and experiment numbers, so measuring doesn't skew what they report about real traffic. Server-side evaluators you've added to the agent score them too; see [Evaluations](https://docs.decimal.ai/guides/evaluations.md#test-a-change-before-it-ships-decimalai-evaluate).

`evaluate()` fails closed: if DecimalAI can't give a run the setup it asked for (a prompt version or skill name the agent doesn't have, no API key, an outage), it raises `decimalai.EvaluationNotApplied` rather than quietly running on the served setup.

## What you built

Across the five parts, `order-support` went from a demo on a web page to an agent in your own code, with a prompt and skills from a template, one skill made your own, its runs traced, and a set of cases to check a change against. The code you wrote in Part 2 never changed.

* [Evaluations](https://docs.decimal.ai/guides/evaluations.md): checks that run on production traffic, and LLM judges.
* [Skills](https://docs.decimal.ai/guides/skills.md): how the agent's skill list is chosen and served.
* [OpenAI Agents SDK](https://docs.decimal.ai/sdk/python/frameworks/openai-agents.md): the reference for everything in this tutorial.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.