5. Measure it

Part 4 showed what one run did. To know whether a change helps, you need more than one run: a set of questions, a way to check each answer, and the same set run against two setups. decimalai.evaluate() does that with order-support, and production keeps serving what it served before. You need decimalai 0.26.0 or later (pip install -U "decimalai[openai-agents]"), the project and DECIMAL_API_KEY from Part 2, and the prompt versions from Part 3. evaluate() works with OpenAI Agents SDK agents.

1. Write the cases

A case is one question, as a JSON object on its own line. input is the question and id names the case in reports. Every other field is yours, for your checks to read: here, the order a case is about and the skill it needs (billing-dispute-resolution, account-deletion-verification-policy or diagnostic-triage-questions).

cases.jsonl

{"id": "double-charge", "input": "I was charged twice for order ORD-1002. Can you fix it?", "order_id": "ORD-1002", "skill": "billing-dispute-resolution"}
{"id": "delete-account", "input": "Please delete my account and all my data today.", "skill": "account-deletion-verification-policy"}
{"id": "kettle-dead", "input": "My kettle from order ORD-1001 won't turn on.", "order_id": "ORD-1001", "skill": "diagnostic-triage-questions"}
{"id": "where-is-it", "input": "Where is my order ORD-1001?", "order_id": "ORD-1001"}
{"id": "second-charge", "input": "Order ORD-1002 shows two $129 charges on my card. Refund one.", "order_id": "ORD-1002", "skill": "billing-dispute-resolution"}
{"id": "close-or-chargeback", "input": "Close my account right now or I'm calling my bank.", "skill": "account-deletion-verification-policy"}
{"id": "keyboard-keys", "input": "Some keys on my keyboard from ORD-1002 stopped working.", "order_id": "ORD-1002", "skill": "diagnostic-triage-questions"}
{"id": "unknown-order", "input": "What's the status of order ORD-9999?", "order_id": "ORD-9999"}

2. Write the checks

A check is a function that gets one run’s trace and returns True (pass), False (fail) or None (doesn’t apply to this case, so it isn’t scored). trace.case is the case’s row, trace.tool_calls the tools the agent called, trace.skills_loaded the skills it loaded, and trace.output its reply.

measure.py

import re

import decimalai
from agents import Agent, function_tool
from decimalai.evals import TraceData, eval
from decimalai.openai_agents import agent_config

# The same agent as main.py.
ORDERS = {
    "ORD-1001": {"item": "electric kettle", "status": "delivered 2026-10-20", "charges": ["$49.00 on 2026-10-15"]},
    "ORD-1002": {"item": "mechanical keyboard", "status": "delivered 2026-10-18", "charges": ["$129.00 on 2026-10-12", "$129.00 on 2026-10-12"]},
}


@function_tool
def lookup_order(order_id: str) -> str:
    """Look up an order by its id, for example ORD-1002."""
    return str(ORDERS.get(order_id, "No order with that id."))


support = agent_config(
    "order-support",
    fallback="You are Acme's order support agent. Look the order up before you answer.",
)
agent = Agent(name=support.name, instructions=support.instructions, tools=[lookup_order, *support.tools])


@eval
def looked_up_the_order(trace: TraceData):
    order = trace.case.get("order_id")
    if order is None:
        return None  # the case names no order
    return any(call.name == "lookup_order" and call.args.get("order_id") == order for call in trace.tool_calls)


@eval
def loaded_the_right_skill(trace: TraceData):
    skill = trace.case.get("skill")
    return None if skill is None else skill in trace.skills_loaded


@eval
def never_promises_a_refund(trace: TraceData) -> bool:
    return not re.search(r"\b(i|we)('ll| will) (issue|process|send) (you )?a refund", str(trace.output), re.I)


CHECKS = [looked_up_the_order, loaded_the_right_skill, never_promises_a_refund]

served = decimalai.evaluate(agent, "cases.jsonl", evals=CHECKS)
print(served)

3. Run it on the served setup

python measure.py

Each case is one run of the agent, on what it serves today: prompt v3 and its skills, your copy of billing-dispute-resolution included. The cases run four at a time. Example output (illustrative numbers):

order-support · served (prompt v3) · 8 cases
  looked_up_the_order       6/6
  loaded_the_right_skill    6/6
  never_promises_a_refund   8/8

6/6 counts only the cases a check applies to: two cases name no order, and two need no skill. A failed check is listed below the counts with its case and a link to the run’s trace.

4. Compare two setups

prompt_version= runs every case on a saved prompt version instead of the served one, and exclude_skills= leaves skills out: they aren’t listed, and load_skill refuses them. Neither changes what production serves. Add this to measure.py:

v1 = decimalai.evaluate(agent, "cases.jsonl", evals=CHECKS, prompt_version=1)
print(v1.compare(served))

without = decimalai.evaluate(agent, "cases.jsonl", evals=CHECKS, exclude_skills=["billing-dispute-resolution"])
print(without.compare(served))

The first compares the one-line prompt you wrote in Part 2 with the Support starter. The second shows what the agent does without the billing skill. Example output (illustrative numbers):

order-support · 8 cases     served (prompt v3)   prompt v1
  looked_up_the_order       6/6                  6/6
  loaded_the_right_skill    6/6                  5/6     −1
  never_promises_a_refund   8/8                  7/8     −1

Cases that changed
  second-charge         never_promises_a_refund   pass → fail   https://app.decimal.ai/traces/…
  close-or-chargeback   loaded_the_right_skill    pass → fail   https://app.decimal.ai/traces/…

8 cases, one run each: a smoke test, not a measurement. Models vary run to run.

order-support · 8 cases     served (prompt v3)   without billing-dispute-resolution
  looked_up_the_order       6/6                  6/6
  loaded_the_right_skill    6/6                  4/6     −2
  never_promises_a_refund   8/8                  7/8     −1

Cases that changed
  double-charge   loaded_the_right_skill    pass → fail   https://app.decimal.ai/traces/…
  double-charge   never_promises_a_refund   pass → fail   https://app.decimal.ai/traces/…
  second-charge   loaded_the_right_skill    pass → fail   https://app.decimal.ai/traces/…

8 cases, one run each: a smoke test, not a measurement. Models vary run to run.

Read the comparison as counts, not as proof. With eight cases run once each, a case can pass on one run and fail on the next with nothing changed. Follow a changed case’s link to see what the agent actually did. To be confident a change helps, use more cases than this.

5. Find the runs

Every case is an ordinary traced run, so it shows on the agent’s Traces tab with your checks’ results attached. These runs are marked as evaluation runs and kept out of the agent’s Home, Versions and experiment numbers, so measuring doesn’t skew what they report about real traffic. Server-side evaluators you’ve added to the agent score them too; see Evaluations. evaluate() fails closed: if DecimalAI can’t give a run the setup it asked for (a prompt version or skill name the agent doesn’t have, no API key, an outage), it raises decimalai.EvaluationNotApplied rather than quietly running on the served setup.

What you built

Across the five parts, order-support went from a demo on a web page to an agent in your own code, with a prompt and skills from a template, one skill made your own, its runs traced, and a set of cases to check a change against. The code you wrote in Part 2 never changed.

Evaluations: checks that run on production traffic, and LLM judges.
Skills: how the agent’s skill list is chosen and served.
OpenAI Agents SDK: the reference for everything in this tutorial.

Get Started

Tutorial: a support agent

Foundation

Ship Safely

Improve with skills

Trust & Safety

Data & Training

Operate

Research

Core Concepts

Platform

1. Write the cases

2. Write the checks

3. Run it on the served setup

4. Compare two setups

5. Find the runs

What you built

​1. Write the cases

​2. Write the checks

​3. Run it on the served setup

​4. Compare two setups

​5. Find the runs

​What you built

1. Write the cases

2. Write the checks

3. Run it on the served setup

4. Compare two setups

5. Find the runs

What you built