Skip to main content
A benchmark tells you how a skill does on test cases. An experiment tells you how it does on your agent’s real runs. DecimalAI splits the agent’s runs into two groups at random, grades both with the same quality score, and when enough runs are graded says whether one group did better, with an interval around the difference. There are two kinds: a skill’s is started from the skill list on the agent’s Overview, a prompt’s from its Prompt page:

Before you start

  • The agent takes its skills, and for a split its prompt, from DecimalAI. Runs take part through agent_config(), on any of its frameworks (which framework does what). Runs that don’t take part get the agent’s current prompt and skills and aren’t counted: runs from an older SDK, runs without a DecimalAI key, skills loaded from disk (a framework’s own skill loader), SkillRouter called directly, and CrewAI’s skill loader. For a split, your code must send the prompt agent_config() hands it (run.system, support.instructions, …), or the result is withheld (see When there is no verdict).
  • A quality score grades the agent’s runs. The outcome is one score, picked when you start: an LLM-judge evaluator or scores you send from your own eval tool. It must have graded the agent’s production runs in the last 30 days. The built-in checks (did the run finish, did it error) don’t count: they say whether a run finished, not whether it was right.
  • Enough graded runs. DecimalAI works out how many graded runs it needs and, from the last seven days, how long that takes. An experiment that would need more than 28 days can’t start, and the message says what would make it fit (a bigger share, a bigger change to detect). Fewer runs are graded when an evaluator samples traces or your plan’s judge quota runs out, which makes an experiment take longer.
  • Editor role. Starting or stopping one takes the Editor role (an API key needs admin permission). An agent runs one experiment at a time.

Measure a skill on live traffic

  1. In the skill list on the agent’s Overview, open the skill’s row menu and choose Measure on live traffic…. It is there for a skill the agent is sent (not one cut by the skill list’s caps). You can also measure a skill you are adding: in Review & apply, tick Measure it on live traffic under the staged add.
  2. Review & apply opens with the experiment. Choose:
    • Runs without it: 10% (the default), 20% or 50% of runs.
    • Change to detect: 5, 10 (the default) or 20 points of pass rate. A smaller change needs many more graded runs.
    • Graded by: the quality score, when more than one graded the agent.
  3. The panel shows how many graded runs it needs, about how many days that takes at the agent’s rate, the share of the agent’s runs that can take part, and when it ends on its own. Tick the confirmation (“about 1 in 10 of support-agent’s runs won’t get refund-tone, for about N days”) and apply.
A held-out run doesn’t get the skill anywhere: it isn’t on that run’s skill list, routing doesn’t pick it, and a load_skill for it is refused. A skill set to always include is held out the same way.

Split traffic between two prompts

  1. On the agent’s Prompt page, pick the version you want to test in the Prompt versions list, and choose Split traffic with this version… from its ⋯ menu.
  2. Choose Runs on the new prompt (10%, 20% or 50%, the default) and the change to detect, then confirm and apply as above.
One group gets the version the agent is served now, the other the version you chose. Testing a prompt you haven’t served yet takes two steps first, because applying a new prompt while the agent follows the latest version serves it straight away: pin the current version (with set_prompt_version, from the API), then save the new text as a draft and apply it. The new version is saved but not served, and you can split traffic with it.

While it runs

  • A banner above the skill list on the agent’s Overview, and on its Prompt page, says what is running, how many graded runs it has of the number it needs, about how long is left, and the date it ends by at the latest. The skill’s row, and the agent on Home, read Measuring.
  • Nothing is shown about how either group is doing until it ends. No pass rate, no difference, not on the banner, the agent’s Versions page or anywhere else. A difference checked again and again finds effects that aren’t there, so the result is read once, when the sample is in (or through the early-stop check below, which is built to be checked repeatedly). The agent’s Traces tab can list one group’s runs (each run shows its group, and the chip filters to it), but don’t read them as a score.
  • Each run waits for DecimalAI’s skill decision before calling the model: the same call it already makes, about 40-50 ms on the server plus the network.
Which group a run is in is decided at random by that routing call. A conversation stays in the group of its first message.

How it ends

DecimalAI checks running experiments every few minutes. One ends when:
  • it reaches its sample: the graded runs it planned for;
  • one group does clearly better or worse before that: an early-stop check that is valid however often it is checked, from 30 graded runs in each group;
  • it reaches its time limit, 28 days after it started;
  • you stop it: Stop… on the banner, then Review & apply;
  • what it measures goes away: the held-out skill comes off the agent, or, for a split, the prompt the agent is served changes. Saving a prompt edit while the agent follows the latest version, pinning or unpinning a version, restoring one or clearing the prompt all end a split; a save while the agent is pinned to the current version doesn’t. Other changes to the agent’s config leave an experiment running.
Then every run gets the agent’s current config again. Starting and ending each record a config version, and rolling back to a version that had one running doesn’t restart it.

Reading the result

The result is one of:
  • Helps or hurts: the interval around the difference is entirely above or below zero. The difference is the group with the skill minus the group without it, or the new prompt minus the current one.
  • No clear difference: the interval includes zero. That isn’t proof there’s no effect, only that this many runs couldn’t tell.
  • No verdict, with the reason (below).
An experiment that reached its sample uses a 95% interval (Newcombe’s score method), planned for 80% power at the change you chose to detect. One that ended early (the early-stop check, a stop, the time limit) uses the wider 99.5% interval that stays valid however often it was checked: it names a direction only if that check did, and otherwise reads no clear difference. Counting the early-stop check, at most 5.5% of experiments with no real effect end with a verdict. Where you see it:
  • Effect on this agent, in the agent’s skill list: a measured skill’s difference and interval, for example +6 pts (+1 to +11), with early stop when it ended early, and Measured on vN once the skill has moved to a newer version. Its row’s edge turns green (Helps here) or red (Hurts here). The skill’s own page adds each group’s graded and passed runs, under On your agents.
  • Home for 14 days: a skill that made the agent worse asks Remove it? and opens Review & apply with the removal staged; a prompt that did better asks Make vN the prompt? and opens the confirmation that makes it the prompt: it pins vN when the agent is pinned (a save would leave the pin in place), and otherwise restores vN as the newest version. Other results are listed without a question.
  • The agent’s Versions page: the versions that started and ended it carry the result, and a version’s pass rate leaves out the runs the experiment kept off that version.

When there is no verdict

A result is withheld, with the reason, when the experiment can’t be trusted:
  • a group has fewer than 30 graded runs;
  • the groups’ sizes don’t match the share, so the assignment can’t be trusted;
  • runs were traced or graded at different rates in the two groups;
  • the skill reached runs it was held out of (over 2% of them), for example from a copy on disk;
  • runs didn’t use the prompt their group was assigned (over 2%), for example code that sends its own prompt instead of the one agent_config() returns.
The banner says so as soon as one of these shows up while it runs.

From the API

  • GET /api/v1/agents/{agent_name}/experiments lists the running experiment and the last ended ones, with each skill’s lift; GET /api/v1/agents/{agent_name}/experiments/{experiment_id} returns one. While one runs these give each group’s run counts, never a rate.
  • Experiments start and stop through the agent’s setup changes: POST /api/v1/agents/{agent_name}/setup/apply with one of start_holdout (skill_id), start_prompt_split (version_number) or stop_experiment, plus share, detect_points, score and confirmed. POST /setup/preview returns the plan without starting anything.
  • GET /api/v1/traces?experiment=<id>&experiment_arm=<held_out|with|a|b> lists one experiment’s runs, or one group’s; each trace carries its experiment and group.
See Common endpoints for the rest of the agent and trace routes.