> ## Documentation Index
> Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments: measure on live traffic

> Find out whether a skill helps an agent, or whether a new prompt does better, from the agent's own runs: hold a skill back from a share of runs, or split runs between two prompts.

A benchmark tells you how a skill does on test cases. An experiment tells you how it does on
**your agent's real runs**. DecimalAI splits the agent's runs into two groups at random, grades
both with the same quality score, and when enough runs are graded says whether one group did
better, with an interval around the difference.

There are two kinds: a skill's is started from the skill list on the agent's **Overview**, a prompt's from its **Prompt** page:

| | What differs between the groups | The question it answers |
| - | - | - |
| **Measure on live traffic** (a holdout) | One group doesn't get the skill | Does this skill make the agent's answers better? |
| **Split traffic** (a prompt split) | The groups get two prompt versions | Does the new prompt do better than the current one? |

## Before you start

* **The agent takes its skills, and for a split its prompt, from DecimalAI.** Runs take part
  through `agent_config()`, on any of its frameworks ([which framework does what](https://docs.decimal.ai/sdk/python/frameworks.md#capability-comparison)). Runs that don't take part get the agent's
  current prompt and skills and aren't counted: runs from an older SDK, runs without a DecimalAI
  key, skills loaded from disk (a framework's own skill loader), `SkillRouter` called directly, and CrewAI's
  skill loader. For a split, your code must send the prompt `agent_config()`
  hands it (`run.system`, `support.instructions`, …), or the result is withheld (see
  [When there is no verdict](#when-there-is-no-verdict)).
* **A quality score grades the agent's runs.** The outcome is one score, picked when you start: an
  LLM-judge [evaluator](https://docs.decimal.ai/guides/evaluations.md) or scores you send from your own eval tool. It must have
  graded the agent's production runs in the last 30 days. The built-in checks (did the run finish,
  did it error) don't count: they say whether a run finished, not whether it was right.
* **Enough graded runs.** DecimalAI works out how many graded runs it needs and, from the last
  seven days, how long that takes. An experiment that would need more than 28 days can't start,
  and the message says what would make it fit (a bigger share, a bigger change to detect). Fewer
  runs are graded when an evaluator samples traces or your plan's judge quota runs out, which makes
  an experiment take longer.
* **Editor role.** Starting or stopping one takes the Editor role (an API key needs admin
  permission). An agent runs one experiment at a time.

## Measure a skill on live traffic

1. In the skill list on the agent's **Overview**, open the skill's row menu and choose **Measure
   on live traffic…**. It is there for a skill the agent is sent (not one cut by the skill list's caps).
   You can also measure a skill you are adding: in **Review & apply**, tick **Measure it on live
   traffic** under the staged add.
2. **Review & apply** opens with the experiment. Choose:
   * **Runs without it**: 10% (the default), 20% or 50% of runs.
   * **Change to detect**: 5, 10 (the default) or 20 points of pass rate. A smaller change needs
     many more graded runs.
   * **Graded by**: the quality score, when more than one graded the agent.
3. The panel shows how many graded runs it needs, about how many days that takes at the agent's
   rate, the share of the agent's runs that can take part, and when it ends on its own. Tick the
   confirmation ("about 1 in 10 of support-agent's runs won't get refund-tone, for about N days")
   and apply.

A held-out run doesn't get the skill anywhere: it isn't on that run's skill list, routing doesn't
pick it, and a `load_skill` for it is refused. A skill set to always include is held out the same
way.

## Split traffic between two prompts

1. On the agent's **Prompt** page, pick the version you want to test in the **Prompt versions**
   list, and choose **Split traffic with this version…** from its **⋯** menu.
2. Choose **Runs on the new prompt** (10%, 20% or 50%, the default) and the change to detect, then
   confirm and apply as above.

One group gets the version the agent is served now, the other the version you chose. Testing a
prompt you haven't served yet takes two steps first, because applying a new prompt while the
agent follows the latest version serves it straight away: **pin** the current version (with
`set_prompt_version`, [from the API](https://docs.decimal.ai/guides/agent-setup-versions.md#from-the-api)), then save the new
text as a draft and apply it. The new version is saved but not served, and you can split traffic
with it.

## While it runs

* A banner above the skill list on the agent's **Overview**, and on its **Prompt** page, says what is running, how many graded runs it has of the number
  it needs, about how long is left, and the date it ends by at the latest. The skill's row, and the
  agent on Home, read **Measuring**.
* **Nothing is shown about how either group is doing until it ends.** No pass rate, no difference,
  not on the banner, the agent's Versions page or anywhere else. A difference checked again and
  again finds effects that aren't there, so the result is read once, when the sample is in (or
  through the early-stop check below, which is built to be checked repeatedly). The agent's
  **Traces** tab can list one group's runs (each run shows its group, and the chip filters to it),
  but don't read them as a score.
* Each run waits for DecimalAI's skill decision before calling the model: the same call it already
  makes, about 40-50 ms on the server plus the network.

Which group a run is in is decided at random by that routing call. A conversation stays in the group
of its first message.

## How it ends

DecimalAI checks running experiments every few minutes. One ends when:

* **it reaches its sample**: the graded runs it planned for;
* **one group does clearly better or worse** before that: an early-stop check that is valid however
  often it is checked, from 30 graded runs in each group;
* **it reaches its time limit**, 28 days after it started;
* **you stop it**: **Stop…** on the banner, then Review & apply;
* **what it measures goes away**: the held-out skill comes off the agent, or, for a split, the
  prompt the agent is served changes. Saving a prompt edit while the agent follows the latest
  version, pinning or unpinning a version, restoring one or clearing the prompt all end a split; a
  save while the agent is pinned to the current version doesn't. Other changes to the agent's config
  leave an experiment running.

Then every run gets the agent's current config again. Starting and ending each record a
[config version](https://docs.decimal.ai/guides/agent-setup-versions.md), and rolling back to a version that had one running
doesn't restart it.

## Reading the result

The result is one of:

* **Helps** or **hurts**: the interval around the difference is entirely above or below zero. The
  difference is the group with the skill minus the group without it, or the new prompt minus the
  current one.
* **No clear difference**: the interval includes zero. That isn't proof there's no effect, only
  that this many runs couldn't tell.
* **No verdict**, with the reason (below).

An experiment that reached its sample uses a 95% interval (Newcombe's score method), planned for
80% power at the change you chose to detect. One that ended early (the early-stop check, a stop, the
time limit) uses the wider 99.5% interval that stays valid however often it was checked: it names a
direction only if that check did, and otherwise reads no clear difference. Counting the early-stop
check, at most 5.5% of experiments with no real effect end with a verdict.

Where you see it:

* **Effect on this agent**, in the agent's skill list: a measured skill's difference and
  interval, for example `+6 pts (+1 to +11)`, with **early stop** when it ended early, and
  **Measured on vN** once the skill has moved to a newer version. Its row's edge turns green
  (*Helps here*) or red (*Hurts here*). The skill's own page adds each group's graded and passed
  runs, under [On your agents](https://docs.decimal.ai/guides/registry.md#on-your-agents).
* **Home** for 14 days: a skill that made the agent worse asks **Remove it?** and opens Review &
  apply with the removal staged; a prompt that did better asks **Make vN the prompt?** and opens the
  confirmation that makes it the prompt: it pins vN when the agent is pinned (a save would leave
  the pin in place), and otherwise restores vN as the newest version. Other results are listed
  without a question.
* The agent's **Versions** page: the versions that started and ended it carry the result, and a
  version's pass rate leaves out the runs the experiment kept off that version.

### When there is no verdict

A result is withheld, with the reason, when the experiment can't be trusted:

* a group has fewer than 30 graded runs;
* the groups' sizes don't match the share, so the assignment can't be trusted;
* runs were traced or graded at different rates in the two groups;
* the skill reached runs it was held out of (over 2% of them), for example from a copy on disk;
* runs didn't use the prompt their group was assigned (over 2%), for example code that sends its
  own prompt instead of the one `agent_config()` returns.

The banner says so as soon as one of these shows up while it runs.

## From the API

* `GET /api/v1/agents/{agent_name}/experiments` lists the running experiment and the last ended
  ones, with each skill's lift; `GET /api/v1/agents/{agent_name}/experiments/{experiment_id}`
  returns one. While one runs these give each group's run counts, never a rate.
* Experiments start and stop through the agent's setup changes:
  `POST /api/v1/agents/{agent_name}/setup/apply` with one of `start_holdout` (`skill_id`),
  `start_prompt_split` (`version_number`) or `stop_experiment`, plus `share`, `detect_points`,
  `score` and `confirmed`. `POST /setup/preview` returns the plan without starting anything.
* `GET /api/v1/traces?experiment=<id>&experiment_arm=<held_out|with|a|b>` lists one experiment's
  runs, or one group's; each trace carries its `experiment` and group.

See [Common endpoints](https://docs.decimal.ai/api-reference/common-endpoints.md) for the rest of the agent and trace routes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.