> ## Documentation Index
> Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# skillevaluation (eval.yaml)

> The open A/B benchmark format + runner for agent skills. Run it locally for free on your own API key; push results to DecimalAI; verified runs power the registry.

`skillevaluation` is an open specification **and reference runner** for benchmarking an agent skill via declarative A/B test cases. The format lives next to `SKILL.md` as `eval.yaml` and answers a single question:

> **Does this skill actually help an agent? By how much?**

The spec and runner are independent of DecimalAI — `pip install "skillevaluation[runner]"` executes a full A/B benchmark on your machine, on your own API key, with no account. DecimalAI is a conforming hosted runner of the same spec (it imports the same judge and validator code) and adds what a local run can't: history, verified results, rankings, and distribution.

<CardGroup cols={2}>
  <Card title="Package" icon="box" href="https://pypi.org/project/skillevaluation/">
    `skillevaluation` on PyPI — Apache 2.0
  </Card>

  <Card title="Schema" icon="code" href="https://pypi.org/project/skillevaluation/">
    JSON Schema for `eval.yaml` — ships in the wheel: `from skillevaluation.resources import load_schema`
  </Card>
</CardGroup>

## Why a separate spec

`SKILL.md` tells an agent **how** to do something. `eval.yaml` tells DecimalAI (or any conforming runner) **how to measure** whether it's working. Keeping the two side-by-side on disk means:

* Skill authors version eval cases with the skill itself
* A skill pulled via `decimalai skills pull` brings its eval suite with it
* A regression is detectable against the same cases that proved the skill worked

## The 60-second format

```yaml theme={null}
# eval.yaml — lives next to SKILL.md
cases:
  - name: tracks_with_id
    prompt: "Classify these schema fields: email, ip_address, name, age."
    expectations:
      - "The response classifies email as PII"
      - "The response identifies ip_address as pseudonymous"
    validators:
      - cmd: "jq -e '.email.category == \"PII\"' output.json"

  - name: ignores_weather
    prompt: "What's the weather in Paris?"
    expectations:
      - "The response answers the weather question directly"
      - "The response does not invoke the GDPR classifier"
```

Each case runs **twice** — once with the skill loaded into the agent's manifest, once without. The runner classifies each case into one of five outcomes:

| Outcome        | Meaning                                                  |
| -------------- | -------------------------------------------------------- |
| `flip_to_pass` | The skill rescued the agent — the "win"                  |
| `pass_kept`    | Both arms passed; skill didn't discriminate on this case |
| `fail_kept`    | Both arms failed; skill didn't rescue                    |
| `flip_to_fail` | The skill HURT — regression marker                       |
| `error`        | A transient failure prevented evaluation                 |

The aggregate — the skill's lift (the with-vs-without improvement) — is what becomes the registry headline, e.g. "+34 pts pass rate, −46% agent turns" (illustrative).

## Assertion kinds

Each case can mix two assertion kinds:

* **`expectations`** — natural-language claims, graded by an LLM judge
* **`validators`** — shell commands, graded by exit code

A case MUST have at least one of either. Use validators when a precise structural check is possible (cheaper, deterministic); use expectations for genuinely semantic claims. (From spec 0.3.0 there is one exception: a *trigger-only* case — `should_trigger` with no graders — see below.)

## Run it locally (free)

The open-source package ships a complete reference runner. Your own API key, your machine — nothing is sent to DecimalAI:

<CodeGroup>
  ```bash Anthropic theme={null}
  pip install "skillevaluation[runner]"
  export ANTHROPIC_API_KEY=sk-ant-...

  skillevaluation run ./my-skill --model claude-haiku-4-5
  ```

  ```bash OpenAI theme={null}
  pip install "skillevaluation[runner]"
  export OPENAI_API_KEY=sk-...

  skillevaluation run ./my-skill --model gpt-4o-mini
  ```

  ```bash Google theme={null}
  pip install "skillevaluation[runner]"
  export GEMINI_API_KEY=...

  skillevaluation run ./my-skill --model gemini-3.5-flash
  ```
</CodeGroup>

Each case executes twice (with the skill / without), both arms are graded with the same expectations + validators, and you get the delta table plus a `results.json` conforming to the open wire schema. The without-skill baseline is cached locally, so re-runs while you iterate on `SKILL.md` cost half. Gate it in CI with `--fail-on-verdict fail --min-delta-pts 10`, or dry-run the plumbing for free with `--adapter mock`.

Local runs are **unlimited and unmetered** — iterate as much as you like.

## `--runs`: average out the luck

Model behavior is probabilistic — one run per case measures luck. `skillevaluation run … --runs 3` re-runs the **whole suite** three times, uniformly, and averages the per-case results by **mean**: each (case, run) record enters the aggregate at equal weight, so the headline pass-rate is a mean whose expected value does **not** depend on the run count — more runs only narrow the error bars. Use it on any suite whose verdict you intend to act on. Repetition is a runner flag, not a per-case field: there is no way for one case to be weighted more heavily than another (that was the flaw in the old per-case `trials`/pass^k knob, retired in 0.6.0 — see ADR-0007). Flakiness stays visible in the per-run records rather than collapsing the headline.

## Spec 0.3.0 additions

<Note>
  Available from **skillevaluation 0.3.0** on PyPI. These are additive: existing suites parse unchanged. But strict 0.2.x parsers *reject* suites that use the new fields, so pin `skillevaluation>=0.3.0` before adopting them.
</Note>

### Trigger cases: `should_trigger`

Graded cases prove a skill helps *when it's loaded*. Trigger cases prove it *loads at the right times* — the failure mode graded cases can't see. Mark a case with `should_trigger`:

```yaml theme={null}
- prompt: "clean up this commit line: 'Fixed Stuff.'"    # this SHOULD surface the skill
  should_trigger: true
- prompt: "write the PR description for this branch"     # near-miss: should NOT
  should_trigger: false
```

A case with `should_trigger` and **no** expectations/validators is a *trigger-only* case (exempt from the at-least-one-grader rule); a case can also carry both and be graded for lift *and* trigger.

**What the open-source runner does with them:** it records each `should_trigger` case and discloses `cases_skipped_trigger_only` in the results document — trigger-only cases carry no lift evidence, so they're kept out of the A/B divisor. The runner does **not** score trigger *accuracy* locally.

**Measuring trigger accuracy is a hosted feature.** The menu-selection simulation — build a skill menu (your name + description alongside distractor rows), ask the model per prompt which skill it would use, and roll up `menu_selection_rate` (should-fire recall) + `false_fire_rate` (should-NOT-fire) — runs on the DecimalAI hosted runner, together with `router_recall` (the retrieval stage against the live skill index, which can only be measured server-side). This moved out of the open-source runner in **0.6.0** (ADR-0007): the OSS spec keeps the `should_trigger` boolean, the trigger-only grader exemption, and the `cases_skipped_trigger_only` disclosure; the simulation contract is platform policy.

When trigger cases fail on the hosted runner, fix the **description**, not the body — the description is what both readers (retrieval and menu) see. See [Authoring Skills](/guides/authoring-skills#step-2-write-the-description-as-the-trigger).

### Error-dominated runs: no headline from an outage

When more than 25% of a run's cases errored (a provider outage, a rate-limit storm), the result is stamped `error_dominated: true` and the headline pass-rate delta is nulled. A lift number computed from the few surviving cases isn't a measurement — re-run instead of shipping it. The hosted runner applies the same floor, so a local run and a verified run can never disagree about what counts as valid.

### `setup.files`: declarative workspace files

Cases that need files in the workspace can declare them directly, instead of echo-ing them via shell commands:

```yaml theme={null}
- name: fixes_port
  prompt: "Set the database port in config.json to 5432."
  setup:
    files:
      config.json: '{"port": 8080}'
    commands:
      - "git init -q"
```

Files are written before any setup command runs. The legacy list-of-commands `setup:` form still parses.

## Push results to DecimalAI

Attach a local run to your skill's Benchmark tab (free — it's a JSON upload, no quota consumed):

```bash theme={null}
decimalai skills push results.json
```

Pushed runs are tagged **unverified**: they show on your skill's own page but never feed registry rankings — self-reported numbers can't poison the leaderboard.

## Verified runs (hosted)

A **verified** run is one the DecimalAI runner executed — same open-spec judge and validators (the platform literally imports them from the `skillevaluation` package), but in a trusted environment, stamped with model + date. Only verified runs feed registry cards, rankings, and SkillScore.

Two ways to get one:

```bash theme={null}
decimalai skills benchmark ./my-skill/    # metered against your plan's monthly case quota
```

…or **publish the skill** — publishing automatically triggers a verification run, free and quota-exempt.

Hosted runs are metered in **cases** (one metered case = one eval case executed, both arms + judge included; errored cases refunded). See [Pricing → Quota enforcement](/pricing#quota-enforcement).

|                                 | Local run       | Pushed result         | Verified run                    |
| ------------------------------- | --------------- | --------------------- | ------------------------------- |
| Costs you                       | your LLM tokens | nothing               | plan quota (or free on publish) |
| Shows on your skill page        | —               | ✓ (tagged unverified) | ✓                               |
| Feeds rankings / registry cards | —               | —                     | ✓                               |

## Bring your own runner

The full runner contract (`spec/runner-contract.md`) and the golden `compatibility-tests/` fixtures ship inside the [`skillevaluation` package](https://pypi.org/project/skillevaluation/) — load them with `from skillevaluation.resources import load_schema`. Any implementation that reproduces those fixtures is conforming; the reference runner itself passes the suite — compare against it.

## Composing with [agentversion](/concepts/versioning)

A `skillevaluation` run produces a numeric score. That score can be recorded on an AgentVersion manifest's `evaluation.gates[]` via the `skillevaluation://` URI scheme:

```json theme={null}
{
  "evaluation": {
    "gates": [
      {
        "name": "skillevaluation:gdpr-pii-classifier",
        "actual_score": 0.92,
        "threshold": 0.80,
        "passed": true,
        "evaluator_ref": "skillevaluation://abc123def456@v0.1.0",
        "ran_at": "2026-05-28T14:00:00Z"
      }
    ]
  }
}
```

Each gate object carries the fields below:

<ResponseField name="name" type="string" required>
  Human-readable identifier for the gate, e.g. `skillevaluation:gdpr-pii-classifier`.
</ResponseField>

<ResponseField name="actual_score" type="number" required>
  The score the run produced, `0.0`–`1.0` (pass rate of the with-skill arm).
</ResponseField>

<ResponseField name="threshold" type="number" required>
  The minimum `actual_score` required for the gate to pass.
</ResponseField>

<ResponseField name="passed" type="boolean" required>
  Whether `actual_score` met `threshold`.
</ResponseField>

<ResponseField name="evaluator_ref" type="string" required>
  A `skillevaluation://` URI pinning the exact eval suite + version that produced the score, e.g. `skillevaluation://abc123def456@v0.1.0`.
</ResponseField>

<ResponseField name="ran_at" type="string">
  ISO 8601 timestamp of when the run completed.
</ResponseField>

This lets a lifecycle transition cite a specific eval suite's verdict as evidence — "this manifest reached production because the gdpr-pii-classifier benchmark passed at 92%."

## Status

* The package is pre-stable (breaking changes possible before v1.0). By spec/feature line: the **0.2** releases added the reference runner + CLI; **0.3.0** shipped spec 0.3.0 (trigger cases, the error-dominated floor, `setup.files` — see above); **0.4.0** added `skillevaluation.safety` — the same deterministic static scanner behind the registry's Tier-1 safety gate — and the `skillevaluation scan` CLI (text / JSON / SARIF output); **0.6.0** is schema rev 2 (ADR-0007): one execution contract, per-case `mode`/`trials`/`simulator`/`policy_check` removed, and runner-level `--runs N` (mean-averaged) replacing pass^k. All of these are on PyPI; check the [package changelog](https://pypi.org/project/skillevaluation/) for the current release. The wheel ships `spec/` + `schemas/` (`from skillevaluation.resources import load_schema`).
* DecimalAI's hosted runner consumes the same package — judge and validator behavior is shared by construction, not by copy
* Conformance suite + JSON Schemas ship inside the [`skillevaluation` package](https://pypi.org/project/skillevaluation/) (`from skillevaluation.resources import load_schema`)
* Want a different language implementation? The package's `CONFORMANCE.md` + golden in/out fixtures define conformance — anything that reproduces them conforms