skillevaluation (eval.yaml)

skillevaluation is an open format and a reference runner for benchmarking an agent skill. You write test cases in eval.yaml, next to SKILL.md, and they answer one question:

Does this skill actually help an agent? By how much?

Each case runs twice, with the skill and without it. The difference is the skill’s lift (how much better the agent does with the skill than without). The format and runner don’t need DecimalAI. pip install "skillevaluation[runner]" runs a full benchmark on your machine, on your own API key, with no account. DecimalAI’s hosted runner imports the same judge and scoring code, and adds history, verified results and rankings. The package is Apache 2.0. Keeping the suite beside the skill means you version the cases with the skill, and decimalai skills pull brings them along.

The 60-second format

# eval.yaml, next to SKILL.md
cases:
  - name: classifies_pii
    prompt: "Classify these schema fields: email, ip_address, name, age."
    expectations:
      - "The response classifies email as PII"
      - "The response identifies ip_address as pseudonymous"

  - name: answers_in_json
    prompt: "Classify the field email. Reply with JSON only."
    validators:
      - cmd: "jq -e '.email == \"PII\"' response.txt"
        label: "email is PII"

  - name: ignores_weather
    prompt: "What's the weather in Paris?"
    expectations:
      - "The response answers the weather question directly"
      - "The response does not classify any data fields"

cases is the only top-level key. Unknown keys are rejected at every level, so a typo such as expectation: fails loudly instead of being dropped.

Case field	Required	What it holds
`name`	Yes	A non-empty string, unique in the file
`prompt`	Yes	The user message, the same in both runs
`expectations`	One grader	Claims in plain language, graded by an LLM judge over the whole transcript: actions, tool calls and the final response
`validators`	One grader	Shell commands, graded by exit code. Keys: `cmd` (required), `expect_exit_code` (default `0`), `label`
`setup`	No	Prepares the workspace: a list of shell commands, or a mapping with `files` (relative path to content) and `commands`. Files are written before any command runs
`max_turns`	No	A positive integer that caps the agent’s tool steps for this case
`should_trigger`	No	A boolean that marks a trigger case
`tags`	No	A list of strings, for your own organization
`description`	No	What the case tests. It has no effect on the run

Every case needs expectations or validators, or a should_trigger flag. Use a validator when a program can check the answer, and an expectation for a claim about meaning. Validators and setup commands run in the case’s workspace. A validator finds the agent’s final output in ./response.txt and $RESPONSE_TEXT. It passes when its exit code equals expect_exit_code. A timeout, a command that can’t start, or an exit code that is neither 0, 1 nor the declared one gives no verdict, and the case drops out of the lift rather than counting as a model failure. The old keys mode, user_goal, environment, simulator, policy_check and trials are rejected, with a message saying what to do instead.

What a run reports

Each case lands in one of five outcomes:

Outcome	Meaning
`flip_to_pass`	The skill rescued the agent: the win
`pass_kept`	Both runs passed, so this case didn’t separate them
`fail_kept`	Both runs failed, so the skill didn’t rescue it
`flip_to_fail`	The skill hurt
`error`	A transient failure prevented evaluation

The total is the skill’s lift, reported as something like “+34 pts pass rate, −46% agent turns” (illustrative). A run that mostly errored has no headline. When more than 25% of a run’s cases error, such as during a provider outage, the result is stamped error_dominated: true and the pass-rate difference is left empty. Re-run it. The hosted runner applies the same floor.

Trigger cases: `should_trigger`

Graded cases prove a skill helps when it’s loaded. Trigger cases prove it loads at the right times.

- name: fires_on_rewrite
  prompt: "clean up this commit line: 'Fixed Stuff.'"
  should_trigger: true
- name: near_miss_pr_description
  prompt: "write the PR description for this branch"
  should_trigger: false

A case with should_trigger and no grader is trigger-only. A case can also carry both, and then it’s graded for lift too. The local runner doesn’t score trigger accuracy. It keeps trigger-only cases out of the lift and reports how many in cases_skipped_trigger_only. A hosted run does score them: it builds a skill list with your skill among others, asks the model which it would use, and reports menu_selection_rate and false_fire_rate, plus router_recall for retrieval against the live skill index. When these fail, fix the description, not the body.

Run it locally (free)

Your API key, your machine, and nothing is sent to DecimalAI:

pip install "skillevaluation[runner]"
export ANTHROPIC_API_KEY=sk-ant-...

skillevaluation run ./my-skill --model claude-haiku-4-5

pip install "skillevaluation[runner]"
export OPENAI_API_KEY=sk-...

skillevaluation run ./my-skill --model gpt-4o-mini

pip install "skillevaluation[runner]"
export GEMINI_API_KEY=...

skillevaluation run ./my-skill --model gemini-3.5-flash

You get a table of differences and a results.json. Local runs are unlimited.

Command or flag	What it does
`skillevaluation validate ./my-skill`	Checks the file without running anything
`--adapter mock`	Tries the plumbing for free, with no model calls
`--adapter claude-code`	Runs through Claude Code instead of a single completion
`--fail-on-verdict fail --min-delta-pts 10`	Exits 1 for CI when the run fails or the lift is under 10 points
`--no-cache`	Re-runs the without-skill side, which is otherwise cached, so re-runs spend less

The default adapter makes one completion per run with the SKILL.md body in the prompt, so bundled references/ aren’t read.

`--runs`: average out the luck

Model behavior is probabilistic, and one run per case measures luck. skillevaluation run ./my-skill --runs 3 repeats the whole suite three times and averages by mean, up to 10 runs. The expected pass rate doesn’t change with the count: more runs only narrow the error bars. Repetition is a runner flag, so no case can weigh more than another. results.json holds one record per case, with rates averaged over every run. To inspect flakiness run by run, execute the suite once per run.

Push results to DecimalAI

Attach a local run to your skill’s Benchmark tab. It’s free and uses no quota. It needs the decimalai package and an API key, and you sync first, so the platform knows the skill and its cases:

pip install decimalai
export DECIMAL_API_KEY="dai_sk_..."

decimalai skills sync .               # the directory containing ./my-skill/
decimalai skills push results.json    # then attach the local run

Pushed runs are tagged unverified. They show on your skill’s own page and never feed registry rankings.

Verified runs (hosted)

A verified run is one DecimalAI’s runner executed, stamped with the model and date. Only verified runs feed registry cards, rankings and SkillScore.

decimalai skills benchmark ./my-skill/    # or --model, or --runs N

Or click Run benchmark on the skill’s page. Publishing doesn’t run one for you. Hosted runs are metered in cases: one case is both runs plus the judge, errored cases are refunded, and a single run holds at most 200 cases. See Pricing → Quota enforcement. The hosted runner doesn’t run your code, and that decides how you grade:

In the suite	Local run	Hosted run
`expectations`	Runs	Runs
`validators`, or `setup` commands	Runs	Refused. One anywhere refuses the whole suite: run it locally and push
`setup.files`	Written to the workspace	Not uploaded, so those cases run without the files

Two more rules apply to the verified number:

An expectation has to earn its place. It counts only when the with-skill side passes it more often than the without-skill side. One that doesn’t separate them is shown but not counted. If none count, the run has no lift.
A public lift needs at least 8 cases that count. Below that, the run is recorded and no lift appears on the skill’s public page.

	Local run	Pushed result	Verified run
Costs you	your LLM tokens	nothing	plan quota
Shows on your skill’s page	no	yes, tagged unverified	yes
Feeds rankings and registry cards	no	no	yes
Satisfies the publish check	no	yes	yes

Bring your own runner

The runner contract (spec/runner-contract.md), CONFORMANCE.md and the golden compatibility-tests/ fixtures ship inside the package. Any implementation that reproduces the fixtures conforms. The JSON Schemas load by short name:

from skillevaluation.resources import load_schema, list_schemas

list_schemas()                    # ['eval-yaml', 'judge-result', 'test-case-result', 'test-run-result']
schema = load_schema("eval-yaml") # parsed JSON Schema dict

Composing with agentversion

A manifest can record a run’s score under evaluation.gates[], so a lifecycle change can cite a suite’s verdict as evidence:

{
  "evaluation": {
    "gates": [
      {
        "name": "skillevaluation:gdpr-pii-classifier",
        "actual_score": 0.92,
        "threshold": 0.80,
        "passed": true,
        "evaluator_ref": "skillevaluation://abc123def456@v0.1.0",
        "ran_at": "2026-05-28T14:00:00Z"
      }
    ]
  }
}

name, actual_score, threshold, passed and ran_at are required. evaluator_ref is optional and names the suite: its content hash and the spec version it was written against.

Status

The package is at 0.8.0 on PyPI, in beta, and breaking changes are possible before 1.0. The eval.yaml shape on this page is schema rev 2. See the package changelog for each release.

Start here

Tutorial: a support agent

Use skills

Manage and measure

Publish a skill

More

skillevaluation (eval.yaml)

The 60-second format

What a run reports

Trigger cases: `should_trigger`

Run it locally (free)

`--runs`: average out the luck

Push results to DecimalAI

Verified runs (hosted)

Bring your own runner

Composing with agentversion

Status

​The 60-second format

​What a run reports

​Trigger cases: should_trigger

​Run it locally (free)

​--runs: average out the luck

​Push results to DecimalAI

​Verified runs (hosted)

​Bring your own runner

​Composing with agentversion

​Status

The 60-second format

What a run reports

Trigger cases: `should_trigger`

Run it locally (free)

`--runs`: average out the luck

Push results to DecimalAI

Verified runs (hosted)

Bring your own runner

Composing with agentversion

Status