skillevaluation (eval.yaml)

skillevaluation is an open specification and reference runner for benchmarking an agent skill via declarative A/B test cases. The format lives next to SKILL.md as eval.yaml and answers a single question:

Does this skill actually help an agent? By how much?

The spec and runner are independent of DecimalAI — pip install "skillevaluation[runner]" executes a full A/B benchmark on your machine, on your own API key, with no account. DecimalAI is a conforming hosted runner of the same spec (it imports the same judge and validator code) and adds what a local run can’t: history, verified results, rankings, and distribution.

Package

skillevaluation on PyPI — Apache 2.0

Schema

JSON Schema for eval.yaml — ships in the wheel: from skillevaluation.resources import load_schema

Why a separate spec

SKILL.md tells an agent how to do something. eval.yaml tells DecimalAI (or any conforming runner) how to measure whether it’s working. Keeping the two side-by-side on disk means:

Skill authors version eval cases with the skill itself
A skill pulled via decimalai skills pull brings its eval suite with it
A regression is detectable against the same cases that proved the skill worked

The 60-second format

# eval.yaml — lives next to SKILL.md
cases:
  - name: tracks_with_id
    prompt: "Classify these schema fields: email, ip_address, name, age."
    expectations:
      - "The response classifies email as PII"
      - "The response identifies ip_address as pseudonymous"
    validators:
      - cmd: "jq -e '.email.category == \"PII\"' output.json"

  - name: ignores_weather
    prompt: "What's the weather in Paris?"
    expectations:
      - "The response answers the weather question directly"
      - "The response does not invoke the GDPR classifier"

Each case runs twice — once with the skill loaded into the agent’s manifest, once without. The runner classifies each case into one of five outcomes:

Outcome	Meaning
`flip_to_pass`	The skill rescued the agent — the “win”
`pass_kept`	Both arms passed; skill didn’t discriminate on this case
`fail_kept`	Both arms failed; skill didn’t rescue
`flip_to_fail`	The skill HURT — regression marker
`error`	A transient failure prevented evaluation

The aggregate — the skill’s lift (the with-vs-without improvement) — is what becomes the registry headline, e.g. “+34 pts pass rate, −46% agent turns” (illustrative).

Assertion kinds

Each case can mix two assertion kinds:

expectations — natural-language claims, graded by an LLM judge
validators — shell commands, graded by exit code

A case MUST have at least one of either. Use validators when a precise structural check is possible (cheaper, deterministic); use expectations for genuinely semantic claims. (From spec 0.3.0 there is one exception: a trigger-only case — should_trigger with no graders — see below.)

Run it locally (free)

The open-source package ships a complete reference runner. Your own API key, your machine — nothing is sent to DecimalAI:

pip install "skillevaluation[runner]"
export ANTHROPIC_API_KEY=sk-ant-...

skillevaluation run ./my-skill --model claude-haiku-4-5

pip install "skillevaluation[runner]"
export OPENAI_API_KEY=sk-...

skillevaluation run ./my-skill --model gpt-4o-mini

pip install "skillevaluation[runner]"
export GEMINI_API_KEY=...

skillevaluation run ./my-skill --model gemini-3.5-flash

Each case executes twice (with the skill / without), both arms are graded with the same expectations + validators, and you get the delta table plus a results.json conforming to the open wire schema. The without-skill baseline is cached locally, so re-runs while you iterate on SKILL.md cost half. Gate it in CI with --fail-on-verdict fail --min-delta-pts 10, or dry-run the plumbing for free with --adapter mock. Local runs are unlimited and unmetered — iterate as much as you like.

`trials`: pass every time, not best-of-k

Model behavior is probabilistic — one run per case measures luck. trials: 3 on a case runs it three independent times per arm, and the case passes only if it passes all trials (pass^k), not if the best of three passed. The aggregate reports the resulting pass_at_k. Use it on any suite whose verdict you intend to act on; consistency is the production bar. (Executed by the reference runner since 0.2.4.)

Spec 0.3.0 additions

Available from skillevaluation 0.3.0 on PyPI. These are additive: existing suites parse unchanged. But strict 0.2.x parsers reject suites that use the new fields, so pin skillevaluation>=0.3.0 before adopting them.

Trigger cases: `should_trigger`

Graded cases prove a skill helps when it’s loaded. Trigger cases prove it loads at the right times — the failure mode graded cases can’t see. Mark a case with should_trigger:

- prompt: "clean up this commit line: 'Fixed Stuff.'"    # this SHOULD surface the skill
  should_trigger: true
- prompt: "write the PR description for this branch"     # near-miss: should NOT
  should_trigger: false

A case with should_trigger and no expectations/validators is a trigger-only case (exempt from the at-least-one-grader rule); a case can also carry both and be graded for lift and trigger. What the trigger stage measures: the runner builds a skill menu — your skill’s name + description row alongside distractor rows — and asks the model, per prompt, which skill it would use. That’s the same menu selection step a production router runs. Results roll up to:

menu_selection_rate — should-fire cases where the model picked the skill (its trigger recall)
false_fire_rate — should-NOT-fire cases where the model picked it anyway

(router_recall — the retrieval stage against a live skill index — is always null in local results: it can only be measured server-side, and an unmeasured stage reports null, never a fake zero.) Two flags control the rail:

--fail-on-trigger 0.8,0.2 — CI gate: exit 1 when menu_selection_rate < 0.8 or false_fire_rate > 0.2. If trigger cases exist but went unmeasured (skipped or errored), the gate fails — an unproven floor never passes.
--skip-trigger-cases — don’t execute trigger cases, only disclose them in the results document.

Optionally, --distractors DIR points at a directory of sibling skill folders whose name + description become the menu’s distractor rows — hard negatives from your own catalog instead of the builtin generic pool. When trigger cases fail, fix the description, not the body — the description is what both readers (retrieval and menu) see. See Authoring Skills.

Error-dominated runs: no headline from an outage

When more than 25% of a run’s cases errored (a provider outage, a rate-limit storm), the result is stamped error_dominated: true and the headline pass-rate delta is nulled. A lift number computed from the few surviving cases isn’t a measurement — re-run instead of shipping it. The hosted runner applies the same floor, so a local run and a verified run can never disagree about what counts as valid.

`setup.files`: declarative workspace files

Cases that need files in the workspace can declare them directly, instead of echo-ing them via shell commands:

- name: fixes_port
  prompt: "Set the database port in config.json to 5432."
  setup:
    files:
      config.json: '{"port": 8080}'
    commands:
      - "git init -q"

Files are written before any setup command runs. The legacy list-of-commands setup: form still parses.

Push results to DecimalAI

Attach a local run to your skill’s Benchmark tab (free — it’s a JSON upload, no quota consumed):

decimalai skills push results.json

Pushed runs are tagged unverified: they show on your skill’s own page but never feed registry rankings — self-reported numbers can’t poison the leaderboard.

Verified runs (hosted)

A verified run is one the DecimalAI runner executed — same open-spec judge and validators (the platform literally imports them from the skillevaluation package), but in a trusted environment, stamped with model + date. Only verified runs feed registry cards, rankings, and SkillScore. Two ways to get one:

decimalai skills benchmark ./my-skill/    # metered against your plan's monthly case quota

…or publish the skill — publishing automatically triggers a verification run, free and quota-exempt. Hosted runs are metered in cases (one metered case = one eval case executed, both arms + judge included; errored cases refunded). See Pricing → Quota enforcement.

	Local run	Pushed result	Verified run
Costs you	your LLM tokens	nothing	plan quota (or free on publish)
Shows on your skill page	—	✓ (tagged unverified)	✓
Feeds rankings / registry cards	—	—	✓

Bring your own runner

The full runner contract (spec/runner-contract.md) and the golden compatibility-tests/ fixtures ship inside the skillevaluation package — load them with from skillevaluation.resources import load_schema. Any implementation that reproduces those fixtures is conforming; the reference runner itself passes the suite — compare against it.

Composing with agentversion

A skillevaluation run produces a numeric score. That score can be recorded on an AgentVersion manifest’s evaluation.gates[] via the skillevaluation:// URI scheme:

{
  "evaluation": {
    "gates": [
      {
        "name": "skillevaluation:gdpr-pii-classifier",
        "actual_score": 0.92,
        "threshold": 0.80,
        "passed": true,
        "evaluator_ref": "skillevaluation://abc123def456@v0.1.0",
        "ran_at": "2026-05-28T14:00:00Z"
      }
    ]
  }
}

Each gate object carries the fields below:

name

string

required

Human-readable identifier for the gate, e.g. skillevaluation:gdpr-pii-classifier.

actual_score

number

required

The score the run produced, 0.0–1.0 (pass rate of the with-skill arm).

threshold

number

required

The minimum actual_score required for the gate to pass.

passed

boolean

required

Whether actual_score met threshold.

evaluator_ref

string

required

A skillevaluation:// URI pinning the exact eval suite + version that produced the score, e.g. skillevaluation://abc123def456@v0.1.0.

ran_at

string

ISO 8601 timestamp of when the run completed.

This lets a lifecycle transition cite a specific eval suite’s verdict as evidence — “this manifest reached production because the gdpr-pii-classifier benchmark passed at 92%.”

Status

The package is pre-stable (breaking changes possible before v1.0). By spec/feature line: the 0.2 releases added the reference runner + CLI and trials/pass^k execution; 0.3.0 shipped spec 0.3.0 (trigger cases, the error-dominated floor, setup.files — see above); 0.4.0 added skillevaluation.safety — the same deterministic static scanner behind the registry’s Tier-1 safety gate — and the skillevaluation scan CLI (text / JSON / SARIF output). All of these are on PyPI; check the package changelog for the current release. The wheel ships spec/ + schemas/ (from skillevaluation.resources import load_schema).
DecimalAI’s hosted runner consumes the same package — judge and validator behavior is shared by construction, not by copy
Conformance suite + JSON Schemas ship inside the skillevaluation package (from skillevaluation.resources import load_schema)
Want a different language implementation? The package’s CONFORMANCE.md + golden in/out fixtures define conformance — anything that reproduces them conforms

Package

Schema

​Why a separate spec

​The 60-second format

​Assertion kinds

​Run it locally (free)

​trials: pass every time, not best-of-k

​Spec 0.3.0 additions

​Trigger cases: should_trigger

​Error-dominated runs: no headline from an outage

​setup.files: declarative workspace files

​Push results to DecimalAI

​Verified runs (hosted)

​Bring your own runner

​Composing with agentversion

​Status

Why a separate spec

The 60-second format

Assertion kinds

Run it locally (free)

`trials`: pass every time, not best-of-k

Spec 0.3.0 additions

Trigger cases: `should_trigger`

Error-dominated runs: no headline from an outage

`setup.files`: declarative workspace files

Push results to DecimalAI

Verified runs (hosted)

Bring your own runner

Composing with agentversion

Status