Skip to main content
basepointer · September 2026

​
The finding

We publish a lift number for agent skills: the pass rate with the skill installed, minus the pass rate without it, on the same case set. Forty-five public skills in our registry carry that number on two different models, each measured over at least eight cases. We compared them. All forty-five moved. Not one held its number. Thirty-four fell, eleven rose, mean change −17.2 points, median −18.2. The largest drop was markdown-semantic-wrap, 95.8 → 34.8; the largest rise was a-set-of-resources-formatter, 40.0 → 81.8. The tempting headline is “lift decays with newer models.” We are not making that claim, because our own data refutes it — a quarter of these skills got better. The honest finding is narrower, and more useful:
A skill’s lift is not a property of the skill. It is a property of the skill, the model, the harness and the case set, together. Report it without pinning those and you have published a number nobody can reproduce — including you, three weeks later.

​
The cleanest example

The obvious objection is that two runs might not be comparable. It is a fair objection, and it applies to most of our pairs — see Limits. So here is the finding on a pair where it cannot apply. postgresql-conventions ran the same 24 cases on both models: Same skill, same suite, same grader. One model upgrade, and four fifths of the measured benefit is gone. Nothing about the skill changed, and nothing about it was wrong. The number was not portable, and nothing on its scorecard said so. Three other pairs graded the identical case set on both models and behave the same way: svg-ui-panel-conventions (91.3 → 73.9 on 23 cases), thesis-dissertation-compliance (50.0 → 41.7 on 24 cases) and setup-husky-pre-commit (65.2 → 34.8 on 23 cases, of which the second run headlines 20 after calibration).

​
Prior work, and what this adds

This is not the first measurement of skills across models, and it should not be read as one.
  • Tessl re-ran 21 skills over 100 paired scenarios, held fixed, on Kimi K2.5 and then K2.6 (June 2026). Their aggregate uplift barely moved (+17.05 → +17.20 points) while individual skills moved by double digits and four became redundant on the newer model. Their closing line is this page’s thesis: “Rerun your evals when the model changes. Baselines move unevenly; some skills become redundant, some keep paying. You cannot tell which is which without running the evaluation.” What follows is a replication of that finding on a different vendor’s model family, with more skills and an aggregate that did not stay flat.
  • SkillsBench (arXiv 2602.12670) measures 87 tasks across 18 model–harness configurations and finds that “smaller models with Skills can match larger models without them” — what a skill adds depends on what the model already does without it.
  • The Regression Tax (arXiv 2607.22520) runs nearly 6,000 trials across three model–harness stacks and finds that “the best performing skills outperform others primarily by regressing less, not by gaining more.”
  • Do LLM-Generated Skills Make Better AI Data Scientists? (arXiv 2607.07504) ablates skills over 56 tasks, nine model configurations and three providers — 7,560 runs — and finds “no reliable improvement from full generated skills over No-Skill prompting.”
  • Demystifying Agent Skills (arXiv 2608.14036) normalises 8,135 trial records to isolate “the effects of representation, outcome annotation, retrieval difficulty, and cross-framework robustness of skills.”
  • ACES (arXiv 2608.20614, NVIDIA) defines Skill Lift as “the target skill’s added value for a fixed task, harness, workspace, and scorer.” Pinning the baseline is the textbook control, and it is now written down.
  • Google ran one skill, gemini-api-dev, over the same 117 prompts on Gemini 2.5, 3.0 and 3.1 (March 2026) — the same design as this page, on the same model family, for a single skill. The lift depended on the version: the newer models started from “a low baseline without it (6.8% for both 3.0 Pro and Flash, 28% for 3.1 Pro)”, while “the older 2.5 series of models also benefit, but nowhere near as much.”
  • Tessl again, twice: ~850 scenarios across 500 skills on GLM 5.1 and Sonnet 4.6 (May 2026) found per-skill lift correlated at r = 0.72 between the two models and that on “does this skill help?” they “agreed 88.5% of the time” — so roughly one skill in nine changes its verdict with the model; and the same ~850 scenarios on Opus 4.7 and 4.8 (May 2026) “finished level” on the headline while 4.8 “depends on the skill slightly less”.
Every one of those is a larger study than this one, and two of them already varied the model version with the skill held fixed. Read this page as a replication of that result on an open registry’s own numbers, with one extension: each row here is a lift a public registry published, each carries the agentversion manifest that pins what it was measured against, and four of the pairs graded the identical case rows on both models — so for those four, the model change is the only thing that moved. No public registry we can find stamps the model on the number it shows, and that is the practice this page argues for.

​
Method

The corpus is ours, and it is synthetic. These runs were produced by our own evaluation fleet under pinned and identical conditions — not by third-party production traffic. That makes it a measuring instrument, which is what this report uses it as. It would not support a claim about adoption, and we are not making one. Each pair is the most recent sound benchmark run per (skill, model). A run is sound when its suite survived calibration and produced a headline lift. A pair enters the figures above when both of its runs headline at least 8 cases — the floor the registry itself applies before it prints a lift. Three further pairs ship in the dataset and are not in the figures: their first-model run headlined one, two or three cases. They are also the two most extreme movements in the file (100.0 → 4.5 and 0.0 → 59.1), which is the reason the floor exists.

​
Limits, stated before you find them

  • 41 of the 45 pairs did not grade the same case set on the two models, because suites get re-cut over time. That is exactly why the worked example above is one of the four that did. Case-set drift moves a pass rate on its own, so treat the population figure as directional and the four clean pairs as the evidence. “Same case set” here means the same case rows, identified by id in skill_lift_pairs.csv’s case_set_hash — not the same case count, and not the same prompt text, which in this corpus is a shared placeholder on 1,736 of the 2,038 case rows and would have made unrelated skills look identical.
  • Two models, one family. We have not shown this holds across vendors.
  • Our headline floor is 8 cases. An 8-case suite can move 12.5 points on one case. Small-n pairs are in the population figure and are not in the worked example.
  • We publish these scores ourselves. That is a conflict, and stating it is cheaper than having it pointed out. Everything below ships so you can recompute it.

​
What ships with this

  • The raw per-case rows for all 48 pairs — the 45 in the figures and the 3 below the floor — not a summary table.
  • An agentversion manifest pinning model, harness and case set for every row. That is the whole point: it is what makes a lift number comparable, and it is why these pairs could be compared at all.
  • A reproduction that runs in seconds from the shipped rows, with no database.
All of it is public, in the skillevaluation repository: Reproduce the figures from the shipped rows:
Or re-pull the rows themselves from the production registry, which is how the CSVs were made (read-only; the script opens a read-only transaction and issues SELECTs, and needs database access we hold, not a public endpoint):
The rows were pulled 2026-08-28 and re-pulled on 2026-09-05 and 2026-09-13; all 96 were identical each time. Nothing here is recomputed for the report — every lift is read back from the run the platform published. As a check, recompute.py also re-derives each lift from the per-case outcomes alone: 95 of the 96 rows agree exactly, and the one that does not (progressive-tax-bracket-math on gemini-3.6-flash, published 80.0, derived 81.8) differs because its headline was computed over a calibration-gated subset of its cases, which the CSV shows as headline_cases below total_cases.
The per-case rows carry verdicts, not transcripts. Every run behind these pairs completed in July, and the platform’s payload sweep (30 days when they ran; 180 days since 2026-09-05) has since nulled the stored model outputs — the outputs_pruned column says so on all 96 rows. What survives per case is the A/B outcome (pass_kept, flip_to_pass, flip_to_fail, fail_kept), which is what the pass rates are computed from, so the numbers are fully reproducible from the data here. The text each arm produced is not recoverable.