Skills Observability

This tutorial shows how DecimalAI tracks your agent’s skills — from initial discovery through activation detection and effectiveness analysis. By the end, you’ll know which skills are being used and how they are scoring. Every code block below runs on a bare pip install decimalai — no agent framework required. Skills observability is framework-neutral: the SDK reads SKILL.md files and records activations, and none of that depends on which agent library (if any) you use.

The Scenario

You have a coding assistant with 3 skills: code-review, sql-optimizer, and deploy-checklist. You want to know:

Which skills are being used most?
Which version of code-review performs best?
Should you keep deploy-checklist or retire it?

Prerequisites. pip install decimalai and an API key in DECIMAL_API_KEY (see Settings → API keys). Every script below starts with the same two lines:

API_KEY  = os.environ["DECIMAL_API_KEY"]
BASE_URL = os.environ.get("DECIMAL_BASE_URL", "https://api.decimal.ai")

Leave DECIMAL_BASE_URL unset unless you run a self-hosted backend.

Discover and sync your skills

Your skills live in .claude/skills/ as SKILL.md files:

.claude/skills/
├── code-review/
│   └── SKILL.md
├── sql-optimizer/
│   └── SKILL.md
└── deploy-checklist/
    └── SKILL.md

discover_skills() scans the standard directories and parses the frontmatter; sync_to_platform() pushes what it found:

import os

import decimalai
from decimalai.skills import discover_skills, sync_to_platform

API_KEY = os.environ["DECIMAL_API_KEY"]
BASE_URL = os.environ.get("DECIMAL_BASE_URL", "https://api.decimal.ai")

decimalai.init(api_key=API_KEY, base_url=BASE_URL)

for s in discover_skills():
    print(f"{s['name']:<18} {s['hash']}")

print(sync_to_platform(api_key=API_KEY, base_url=BASE_URL))

code-review        sha256:591fd62a8e92e320d2eb072fb17f3cf9543c3eb89a51e03cab8a1a78ec61a1cf
deploy-checklist   sha256:b8760a49e8374d49a664942fe56aa3e44f5c8c4f34e5f5dc9aaabbb89699acab
sql-optimizer      sha256:2b927d019e0db3daeec929482de352e8c7e2b36cb6ecfdc271a28adb19b65644
{'status': 'ok', 'created': 3, 'updated': 0, 'unchanged': 0, 'pulled': 0, 'failures': [], 'actions': None}

A skill’s identity is the SHA-256 of its SKILL.md body — sha256: plus 64 hex characters, 71 characters in all. That string is what shows up as skill_hash on every trace and as versions_seen in the analytics below, so it is worth recognising on sight.

sync_to_platform() does not read init()’s configuration. It takes its own api_key and its own base_url (which defaults to https://api.decimal.ai). Pass both, or a self-hosted setup will silently sync to the wrong host.

Using an agent framework? The framework adapters do this sync for you as a side effect of instrumenting — decimalai.init(openai_agents=True), or the equivalent from decimalai.openai_agents import instrument. Each adapter is an extra, so that path needs pip install "decimalai[openai-agents]" (likewise [langchain], [pydantic-ai], [adk], [llamaindex], [claude-agent-sdk]) — calling instrument() without the extra raises ImportError. Nothing else on this page needs a framework, which is why the code above calls sync_to_platform() directly instead. Note also that decimalai.init(api_key=...) alone does not sync anything: the sync lives inside each adapter’s instrument(), or in the explicit call above.The adapter entry point was called install() in 0.10.0 and earlier and was renamed to instrument() in 0.10.2 — same arguments, same behaviour — because install had come to mean adding a skill to your workspace. install() still works and emits a DeprecationWarning. See Vocabulary.

You should now see: a created/updated/unchanged count summing to the number of skills you have, and all three skills listed on your installed skills page. If discover_skills() printed nothing, discovery found nothing — check the files live under a standard project-local directory like .claude/skills/ (personal/global directories are opt-in, via discover_skills(include_global=True); see skill discovery), that each SKILL.md has valid frontmatter, and that each body is at least 10 characters — a shorter body is skipped without an error.

Run your agent

Skill activations are recorded per trace. The script below stands in for your agent loop: one trace per turn, each one naming the skill that fired and the LLM call it wrapped.

import os

import decimalai
from decimalai.skills import discover_skills

API_KEY = os.environ["DECIMAL_API_KEY"]
BASE_URL = os.environ.get("DECIMAL_BASE_URL", "https://api.decimal.ai")

decimalai.init(api_key=API_KEY, base_url=BASE_URL)
hashes = {s["name"]: s["hash"] for s in discover_skills()}

# One tuple per agent turn: (skill the agent loaded, user message, reply).
TURNS = (
    [("code-review", "Review this PR for security issues",
      "Line 42 concatenates user input into the query - use a bound parameter.")] * 6
    + [("sql-optimizer", "Optimize this slow SQL query",
        "The planner is doing a seq scan; add an index on orders.customer_id.")] * 4
    + [("deploy-checklist", "What should I check before deploying?",
        "")] * 2
)

for skill, question, answer in TURNS:
    with decimalai.start_trace(agent_name="coding-assistant"):
        decimalai.log_skill_activation(name=skill, hash=hashes[skill])
        decimalai.log_llm_call(
            model="gpt-4o-mini",
            provider="openai",
            input=[{"role": "user", "content": question}],
            output={"content": answer},
            input_tokens=900,
            output_tokens=120,
            latency_ms=1800,
        )

decimalai.flush()
print(f"sent {len(TURNS)} traces")

sent 12 traces

Three API shapes are easy to get wrong. All three raise loudly rather than degrading, so you will know at once — here is what each one says:

start_trace is the context manager; trace is the decorator. with decimalai.trace(...) gives TypeError: 'function' object does not support the context manager protocol.
log_llm_call takes model=, not model_name=. model_name is the field name on the wire, not the argument: TypeError: log_llm_call() got an unexpected keyword argument 'model_name'.
output= takes a dict, not a string. output="..." gives ValidationError: Input should be a valid dictionary. Use output={"content": "..."}.

Calling log_skill_activation outside an active trace raises DecimalConfigError: No active trace — it never silently drops the activation.

Why the explicit log_skill_activation. Nothing infers an activation for you. Comparing the rendered prompt against known skill bodies shows the skill was put in front of the model, which is the delivered rung — the adapters record that automatically, and they leave active_skills empty rather than guess. Activation is observed only when the model asks, by calling load_skill; with no router in the loop there is no such event, so you name the skill yourself. This is the one channel that carries a sha256:-prefixed body hash, because it is the one place a version is actually known.

A tight loop will hit the rate limit. On the free plan that is 60 requests a minute, shared by ingest and reads. The SDK retries on 429 and prints Rate limited (429). Retrying in 1.0s — that is the retry working, not a failure, and all 12 traces land. A curl you run immediately afterwards has no such retry and may come back {"detail": "Rate limit exceeded. Retry after 1s", "plan": "free", "limit": 60}; wait a second and repeat it.

You should now see: each trace’s detail view listing the skill you declared under Active skills. On a framework adapter, expect Active skills to be empty unless the model called load_skill — that is correct, not a gap. Look at Delivered instead to confirm the body reached the model; a menu row alone (name + description) counts only as offered. See the silent no-ops list at the bottom of this page.

Read the analytics

Per-skill numbers are on the agent’s Versions page, in the folded Skills section: app.decimal.ai/agents/coding-assistant/versions. The table’s columns are Skill, Status, Usages, Pass Rate, Version, and Trend.The same numbers come back from the API, which is the quickest way to check your run landed:

curl -s -H "Authorization: Bearer $DECIMAL_API_KEY" \
  "$DECIMAL_BASE_URL/api/v1/skills/analytics/metrics?agent_name=coding-assistant&window_days=30"

{
  "agent_name": "coding-assistant",
  "window_days": 30,
  "skills": [
    {
      "name": "code-review",
      "activation_count": 6,
      "versions_seen": ["sha256:591fd62a8e92e320d2eb072fb17f3cf9543c3eb89a51e03cab8a1a78ec61a1cf"],
      "metrics": {"success_rate": 1.0, "avg_eval_score": 0.959, "evaluated_count": 30, "pass_rate": 1.0}
    },
    {
      "name": "deploy-checklist",
      "activation_count": 2,
      "versions_seen": ["sha256:b8760a49e8374d49a664942fe56aa3e44f5c8c4f34e5f5dc9aaabbb89699acab"],
      "metrics": {"success_rate": 1.0, "avg_eval_score": 0.759, "evaluated_count": 10, "pass_rate": 0.8}
    },
    {
      "name": "sql-optimizer",
      "activation_count": 4,
      "versions_seen": ["sha256:2b927d019e0db3daeec929482de352e8c7e2b36cb6ecfdc271a28adb19b65644"],
      "metrics": {"success_rate": 1.0, "avg_eval_score": 0.959, "evaluated_count": 20, "pass_rate": 1.0}
    }
  ]
}

Skill	Usages	Pass rate
`code-review`	6	100%
`sql-optimizer`	4	100%
`deploy-checklist`	2	80%

You get a pass rate without configuring an evaluator. Five deterministic checks — completion, has_output, tool_compliance, latency, token_efficiency — are computed on the server at ingest from fields already in your payload, so they run no matter how the trace arrived. deploy-checklist scores 80% here for a concrete reason: its two turns log an empty output, so has_output fails on both (2 of its 10 eval rows). Add your own evaluators when you want the pass rate to mean something more than “the trace was well-formed”.

evaluated_count is not a count of traces, and its unit changes. Above it reads 30 for 6 activations, because it counts eval rows — 5 built-ins × 6 traces. It counts those rows only until the first Skill Rater report exists for the skill, and then switches to counting reports. On this workspace the same 12 code-review activations reported evaluated_count: 66 before the Rater ran and evaluated_count: 4 after it rated four traces. Since the dashboard hides a pass rate until this number reaches 5, a skill can lose its displayed pass rate at the moment more analysis arrives. Read activation_count when you want to know how much data you have.

The Trend column does not move yet. Neither /skills/analytics/metrics nor /skills/analytics/leaderboard returns a trend field — you can see that in the response above — so the column renders the neutral → for every skill that has usages. A live trend is computed, but only inside smart routing (see below), where it is returned per skill as trend. Judge a skill by comparing versions (step 5), not by that arrow.

Improve a skill

Edit the code-review SKILL.md to add better instructions:

---
name: code-review
description: Reviews code for security vulnerabilities and bugs
---

# Code Review

## Step 1: Security Scan
Check for these specific vulnerabilities:
1. SQL injection — look for string concatenation in queries
2. XSS — check for unescaped user input in HTML templates
3. CSRF — verify token validation on state-changing endpoints

## Step 2: Report
Quote the offending line and name the fix. Do not report style issues.

Re-run the step 1 script. The body hash changes, so the platform records a new version:

code-review        sha256:d4af14e97244f9e4d43be7b2115db46a71dfc94a0ed8b36e374adf998f0b35c7
deploy-checklist   sha256:b8760a49e8374d49a664942fe56aa3e44f5c8c4f34e5f5dc9aaabbb89699acab
sql-optimizer      sha256:2b927d019e0db3daeec929482de352e8c7e2b36cb6ecfdc271a28adb19b65644
{'status': 'ok', 'created': 0, 'updated': 1, 'unchanged': 2, 'pulled': 0, 'failures': [], 'actions': None}

Now re-run the step 2 script so v2 collects its own traces. It re-reads the hashes from disk, so the new activations carry the new hash automatically.

Two spellings of the same digest. GET /skills/code-review/versions reports content_hash as the bare 64-hex digest. Traces and the analytics endpoints use the sha256:-prefixed form. The comparison in the next step matches on the trace-side value, so take the hashes from versions_seen, not from content_hash.

Compare versions

Ask for both hashes side by side:

curl -s -G -H "Authorization: Bearer $DECIMAL_API_KEY" \
  --data-urlencode "skill_name=code-review" \
  --data-urlencode "baseline_hash=sha256:591fd62a...a1cf" \
  --data-urlencode "candidate_hash=sha256:d4af14e9...b5c7" \
  --data-urlencode "agent_name=coding-assistant" \
  "$DECIMAL_BASE_URL/api/v1/skills/analytics/compare"

{
  "skill_name": "code-review",
  "baseline":  {"hash": "sha256:591fd62a...", "trace_count": 6, "avg_score": 0.966},
  "candidate": {"hash": "sha256:d4af14e9...", "trace_count": 6, "avg_score": 0.959},
  "delta": {"avg_score": -0.007},
  "verdict": "insufficient_data",
  "statistical_significance": {
    "test": null, "p_value": null, "significant": null,
    "computed": false, "reason": "insufficient_sample",
    "alpha": 0.05, "sample_sizes": [36, 30], "min_sample_per_arm": 30
  }
}

Read this as: v2 ran on 6 traces and moved the average eval score by −0.007 — noise, on a sample this small, and nowhere near a claim you could defend. Keep v2 running until the counts are in the hundreds.

Read computed before you read significant. They answer different questions, and conflating them is the mistake this response is shaped to prevent.

computed: false — no test ran. significant is null (not false), and reason says why: here, insufficient_sample, because each arm needs at least min_sample_per_arm evaluated rows. The verdict is insufficient_data, which is not a finding about your skill.
computed: true — a two-sided Fisher’s exact test ran on the pass/fail counts. Now significant is a real true/false against alpha, and verdict is improved, regressed, or no_significant_change — that last one being a genuine finding: a test ran and found nothing.

Until you clear the sample floor, delta.avg_score and the two trace_counts are the real content of this response. Judge the size of the change yourself, and keep v2 running.

sample_sizes counts eval rows, not traces, and the two sides need not be symmetric: [36, 30] above is 6 baseline traces × 6 rows against 6 candidate traces × 5 rows. The extra baseline row is a one-off model-compatibility eval recorded when the manifest was first registered — it has nothing to do with the skill.

Smart Routing

Smart routing takes a query and returns the skills to put in front of the model:

from decimalai.skill_router import SkillRouter

router = SkillRouter(api_key=API_KEY, base_url=BASE_URL, strategy="auto")

prompt_fragment = router.get_menu_prompt(query="Review this PR for security issues")

What it returns depends on how many skills your workspace has, and the difference is large enough that you should know which side of the line you are on. The response carries the answer in strategy.

Small workspace: everything is returned, the query only reorders

While your eligible skills fit under the menu caps — 30 rows, and roughly 1,500 tokens of descriptions — routing short-circuits to the full menu. Every skill comes back; the query changes only the order. Ranking here is lexical overlap against each skill’s name and description (a name hit counts double), not embeddings. Against the 3-skill workspace from this tutorial, POST /api/v1/skills/route:

Query	`strategy`	`query_ranked`	Skills returned, in order
”Review this PR for security issues”	`full_menu`	`true`	code-review, sql-optimizer, deploy-checklist
”Optimize this slow SQL query”	`full_menu`	`true`	sql-optimizer, deploy-checklist, code-review
”What should I check before deploying to production”	`full_menu`	`true`	deploy-checklist, sql-optimizer, code-review
”How do I bake sourdough bread”	`full_menu`	`false`	sql-optimizer, deploy-checklist, code-review

The right skill leads in each of the first three. The fourth query matches no skill name or description, so query_ranked is false and the order falls back to effectiveness-then-recency — query_ranked is the field to check, because an unranked menu and a ranked one are otherwise indistinguishable. If the field is absent from the response altogether, you are talking to a backend older than 2026-08-15, where the full-menu path ignored the query entirely and returned a byte-identical list for every question. The prompt fragment get_menu_prompt() hands to your model says the same thing — every skill is in it, and the query decides who is at the top:

QUERY: Review this PR for security issues
## Available Skills

| Skill | Description |
|-------|-------------|
| code-review | Reviews code for security vulnerabilities and bugs |
| sql-optimizer | Rewrites slow SQL queries into faster equivalents |
| deploy-checklist | Pre-deploy checklist for production releases |

To use a skill, state: "I'll use the [skill-name] skill for this task."
--------------------------------------------------
QUERY: How do I bake sourdough bread
## Available Skills

| Skill | Description |
|-------|-------------|
| sql-optimizer | Rewrites slow SQL queries into faster equivalents |
| deploy-checklist | Pre-deploy checklist for production releases |
| code-review | Reviews code for security vulnerabilities and bugs |

To use a skill, state: "I'll use the [skill-name] skill for this task."

Under the caps, routing never withholds a skill. All three came back every time, including the poorly-performing deploy-checklist and including the sourdough query. Low relevance and a low pass rate push a skill down the list; they do not remove it. If you are relying on routing to keep a bad skill out of the prompt, at this size it will not.

Larger workspace: the set is genuinely narrowed

Past the caps, strategy becomes smart_routing and retrieval runs for real. Growing the same workspace to 31 skills and re-asking:

Query	`strategy`	Skills returned
”Review this PR for security issues”	`smart_routing`	code-review
”Optimize this slow SQL query”	`smart_routing`	sql-optimizer
”How do I bake sourdough bread”	`full_menu` (fallback)	30 rows, `query_ranked: false`

Each skill now comes back with its own relevance, performance, score, activation_count, and trend. A query that retrieves nothing falls back to the unranked menu and is flagged.

Watch degraded and degraded_reason. Retrieval has two legs — dense (embeddings) and sparse (full-text). If embeddings are unavailable the router still ranks lexically but sets "degraded": true with "degraded_reason": "embedding_unavailable"; "no_retrieval_candidates" means neither leg matched and you are looking at the plain menu. The runs above were measured on a backend with no embedding provider configured, so they are the lexical-only path — the ordering shown is the floor, not the ceiling.

Key takeaway: DecimalAI turns skills from “static instructions” into observable, measurable components. Activation counts and version identity are exact from the first trace. Quality signals are only as good as the evaluators behind them — the built-ins tell you a trace was well-formed, and everything past that you configure yourself.

If a skill never shows up

Skill plumbing fails quietly by design — the agent keeps running with fewer skills instead of crashing. These are the usual suspects: The SDK is deliberately fail-open: a misconfiguration degrades quietly instead of crashing your agent. These are the six places that bites, in the order people hit them.

1. A registry skill is never offered to your agent

Browsing or previewing a skill on the public registry doesn’t make it routable — you have to adopt it into your org first. Either way works — Install (the Install button, or router.use("name") in the SDK) creates a linked pointer that tracks the author’s updates, and Fork (the Fork a copy button, or router.fork("name"), or router.install("name") to also write it to disk) creates an editable copy you own. The Skill Router offers both. If you use per-agent assignments, also assign it to the requesting agent. Until you install or fork it, the skill never appears in your agent’s menu, with no warning anywhere.

2. Only menu rows reach the model — never the skill body

By default build_prompt_fragment injects one-line menu rows (name + description). The skill’s actual instructions reach the model only if you pass inject_body=True to the SkillRouter (smart-routed queries), or enable the load_skill tool so the model can pull bodies on demand. Menu rows count as offered — usage panels show rung-labeled counts for them, and activation isn’t measurable for bare prompt-injection usage. If a skill “isn’t working,” check whether its body ever actually reached the model.

3. The load_skill tool never appears

The on-demand body loader is opt-in: pass enable_skill_loader=True to instrument() on the openai_agents or pydantic_ai adapters. On the anthropic adapter there is no tool loop — enable_skill_loader=True there injects the skill menu into system (the offered rung only); full bodies are injected by default once the loader is on (0.12.0+; init(inject_skill_body=False) turns that off), and note enable_load_skill_tool is accepted but dormant. Kill-switch: init(load_skill_tool=False) or DECIMALAI_LOAD_SKILL_TOOL=0.

4. An integration flag whose package is missing

decimalai.init(langchain=True) (and every other framework flag) logs a warning and continues untraced when the framework package isn’t importable — your agent runs, nothing is recorded. The same applies to DECIMAL_AUTO_TRACE (auto-init warns and skips) and to the raw-provider flags (openai=True etc.), which soft-skip when the matching OpenInference instrumentor is absent. Fix: for a framework flag, install the matching extra, e.g. pip install "decimalai[langchain]"; for a raw-provider flag, install the instrumentor package the startup warning names, e.g. pip install openinference-instrumentation-openai (the [openai] extra covers only the provider SDK, not the instrumentor). Then re-check startup logs.

5. Skill injection skips callable instructions and prebuilt prompts

Adapters only inject skills into prompt shapes they can safely rewrite. On openai_agents, an agent whose instructions is a user-supplied callable is left untouched; on langchain, a prebuilt PromptValue (or any unrecognized prompt shape) passes through unchanged. The agent runs normally — with zero skills injected and no error raised.

6. The impact report says 0 traces (or 'first run')

The regression check needs two things before it can say anything real: a baseline manifest (recorded automatically on the Action’s first run, or by decimalai.init() running in production) and ingested traces to measure blast radius against. A fresh workspace legitimately reports “first run — no baseline” and then near-zero affected traces. That’s honesty, not breakage — reports gain weight over days as production trace volume accumulates.

You’ve done it

Auto-discovered skills from .claude/skills/ and synced them with no framework installed

Recorded activations and read per-skill usage counts and pass rates back

Versioned a skill by editing it, and compared the two versions on production traces

Measured what smart routing returns at your workspace’s size

Next Steps

Skills Guide

Versioning, forking, publishing to the public registry.

Skills Registry

Browse published skills ranked by SkillScore.

Get Started

Foundation

Ship Safely

Improve with skills

Trust & Safety

Data & Training

Operate

Research

Core Concepts

Platform

Skills Observability

The Scenario

Smart Routing

Small workspace: everything is returned, the query only reorders

Larger workspace: the set is genuinely narrowed

If a skill never shows up

You’ve done it

Next Steps

Skills Guide

Skills Registry

​The Scenario

​Smart Routing

​Small workspace: everything is returned, the query only reorders

​Larger workspace: the set is genuinely narrowed

​If a skill never shows up

​You’ve done it

​Next Steps

Skills Guide

Skills Registry

The Scenario

Smart Routing

Small workspace: everything is returned, the query only reorders

Larger workspace: the set is genuinely narrowed

If a skill never shows up

You’ve done it

Next Steps