Skills Observability

This tutorial shows how DecimalAI tracks your agent’s skills — from initial discovery through activation detection and effectiveness analysis. By the end, you’ll know which skills are working and which need improvement.

The Scenario

You have a coding assistant with 3 skills: code-review, sql-optimizer, and deploy-checklist. You want to know:

Which skills are being used most?
Which version of code-review performs best?
Should you keep deploy-checklist or retire it?

Auto-Discover Skills

Your skills live in .claude/skills/ as SKILL.md files:

.claude/skills/
├── code-review/
│   └── SKILL.md
├── sql-optimizer/
│   └── SKILL.md
└── deploy-checklist/
    └── SKILL.md

Initialize DecimalAI — skills are discovered automatically:

import decimalai
decimalai.init(api_key="dai_sk_...")

from decimalai.openai_agents import install
install()
# ✓ Discovered 3 skills: code-review, sql-optimizer, deploy-checklist
# ✓ Pushed to platform (synced hashes)
# ✓ Registered in manifest as skill_registry

No configuration needed. The SDK scans standard directories, parses SKILL.md frontmatter, and registers the skill registry.

You should now see: startup logs reporting the synced skill count, and all three skills listed on your installed skills page. If the log reports 0 skills, discovery found nothing — check the files live under a standard project-local directory like .claude/skills/ (personal/global directories are opt-in; see skill discovery) and that each SKILL.md has valid frontmatter.

Run Your Agent

Your agent handles requests as normal. For each trace, the SDK detects which skills were activated by comparing the LLM’s rendered prompt against known skill bodies.

# User asks: "Review this PR for security issues"
# → Agent loads code-review skill
# → Trace records: active_skills: [{"name": "code-review", "hash": "a1b2c3"}]

# User asks: "Optimize this SQL query"
# → Agent loads sql-optimizer skill
# → Trace records: active_skills: [{"name": "sql-optimizer", "hash": "d4e5f6"}]

No extra code needed — activation detection is automatic across all framework integrations.

You should now see: each trace’s detail view listing the skill(s) whose body reached the model under Active skills. If traces arrive with no active skills, the body probably never reached the model — a menu row alone (name + description) counts as offered, not used. See the silent no-ops list at the bottom of this page.

View Skill Analytics

After accumulating traces, check the Skills page in the dashboard. Each skill shows:

Activation count: How many traces used this skill
Pass rate: % of skill traces that passed evaluators
Effectiveness: Composite score
Trend: Improving, stable, or declining

Example data after 1 week:

Skill	Activations	Pass rate	Trend
`code-review`	145	88%	↗ trending up
`sql-optimizer`	67	92%	→ stable
`deploy-checklist`	3	33%	↘ trending down

deploy-checklist is barely used and performs poorly. Consider retiring it or rewriting the instructions.

Improve a Skill

Edit the code-review SKILL.md to add better instructions:

---
name: code-review
description: Reviews code for security vulnerabilities and bugs
---

# Code Review (Updated)

## Step 1: Security Scan
Check for these specific vulnerabilities:
1. SQL injection — look for string concatenation in queries
2. XSS — check for unescaped user input in HTML templates
3. CSRF — verify token validation on state-changing endpoints

Restart your agent. The SDK detects the content hash changed, creates v2, and registers a new manifest version.

Compare Versions

After v2 collects traces, compare the two versions:

Version	Traces	Pass rate	Delta
`code-review` v1	145	88%	—
`code-review` v2	52	94%	↗ +6%

The improved instructions are catching more security issues. DecimalAI’s comparison includes statistical significance testing — so you know the improvement is real, not noise.

Smart Routing (Bonus)

Use smart routing for large skill sets

When you have many skills, use smart routing to select the best ones for each query:

from decimalai.skill_router import SkillRouter

router = SkillRouter(api_key="dai_sk_...", strategy="auto")

prompt_fragment = router.get_menu_prompt(
    query="Review this PR for security issues"
)
# → Returns code-review (high relevance + high pass rate)
# → Does NOT return deploy-checklist (low relevance + low pass rate)

Smart routing combines semantic similarity with historical effectiveness — skills that perform well on similar queries get boosted.

Key takeaway: DecimalAI turns skills from “static instructions” into observable, measurable, improvable components. You can see exactly which skills contribute to good outputs, which need improvement, and which should be retired — all backed by production data, not guesswork.

If a skill never shows up

Skill plumbing fails quietly by design — the agent keeps running with fewer skills instead of crashing. These are the usual suspects: The SDK is deliberately fail-open: a misconfiguration degrades quietly instead of crashing your agent. These are the six places that bites, in the order people hit them.

1. A registry skill is never offered to your agent

Browsing or previewing a skill on the public registry doesn’t make it routable — you have to adopt it into your org first. Either way works — Use (router.use("name") or the Use button) creates a linked pointer that tracks the author’s updates, and Fork (router.fork("name"), or router.install("name") to also write it to disk, or the Fork button) creates an editable copy you own. The Skill Router offers both. If you use per-agent assignments, also assign it to the requesting agent. Until you Use or Fork it, the skill never appears in your agent’s menu, with no warning anywhere.

2. Only menu rows reach the model — never the skill body

By default build_prompt_fragment injects one-line menu rows (name + description). The skill’s actual instructions reach the model only if you pass inject_body=True to the SkillRouter (smart-routed queries), or enable the load_skill tool so the model can pull bodies on demand. Menu rows count as offered — usage panels show rung-labeled counts for them, and activation isn’t measurable for bare prompt-injection usage. If a skill “isn’t working,” check whether its body ever actually reached the model.

3. The load_skill tool never appears

The on-demand body loader is opt-in: pass enable_skill_loader=True to install() on the openai_agents or pydantic_ai adapters. On the anthropic adapter there is no tool loop — enable_skill_loader=True there injects the skill menu into system (the offered rung only); add init(inject_skill_body=True) if you want full bodies delivered, and note enable_load_skill_tool is accepted but dormant. Kill-switch: init(load_skill_tool=False) or DECIMALAI_LOAD_SKILL_TOOL=0.

4. An integration flag whose package is missing

decimalai.init(langchain=True) (and every other framework flag) logs a warning and continues untraced when the framework package isn’t importable — your agent runs, nothing is recorded. The same applies to DECIMAL_AUTO_TRACE (auto-init warns and skips) and to the raw-provider flags (openai=True etc.), which soft-skip when the matching OpenInference instrumentor is absent. Fix: for a framework flag, install the matching extra, e.g. pip install "decimalai[langchain]"; for a raw-provider flag, install the instrumentor package the startup warning names, e.g. pip install openinference-instrumentation-openai (the [openai] extra covers only the provider SDK, not the instrumentor). Then re-check startup logs.

5. Skill injection skips callable instructions and prebuilt prompts

Adapters only inject skills into prompt shapes they can safely rewrite. On openai_agents, an agent whose instructions is a user-supplied callable is left untouched; on langchain, a prebuilt PromptValue (or any unrecognized prompt shape) passes through unchanged. The agent runs normally — with zero skills injected and no error raised.

6. The impact report says 0 traces (or 'first run')

The regression check needs two things before it can say anything real: a baseline manifest (recorded automatically on the Action’s first run, or by decimalai.init() running in production) and ingested traces to measure blast radius against. A fresh workspace legitimately reports “first run — no baseline” and then near-zero affected traces. That’s honesty, not breakage — reports gain weight over days as production trace volume accumulates.

You’ve done it

Auto-discovered skills from .claude/skills/ in production traces

Inspected per-skill activation counts and effectiveness scores

Used smart routing to select the best skills for each query

Next Steps

Skills Guide

Versioning, forking, publishing to the public registry.

Skills Registry

Browse community skills ranked by SkillScore.

Get Started

Foundation

Ship Safely

Improve with skills

Trust & Safety

Data & Training

Operate

Core Concepts

Platform

Skills Observability

The Scenario

Smart Routing (Bonus)

If a skill never shows up

You’ve done it

Next Steps

Skills Guide

Skills Registry

​The Scenario

​Smart Routing (Bonus)

​If a skill never shows up

​You’ve done it

​Next Steps

Skills Guide

Skills Registry

The Scenario

Smart Routing (Bonus)

If a skill never shows up

You’ve done it

Next Steps