Skip to main content
This tutorial shows how DecimalAI tracks your agent’s skills — from initial discovery through activation detection and effectiveness analysis. By the end, you’ll know which skills are working and which need improvement.

​
The Scenario

You have a coding assistant with 3 skills: code-review, sql-optimizer, and deploy-checklist. You want to know:
  • Which skills are being used most?
  • Which version of code-review performs best?
  • Should you keep deploy-checklist or retire it?
1

Auto-Discover Skills

Your skills live in .claude/skills/ as SKILL.md files:
.claude/skills/
├── code-review/
│   └── SKILL.md
├── sql-optimizer/
│   └── SKILL.md
└── deploy-checklist/
    └── SKILL.md
Initialize DecimalAI — skills are discovered automatically:
import decimalai
decimalai.init(api_key="dai_sk_...")

from decimalai.openai_agents import install
install()
# ✓ Discovered 3 skills: code-review, sql-optimizer, deploy-checklist
# ✓ Pushed to platform (synced hashes)
# ✓ Registered in manifest as skill_registry
No configuration needed. The SDK scans standard directories, parses SKILL.md frontmatter, and registers the skill registry.
2

Run Your Agent

Your agent handles requests as normal. For each trace, the SDK detects which skills were activated by comparing the LLM’s rendered prompt against known skill bodies.
# User asks: "Review this PR for security issues"
# → Agent loads code-review skill
# → Trace records: active_skills: [{"name": "code-review", "hash": "a1b2c3"}]

# User asks: "Optimize this SQL query"
# → Agent loads sql-optimizer skill
# → Trace records: active_skills: [{"name": "sql-optimizer", "hash": "d4e5f6"}]
No extra code needed — activation detection is automatic across all framework integrations.
3

View Skill Analytics

After accumulating traces, check the Skills page in the dashboard. Each skill shows:
  • Activation count: How many traces used this skill
  • Pass rate: % of skill traces that passed evaluators
  • Effectiveness: Composite score
  • Trend: Improving, stable, or declining
Example data after 1 week:
SkillActivationsPass rateTrend
code-review14588%↗ trending up
sql-optimizer6792%→ stable
deploy-checklist333%↘ trending down
deploy-checklist is barely used and performs poorly. Consider retiring it or rewriting the instructions.
4

Improve a Skill

Edit the code-review SKILL.md to add better instructions:
---
name: code-review
description: Reviews code for security vulnerabilities and bugs
---

# Code Review (Updated)

## Step 1: Security Scan
Check for these specific vulnerabilities:
1. SQL injection — look for string concatenation in queries
2. XSS — check for unescaped user input in HTML templates
3. CSRF — verify token validation on state-changing endpoints
Restart your agent. The SDK detects the content hash changed, creates v2, and registers a new manifest version.
5

Compare Versions

After v2 collects traces, compare the two versions:
VersionTracesPass rateDelta
code-review v114588%—
code-review v25294%↗ +6%
The improved instructions are catching more security issues. DecimalAI’s comparison includes statistical significance testing — so you know the improvement is real, not noise.

​
Smart Routing (Bonus)

When you have many skills, use smart routing to select the best ones for each query:
from decimalai.skill_router import SkillRouter

router = SkillRouter(api_key="dai_sk_...", strategy="auto")

prompt_fragment = router.get_menu_prompt(
    query="Review this PR for security issues"
)
# → Returns code-review (high relevance + high pass rate)
# → Does NOT return deploy-checklist (low relevance + low pass rate)
Smart routing combines semantic similarity with historical effectiveness — skills that perform well on similar queries get boosted.
Key takeaway: DecimalAI turns skills from “static instructions” into observable, measurable, improvable components. You can see exactly which skills contribute to good outputs, which need improvement, and which should be retired — all backed by production data, not guesswork.

​
You’ve done it

Auto-discovered skills from .claude/skills/ in production traces
Inspected per-skill activation counts and effectiveness scores
Used smart routing to select the best skills for each query

​
Next Steps

Skills Guide

Versioning, forking, publishing to the public registry.

Skills Registry

Browse community skills ranked by SkillScore.