Skip to main content
Tracing an agent and running evaluations tells you how it behaved. DecimalAI starts somewhere else: it tracks your agent’s structural identity — and uses that to catch regressions, measure skills, and keep training data valid. We call this manifest-aware agent change management. It’s a different category from observability.

​
The Problem Every Team Hits

You ship an agent. It has tools, prompts, a model, maybe some skills. Then a developer:
  1. Renames search_docs → search_knowledge_base (tool registry change)
  2. Updates the system prompt to include a new persona (prompt stack change)
  3. Removes compare_competitors (tool removal)
Three things break, in three different places — and a trace view shows you none of them until after deploy.
Illustrative figures. Actual counts depend on your trace volume and how the change touches each surface. See the canonical Impact Report example for a representative breakdown.

​
The Structural Differentiator: Manifest-Aware Detection

The usual way to detect a regression is to run an eval suite. That requires writing eval cases — in our experience most teams either have no eval suite or have one that has gone stale — maintaining them, running the agent in CI (slow, costly, non-deterministic), and paying for LLM-graded judgment. DecimalAI detects regressions by diffing the manifest and querying the trace store. No eval cases. No agent execution. No LLM API keys. Cost per check: <$0.001.

​
Regression testing, reframed

“Regression testing” usually means eval-based regression testing: keep a golden set of cases, re-grade them after every change, and watch the scores. It answers “did quality drop?” — but only for the cases you thought to write, only after you run the agent, and only if the eval set is still current. DecimalAI’s regression check answers a different, earlier question: “what did this change structurally touch, and which production traces are affected?” It runs on the manifest diff before deploy, with no agent execution and no graded judgment. Severity is reported as HIGH / MEDIUM / LOW IMPACT per trace — a representative diff lands at 247 HIGH / 501 MEDIUM / 1,254 LOW across 2,002 traces. The two are complementary, not competing. Eval-based testing measures quality on a curated set; manifest-aware regression measures blast radius on real traffic. In our experience most teams have the second gap, not the first — which is why DecimalAI leads with it.
IMPACT (HIGH / MEDIUM / LOW) answers “was this trace structurally touched?” — a separate axis from the compatibility verdict (keep / repair / flag / replay / drop), which answers “what should I do with the trace for training?”

​
What DecimalAI does

The capability surface, stated plainly — including what isn’t in it.

​
Where DecimalAI fits

Everything above sits on the structural identity layer:
  • Catch regressions without writing eval cases (GitHub Action on every PR)
  • Measure skills with production effectiveness data (pass rates, activation trends)
  • Keep training data valid when the agent changes (auto-classify + repair)
It is deliberately narrow. DecimalAI is not a general-purpose prompt hub and does not ship framework-specific tooling; if that is the gap you’re filling, this isn’t the tool for it.

​
The ROI of Manifest Awareness

The clearest proof is the artifact itself. When a manifest change lands, DecimalAI produces an Impact Report — every affected trace bucketed by IMPACT severity, with a per-trace compatibility verdict (keep / repair / flag / replay / drop). A representative tool-removal change lands at 247 HIGH / 501 MEDIUM / 1,254 LOW across 2,002 traces.

​
Scenario: 10-Agent Production System

Illustrative figures for a representative team — your numbers will vary with update cadence, trace volume, and how often changes touch high-traffic surfaces.

​
When to Use DecimalAI

DecimalAI is the right choice if:
You ship agent changes regularly and want to catch regressions before deploy
You use skills/instructions and want to know which ones actually work
You fine-tune models and need version-aware training data
Your agent’s tools, prompts, or models change frequently
You run multi-agent systems and need to track cross-agent drift
DecimalAI may not be the best fit if:
  • You only need basic LLM tracing, with no change-management workflow on top of it
  • What you actually want is a prompt hub or tooling specific to one agent framework
  • You only need evaluation scoring, without production tracing behind it

​
Getting Started

Pick the path that matches your immediate need:

Catch regressions

Most common entry point. Manifest impact analysis on every PR — no eval cases required.

Track skills

Effectiveness analytics, smart routing, public registry with SkillScore.

Build training data

Versioned SFT datasets that stay valid as the agent evolves.
If you’re migrating from a different tool, the migrations guide maps concepts side-by-side.