Catch agent regressions before they ship

Every agent team ships changes weekly. Every change risks breaking something. Most teams find out only after their users do. DecimalAI tracks your agent’s structural identity — tools, prompts, models, skills — as a versioned manifest, then uses your production traffic as the test set when you propose a change. You don’t write eval cases. The default structural check needs no LLM key and never runs your agent — the one exception is the optional model-swap call replay, which in mode=real re-issues a single recorded call and does need a key for the same provider. Here’s what the check posts on your next PR — structural impact, computed before the change deploys:

🔍 Decimal Manifest Impact — support-agent

  🔴 HIGH IMPACT — 247 traces will break (called the removed `compare_competitors` tool)
  🟡 MEDIUM IMPACT — 501 traces may behave differently
  🟢 LOW IMPACT — 1,254 traces unaffected

  Verdict: Review before merging.

See it in action: Follow the 2-Minute Demo.

Use Claude Code or Cursor? Install a proven skill in 5 minutes

Pick a skill with a verified benchmark off the public registry and watch it fire in your agent — no signup, no SDK, no API key.

Build a support agent in Colab

Runtime → Run all. Pull one skill anonymously, write four sentences of system prompt, wire it into LangChain — then watch the agent stop telling a customer it deleted their account when it did no such thing. Two controls show it was the skill’s content and not just extra text in the prompt.

Not convinced by the number behind it? Audit one in Colab

About a second, no account, no key. Takes one ranked skill and shows the claim, every graded case behind it, one transcript with and without the skill — and how much of our own evidence we currently refuse to display.

Try either demo in 2 minutes

Two of the three capabilities ship with a one-command demo — seeded realistic data in your workspace, so the payoff lands before you instrument your own agent.

pip install decimalai
export DECIMAL_API_KEY="dai_sk_..."   # shown once after signup; more at app.decimal.ai/settings/api-keys

For engineers — catch regressions

decimalai demo regression

Links straight to the impact report: which production traces your next change would break.

For prompt engineers — find skills that work

decimalai demo skills

Links to the registry ranked by seeded effectiveness data — or browse it now, no signup.

What you can do with DecimalAI

Three capabilities, one foundation. Regression checks, the skills registry, and training-data validation each stand alone — but they compound on one thing: manifest-aware versioning, the structural fingerprint of your agent.

Catch regressions before you ship

Open a PR with an agent change; get a structural impact report — which production traces will break, may differ, or are unaffected — before it deploys.

Discover & share proven skills

Install skills from the registry ranked by real production-effectiveness data, then measure how they perform on your own traffic.

Keep training data valid

As your agent evolves, traces are auto-classified keep / repair / replay / drop against the manifest diff — so your training set stays clean.

Why “manifest-aware” matters

The usual way to catch an agent regression is eval-driven: you run an eval suite against the new version. That works only if you’ve written eval cases — which most teams haven’t, and the ones they have are usually stale. DecimalAI works differently. Your production traces are tagged with the manifest they ran under. When you propose a manifest change, we identify which traces depended on what’s changing and tell you the structural blast radius — no eval suite required.

	Eval-driven testing	Manifest-aware (DecimalAI)
What the test set is	Eval cases you write and keep current	Your production traffic
What has to run	Your agent, executed in CI	A query over the trace store
What it tells you	How the new version scores on those cases	Exactly which traces touched the changed surface
Removed-tool changes	Covered where eval coverage exists	Detected structurally
Cost per check	Whatever a full eval-suite run costs, often LLM-graded	<$0.001 (database query)

What DecimalAI does NOT do (yet)

We’re honest about the boundaries:

Boundary	What we do today	Deferred / how to cover
Running your full agent in CI	Pre-deploy regression detection is structural impact analysis — we tell you what’s at risk based on the manifest diff, without executing your agent.	Full end-to-end agent replay is deferred. For changes where structure can only say “everything may be affected” (large prompt rewrites), use a careful canary deploy and our post-deploy bisect view.
Behavioral verification of model swaps	Model-swap call-replay ships now (preview): for each affected trace we re-issue one recorded model call against the candidate model and diff the outputs. Defaults to mock (no spend); `mode=real` does same-provider swaps.	Cross-provider swaps and replaying the full agent (not just a single recorded call) are deferred.
Holding your LLM API keys	Pre-deploy structural analysis runs entirely against our trace store — no keys required.	`mode=real` call-replay needs a key for the same provider; mock replay needs none.
Replacing your observability tool	DecimalAI runs alongside LangSmith, Braintrust, Langfuse, etc. Pipe your traces in via the SDK; we add the manifest layer underneath.	Not a goal — keep your existing tracing in place.

Supported Frameworks

Framework	Integration	Manifest Fidelity
LangChain / LangGraph	`decimalai.langchain`	Full (tools, models, prompts)
OpenAI Agents SDK	`decimalai.openai_agents`	Full (with Agent introspection)
Claude Agent SDK	`decimalai.claude_agent_sdk`	Full (tools, prompt, model, subagents)
Google ADK	`decimalai.adk` — native ADK plugin, `init(adk=True)`	Full (model, instruction, tools, sub-agents)
LlamaIndex	`decimalai.llamaindex`	Full (span-based)
CrewAI, Haystack, AutoGen / AG2	`decimalai.otel`	Model + tool names
Any Python agent	`@decimalai.trace`	Configurable

Next Steps

Start here

Install a Skill (5 min)

No signup: install a benchmarked registry skill into Claude Code or Cursor and watch it fire.

Quickstart

Install the SDK, get your first trace, and add the GitHub Action in under 10 minutes.

2-Minute Demo

One command each: a live impact report and the ranked skills registry, on seeded data.

By role

Engineering teams

Catch agent regressions on every PR with manifest-aware impact analysis.

Prompt engineers

Track skills, browse the registry, and measure effectiveness with production data.

ML teams

Build versioned SFT datasets that stay valid as your agent evolves.

Reference

Concepts

Every term and system explained, with diagrams.

Manifests Guide

How automatic version tracking and compatibility scoring work.

Why DecimalAI?

How manifest-aware change management differs from eval-driven testing.

API Reference

Every REST endpoint with examples and schemas.

Get Started

Foundation

Ship Safely

Improve with skills

Trust & Safety

Data & Training

Operate

Core Concepts

Platform

Catch agent regressions before they ship

Use Claude Code or Cursor? Install a proven skill in 5 minutes

Build a support agent in Colab

Not convinced by the number behind it? Audit one in Colab

Try either demo in 2 minutes

For engineers — catch regressions

For prompt engineers — find skills that work

What you can do with DecimalAI

Catch regressions before you ship

Discover & share proven skills

Keep training data valid

Why “manifest-aware” matters

What DecimalAI does NOT do (yet)

Supported Frameworks

Next Steps

Start here

Install a Skill (5 min)

Quickstart

2-Minute Demo

By role

Engineering teams

Prompt engineers

ML teams

Reference

Concepts

Manifests Guide

Why DecimalAI?

API Reference

Use Claude Code or Cursor? Install a proven skill in 5 minutes

Build a support agent in Colab

Not convinced by the number behind it? Audit one in Colab

​Try either demo in 2 minutes

For engineers — catch regressions

For prompt engineers — find skills that work

​What you can do with DecimalAI

Catch regressions before you ship

Discover & share proven skills

Keep training data valid

​Why “manifest-aware” matters

​What DecimalAI does NOT do (yet)

​Supported Frameworks

​Next Steps

​Start here

Install a Skill (5 min)

Quickstart

2-Minute Demo

​By role

Engineering teams

Prompt engineers

ML teams

​Reference

Concepts

Manifests Guide

Why DecimalAI?

API Reference

Try either demo in 2 minutes

What you can do with DecimalAI

Why “manifest-aware” matters

What DecimalAI does NOT do (yet)

Supported Frameworks

Next Steps

Start here

By role

Reference