minor, moderate or major (how). A compatibility policy is the table that turns each grade into an action on the traces that used that part of the agent. An agent with no saved policy uses the default preset.
What a Policy Decides
A policy only labels traces. It doesn’t re-run your agent or rewrite anything by itself. If a trace used several changed components, the most severe action wins:
drop, then replay, repair, flag, keep.
The choice that matters most is replay or drop: does the input still mean something to the new agent? If you changed the tool that answers “Is SKU-1234 in stock?”, the question is still good, so replay it. If the agent can no longer answer pricing questions at all, drop those traces.
The Three Presets
Each cell reads minor / moderate / major:strict: for clean training data, when you’d rather lose traces than keep stale ones.default: for most teams.permissive: to keep as much as you can. It never drops, and it never repairs.
distillation, synthetic and manual traces on every model change.
Set a Policy
In the dashboard
Select your agent, then open More → Regression checks and expand Guardrails, a folded section at the top of the page.- Pick Strict, Default or Permissive.
- Change any cell in Change Response Rules. The switch beside the title shows the same rules as JSON.
- Choose Save.
Over REST
There’s no CLI command or SDK helper for policies. The routes are under/api/v1/manifests/policies:
POST makes the new policy the agent’s active one. PUT /{policy_id} updates name, preset or rules_json. GET /active?agent_name=... returns the agent’s policy, and is_configured: false means it’s on the default preset. GET /presets returns all three presets as JSON.
For your own rules, add rules_json. Each surface maps on_minor, on_moderate and on_major to an action:
source_type_overrides to rules_json, as in {"distillation": {"model_runtime": {"on_major": "keep"}}}. A grade you leave out of an override is keep.
Preview Impact
In Guardrails, choose Run Preview under Impact preview. It works out how the selected preset would classify traces from the latest version’s parent, and saves nothing. You need at least two versions.- It runs the selected preset. Cells you edited by hand aren’t included.
- The Replay card counts traces whose verdict is
replay. The preset’sflagrules count under Keep. The preview reads the saved preset, so save your edits first. - It covers the newest 5,000 traces.
{"preset": "strict"} for a preset, and compare_to for a version other than the parent. trace_counts.by_action holds the counts, with replays under flag here too.
What a Policy Does on a Live Manifest
- A new version classifies past traces under the saved policy, and stores a report for that version pair.
- Saving a policy leaves past verdicts as they are. Current comparison on Code changes uses the new rules straight away. Stored report keeps its snapshot until you run
POST /api/v1/manifests/{manifest_id}/compatibility/run. - Override, on a trace, sets its verdict by hand. A later classification keeps it.
- Agent-wide status is separate. The policy sets a trace’s verdict for a version pair. A trace’s
compat_statusis a roll-up built from fixed score thresholds and its quality scores, so the two can differ. To act on what your policy decided, replay by pair. - Regression checks show the policy’s action beside model and prompt changes. Whether a check passes depends on severity and
fail-on.
Contract violations
The bottom of Guardrails covers a different question: did a new trace do what its manifest says? This only runs for manifests fromregister_manifest(), which are closed contracts by default. Auto-detected manifests aren’t checked.
Each kind can be ignore, info or warn, and nothing is blocked: tool_not_declared, model_out_of_scope, output_schema_violation, context_source_undeclared, guardrail_missing, guardrail_failed. All default to warn, except context_source_undeclared at info.
Next Steps
Manifests
How changes are detected and graded.
Regression Check
Get the analysis as a PR comment.