Regression Check (GitHub Action)

pip install decimalai
export DECIMAL_API_KEY="dai_sk_..."
decimalai demo regression

"""Registers this PR's candidate manifest for the regression check."""
import decimalai
from myapp.agent import build_agent  # adjust to your agent factory


def main() -> None:
    decimalai.init()  # reads DECIMAL_API_KEY and DECIMALAI_MODE from the environment

    agent = build_agent()  # builds the agent and never runs it, so no model is called

    result = decimalai.flush_manifest_for_ci(
        agent_name="support-agent",  # must match the Action's `agent-name` input
        chain=agent,                 # LangChain/LangGraph: reads tools, prompts and models
    )
    print(f"Manifest registered: {result['manifest_id']} -> {result['output_path']}")


if __name__ == "__main__":
    main()

name: Agent Regression Check
on: [pull_request]

permissions:
  contents: read
  pull-requests: write  # lets the Action post and update its PR comment

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install -e .

      - name: Manifest extraction
        id: manifest  # required: the next step reads this step's output
        env:
          DECIMALAI_MODE: manifest_only
          PYTHONPATH: .  # lets the script import from the repo root
          DECIMAL_API_KEY: ${{ secrets.DECIMAL_API_KEY }}
          OPENAI_API_KEY: dummy_for_init  # placeholder for keys your init code requires
        run: python scripts/init_for_decimal.py

      - name: Agent Regression Check
        uses: decimal-labs/regression-check@v1
        with:
          api-key: ${{ secrets.DECIMAL_API_KEY }}
          agent-name: support-agent  # the name your production agent passes to agent_config(...)
          candidate-manifest-id: ${{ steps.manifest.outputs.decimal_manifest_id }}

🔍 Agent Regression Check — support-agent

Manifest changes:
- 🟡 Tool renamed — lookup_price
- 🟡 Model changed — gpt-4o-mini · moderate (version bump)
- 🟡 Prompt section rewritten — system_prompt · major (88.6% changed)
- 🟢 Optional param added — check_inventory
- 🟢 Tool added — refund_order
- 🔴 Tool removed — compare_competitors

Impact on last 120 production traces:

| Severity         | Affected | Currently passing eval | Currently failing | Unscored |
| all              | 120      | 15                     | 5                 | 100      |
| 🔴 HIGH IMPACT   | 2        |
| 🟡 MEDIUM IMPACT | 118      |
| 🟢 LOW IMPACT    | 0        |

Top affected:
- 2 traces called the removed compare_competitors tool. These will error or
  fall back when this change ships. (2 failing)

---
🔴 HIGH IMPACT — review before merging
⚠️ REGRESSION LIKELY — affects traffic that was passing eval

Behavioral verification — model change detected; the structural check can't
confirm whether outputs differ.
- 120 recorded calls can be replayed against the new model. Add
  behavioral-check: real to verify (spends tokens), or use post-deploy bisect.

View full report →

PYTHONPATH=. DECIMALAI_MODE=manifest_only python scripts/init_for_decimal.py
decimalai regression-check --agent-name support-agent