Regression Check (GitHub Action)

"""Entry point for DecimalAI manifest extraction in CI.

This file runs your existing agent initialization in 'manifest_only' mode.
The SDK captures tools, prompts, and models from the runtime objects
(no source-code parsing) and exits.
"""
from myapp.agent import build_agent  # adjust to your agent factory

if __name__ == "__main__":
    build_agent()

def build_agent():
    from langchain_openai import ChatOpenAI
    from langchain.agents import create_react_agent

    llm = ChatOpenAI(model="gpt-4o")
    tools = [search_tool, refund_tool]
    prompt = load_prompt()
    return create_react_agent(llm, tools, prompt)

name: Decimal Manifest Impact
on: [pull_request]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'

      - name: Install dependencies
        run: pip install -e .

      - name: Manifest extraction
        env:
          DECIMALAI_MODE: manifest_only
          DECIMAL_API_KEY: ${{ secrets.DECIMAL_API_KEY }}
          # Placeholder values for required env vars in your init code.
          # These are NOT called — manifest_only mode suppresses LLM calls.
          OPENAI_API_KEY: dummy_for_init
        run: python scripts/init_for_decimal.py

      - name: Impact check
        uses: decimal-labs/regression-check@v1
        with:
          api-key: ${{ secrets.DECIMAL_API_KEY }}
          agent-name: support-agent  # the same name you use in decimalai.init()

✅ Decimal Manifest Impact — support-agent

First run for this agent. Recorded the current manifest as the baseline.
Future PRs will diff against this.

🔍 Decimal Manifest Impact — support-agent

Manifest diff vs baseline (h_current_abc):
  + tool: refund_order
  ~ tool: search_docs (schema added required 'language' param)
  ~ system_prompt (28% rewritten — refund handling section)
  - tool: compare_competitors

Impact on your last 30 days of production traces (2,002 total):

  🔴  HIGH IMPACT — 247 traces will change behavior
       247 traces called the removed `compare_competitors` tool.

  🟡  MEDIUM IMPACT — 501 traces may behave differently
       412 traces called `search_docs` without specifying `language`
       89 traces depended on the rewritten refund handling section

  🟢  LOW IMPACT — 1,254 traces unaffected

Affected by current eval state:
  ✅  183 currently passing eval  ← regression-risk surface
  ❌   41 currently failing eval  ← opportunity if your change fixes them
  —  1,778 unscored

Structural: 🔴 HIGH IMPACT — 247 traces affected.
Eval:       ⚠️ REGRESSION LIKELY — 183 passing-eval traces affected.

Sample affected traces:
  • trace_a8f2 — "How do I compare with competitor X?"
  • trace_b3c4 — "Search for refund policies"
  • trace_d9e1 — "Refund order #4521"

[View full impact report → https://app.decimal.ai/agents/support-agent/regression/r_xyz]