> ## Documentation Index
> Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation

> Evaluators, eval scores, eval verdicts, and the unified decision engine that combines quality + compatibility.

Evaluations measure whether your agent's outputs are good. DecimalAI provides built-in evaluators and supports custom ones.

## Evaluator

An **evaluator** is a configured quality check that runs on traces. DecimalAI supports three categories:

| Category | How It Works | Example |
| - | - | - |
| **Deterministic** | Rule-based checks, instant, zero cost | Response length, JSON validity, regex match |
| **LLM-as-Judge** | An LLM scores the output | Relevance, helpfulness, safety, coherence |
| **Custom** | User-defined via the SDK `@eval` decorator | Domain-specific checks, business rules |

Evaluators produce scores (0.0–1.0) and verdicts (pass/fail) that feed into:

* The eval dashboard (trends, distributions)
* Dataset filtering (only train on passing traces)
* The decision engine (unified keep/repair/replay/drop)

→ See [Evaluations](/guides/evaluations) for the full guide.

## Eval Score

An **eval score** is a single evaluation result. Each score has:

| Field | Description |
| - | - |
| `name` | Which evaluator produced it (e.g., "relevance") |
| `score` | 0.0 to 1.0 |
| `passed` | Boolean — did it meet the threshold? |
| `source` | Who ran it: `built_in`, `llm_judge`, `sdk`, `compat_engine` |
| `category` | What it measures: `quality` or `compatibility` |

A single trace can have many eval scores from different evaluators.

## Eval Verdict

The **eval verdict** is the aggregate outcome for a trace: `pass`, `fail`, or `review`. Computed from all individual eval scores.

<Note>
  **Eval Score vs Eval Verdict:** An eval score is one evaluator's result (e.g., "relevance: 0.85"). The eval verdict is the roll-up across all scores. A trace with 5 passing scores and 1 failing score gets verdict = `fail`.
</Note>

## Decision Engine

The **decision engine** is the final arbiter. It combines quality eval scores AND compatibility scores into a single verdict per trace. Precedence runs top to bottom — the first rule that matches wins and short-circuits the rest, so a higher rule overrides a lower one (e.g. a drop-level compat score, or quality below 40%, beats a repair-level score).

```mermaid theme={null}
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#f5f5f4','primaryBorderColor':'#a8a29e','primaryTextColor':'#44403c','lineColor':'#a8a29e'}}}%%
flowchart TD
    Start([Trace scores in]) --> Q1{Any compat score<br/>drop-level?}
    Q1 -->|yes| DROP1[drop]
    Q1 -->|no| Q2{Any compat score<br/>replay-level?}
    Q2 -->|yes| REPLAY[replay]
    Q2 -->|no| Q3{Quality average<br/>&lt; 40%?}
    Q3 -->|yes| DROP2[drop]
    Q3 -->|no| Q4{Any compat score<br/>repair-level?}
    Q4 -->|yes| REPAIR[repair]
    Q4 -->|no| KEEP[keep]
    classDef keep fill:#dcfce7,stroke:#16a34a,color:#166534;
    classDef repair fill:#fef9c3,stroke:#eab308,color:#854d0e;
    classDef replay fill:#dbeafe,stroke:#3b82f6,color:#1e40af;
    classDef drop fill:#fee2e2,stroke:#ef4444,color:#991b1b;
    class KEEP keep
    class REPAIR repair
    class REPLAY replay
    class DROP1,DROP2 drop
```

Rule 3 is the one that overrides repair: a trace whose surfaces are only `repair`-level still drops if its quality average falls below 40%, because a low-quality output isn't worth mechanically fixing.

<Note>
  **Quality Score vs Compatibility Score:** Both are stored as eval scores but measure different things. Quality scores (`category="quality"`) ask "is the output good?" Compatibility scores (`category="compatibility"`) ask "does this trace work with the current agent version?" A trace must pass BOTH to be kept for training.
</Note>

## Next

<CardGroup cols={2}>
  <Card title="Skills & Data Pipeline" icon="database" href="/concepts/skills-and-data">
    How filtered + scored traces become reproducible training datasets.
  </Card>

  <Card title="Evaluations Guide" icon="circle-check" href="/guides/evaluations">
    Hands-on guide to setting up evaluators.
  </Card>
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.