> ## Documentation Index
> Fetch the complete documentation index at: https://docs.decimal.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluation

> Evaluators, eval scores, eval verdicts, and the unified decision engine that combines quality + compatibility.

Evaluations measure whether your agent's outputs are good. DecimalAI provides built-in evaluators and supports custom ones.

## Evaluator

An **evaluator** is a configured quality check that runs on traces. DecimalAI supports three categories:

| Category          | How It Works                               | Example                                     |
| ----------------- | ------------------------------------------ | ------------------------------------------- |
| **Deterministic** | Rule-based checks, instant, zero cost      | Response length, JSON validity, regex match |
| **LLM-as-Judge**  | An LLM scores the output                   | Relevance, helpfulness, safety, coherence   |
| **Custom**        | User-defined via the SDK `@eval` decorator | Domain-specific checks, business rules      |

Evaluators produce scores (0.0–1.0) and verdicts (pass/fail) that feed into:

* The eval dashboard (trends, distributions)
* Dataset filtering (only train on passing traces)
* The decision engine (unified keep/repair/replay/drop)

→ See [Evaluations](/guides/evaluations) for the full guide.

## Eval Score

An **eval score** is a single evaluation result. Each score has:

| Field      | Description                                                 |
| ---------- | ----------------------------------------------------------- |
| `name`     | Which evaluator produced it (e.g., "relevance")             |
| `score`    | 0.0 to 1.0                                                  |
| `passed`   | Boolean — did it meet the threshold?                        |
| `source`   | Who ran it: `built_in`, `llm_judge`, `sdk`, `compat_engine` |
| `category` | What it measures: `quality` or `compatibility`              |

A single trace can have many eval scores from different evaluators.

## Eval Verdict

The **eval verdict** is the aggregate outcome for a trace: `pass`, `fail`, or `review`. Computed from all individual eval scores.

<Note>
  **Eval Score vs Eval Verdict:** An eval score is one evaluator's result (e.g., "relevance: 0.85"). The eval verdict is the roll-up across all scores. A trace with 5 passing scores and 1 failing score gets verdict = `fail`.
</Note>

## Decision Engine

The **decision engine** is the final arbiter. It combines quality eval scores AND compatibility scores into a single verdict per trace. Precedence runs top to bottom — the first rule that matches wins and short-circuits the rest, so a higher rule overrides a lower one (e.g. a drop-level compat score, or quality below 40%, beats a repair-level score).

```mermaid theme={null}
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#f5f5f4','primaryBorderColor':'#a8a29e','primaryTextColor':'#44403c','lineColor':'#a8a29e'}}}%%
flowchart TD
    Start([Trace scores in]) --> Q1{Any compat score<br/>drop-level?}
    Q1 -->|yes| DROP1[drop]
    Q1 -->|no| Q2{Any compat score<br/>replay-level?}
    Q2 -->|yes| REPLAY[replay]
    Q2 -->|no| Q3{Quality average<br/>&lt; 40%?}
    Q3 -->|yes| DROP2[drop]
    Q3 -->|no| Q4{Any compat score<br/>repair-level?}
    Q4 -->|yes| REPAIR[repair]
    Q4 -->|no| KEEP[keep]
    classDef keep fill:#dcfce7,stroke:#16a34a,color:#166534;
    classDef repair fill:#fef9c3,stroke:#eab308,color:#854d0e;
    classDef replay fill:#dbeafe,stroke:#3b82f6,color:#1e40af;
    classDef drop fill:#fee2e2,stroke:#ef4444,color:#991b1b;
    class KEEP keep
    class REPAIR repair
    class REPLAY replay
    class DROP1,DROP2 drop
```

Rule 3 is the one that overrides repair: a trace whose surfaces are only `repair`-level still drops if its quality average falls below 40%, because a low-quality output isn't worth mechanically fixing.

<Note>
  **Quality Score vs Compatibility Score:** Both are stored as eval scores but measure different things. Quality scores (`category="quality"`) ask "is the output good?" Compatibility scores (`category="compatibility"`) ask "does this trace work with the current agent version?" A trace must pass BOTH to be kept for training.
</Note>

## Next

<CardGroup cols={2}>
  <Card title="Skills & Data Pipeline" icon="database" href="/concepts/skills-and-data">
    How filtered + scored traces become reproducible training datasets.
  </Card>

  <Card title="Evaluations Guide" icon="circle-check" href="/guides/evaluations">
    Hands-on guide to setting up evaluators.
  </Card>
</CardGroup>