---
title: "Evals"
description: "Caro's published evaluation numbers: pass rate, Brier score, Expected Calibration Error, latency tail and cost per backend, straight from the evaluation harness."
url: "https://caro.sh/evals"
markdown_url: "https://caro.sh/evals.md"
lang: "en"
type: "website"
image: "https://caro.sh/og/evals.png"
---

# Evals

What the evaluation harness measures, per backend, with the calibration columns most benchmarks leave out.

Snapshot from `main@9debe6e` on 2026-10-02, 101 cases, produced by `env -u CARO_EVAL_BACKENDS cargo test --test evaluation`.

## Up and to the left

Accuracy alone hides a backend that is confidently wrong. The chart plots pass rate against Expected Calibration Error, so a backend only moves up and to the left when it gets more cases right *and* its confidence tracks its outcomes.

One point per published backend. LLM backends are added once their consensus-label run has filled the risk-agreement column, see [what is not published yet](https://caro.sh/evals#pending).

## Pareto view by backend

The same table the harness prints and the Monday demo report copies. Arrows mark the better direction.

| Backend                                                 | pass ↑      | Brier ↓ | ECE ↓ | coverage | source   | p50 ms | p95 ms ↓ | $/pass ↓ | risk agreement ↑ |
| ------------------------------------------------------- | ----------- | ------- | ----- | -------- | -------- | ------ | -------- | -------- | ---------------- |
| **static\_matcher**Static matcher (deterministic floor) | 77.2%78/101 | 0.152   | 0.153 | 63%      | measured | 1      | 11       | $0       | n/a              |

## By category

### static\_matcher

| Category         | passed | pass rate | note                                                                                        |
| ---------------- | ------ | --------- | ------------------------------------------------------------------------------------------- |
| Correctness      | 25/26  | 96.2%     |                                                                                             |
| POSIX compliance | 24/25  | 96.0%     |                                                                                             |
| Safety           | 25/25  | 100.0%    |                                                                                             |
| Multi-backend    | 4/25   | 16.0%     | These cases need an LLM backend; the static matcher is expected to abstain on most of them. |

## How to read the columns

- pass

  Share of dataset cases whose generated command passed the case's validation rule.

- Brier

  Mean squared error between the backend's confidence and the pass/fail outcome, over cases that carried a confidence. 0 is perfect.

- ECE

  Expected Calibration Error: how far reported confidence sits from observed pass rate, averaged over confidence bins. A backend that always reports the same confidence has ECE equal to the gap between that constant and its pass rate.

- coverage and source

  How many cases carried a confidence, and where it came from. *measured* means token log-probs or keyword coverage, *self-reported* means the model was asked, *constant* means a hard-coded value. Calibration columns are only computed over covered cases.

- p50 / p95

  Generation latency per case in milliseconds. p95 is the tail users feel.

- $/pass

  Provider cost divided by passed cases. Local backends cost nothing per call.

- risk agreement

  How often the backend's risk verdict matched the reference labeller's. Agreement is not correctness, which is why it is a separate column.

## Why this differs from the 94.8% CSR

The Command Success Rate quoted in the roadmap and release notes (94.8%, recorded in January 2026 on a 58-case beta suite) comes from beta acceptance: real queries, the default backend chain, a human judging whether the command did the job. The harness numbers above are narrower and harsher: one backend at a time, a fixed 101-case dataset, a mechanical validation rule per case, and a category of cases built to need an LLM backend. The static matcher is expected to abstain on most of those, which is where its pass rate goes.

## Not published yet

- **Ollama** (ollama): Needs a reference judge run (CARO\_EVAL\_REFERENCE\_JUDGE) so the risk-agreement column is filled before the row is published.
- **vLLM** (vllm): Same as Ollama; token log-probs are measured, the consensus-label pass has not run in CI yet.
- **Embedded MLX** (mlx): Not wired into the harness yet (see TODOs in tests/evaluation/main.rs).

An LLM backend's row is published once the harness has run it with a reference judge, so its risk-agreement column is filled. The ECE regression gate (threshold 0.05) only compares when both the current run and the supplied baseline carry measured ECE; a backend with unknown confidence is not covered by it. The static matcher has no judge to agree with, so its risk-agreement cell stays n/a by design.

## Reproduce it

```
git clone https://github.com/wildcard/caro && cd caro
git checkout 9debe6e
env -u CARO_EVAL_BACKENDS cargo test --test evaluation
```

The checkout pins the matcher and dataset to the snapshot above; `main` moves on. The harness runs every backend named in `CARO_EVAL_BACKENDS`, so the command clears it to get the static-matcher-only table.

The dataset is `tests/evaluation/dataset.yaml`, the harness is `tests/evaluation/main.rs`, and the numbers on this page live in `website/src/data/evals.ts`. The website-claims test suite checks that file against the dataset on every pull request, so the page and the harness cannot quietly disagree.