Evals - Caro

Up and to the left

Accuracy alone hides a backend that is confidently wrong. The chart plots pass rate against Expected Calibration Error, so a backend only moves up and to the left when it gets more cases right and its confidence tracks its outcomes.

One point per published backend. LLM backends are added once their consensus-label run has filled the risk-agreement column, see what is not published yet.

Pareto view by backend

The same table the harness prints and the Monday demo report copies. Arrows mark the better direction.

Backend	pass ↑	Brier ↓	ECE ↓	coverage	source	p50 ms	p95 ms ↓	$/pass ↓	risk agreement ↑
static_matcherStatic matcher (deterministic floor)	77.2%78/101	0.152	0.153	63%	measured	1	11	$0	n/a

By category

static_matcher

Category	passed	pass rate	note
Correctness	25/26	96.2%
POSIX compliance	24/25	96.0%
Safety	25/25	100.0%
Multi-backend	4/25	16.0%	These cases need an LLM backend; the static matcher is expected to abstain on most of them.

How to read the columns

pass: Share of dataset cases whose generated command passed the case's validation rule.
Brier: Mean squared error between the backend's confidence and the pass/fail outcome, over cases that carried a confidence. 0 is perfect.
ECE: Expected Calibration Error: how far reported confidence sits from observed pass rate, averaged over confidence bins. A backend that always reports the same confidence has ECE equal to the gap between that constant and its pass rate.
coverage and source: How many cases carried a confidence, and where it came from. measured means token log-probs or keyword coverage, self-reported means the model was asked, constant means a hard-coded value. Calibration columns are only computed over covered cases.
p50 / p95: Generation latency per case in milliseconds. p95 is the tail users feel.
$/pass: Provider cost divided by passed cases. Local backends cost nothing per call.
risk agreement: How often the backend's risk verdict matched the reference labeller's. Agreement is not correctness, which is why it is a separate column.

Why this differs from the 94.8% CSR

The Command Success Rate quoted in the roadmap and release notes (94.8%, recorded in January 2026 on a 58-case beta suite) comes from beta acceptance: real queries, the default backend chain, a human judging whether the command did the job. The harness numbers above are narrower and harsher: one backend at a time, a fixed 101-case dataset, a mechanical validation rule per case, and a category of cases built to need an LLM backend. The static matcher is expected to abstain on most of those, which is where its pass rate goes.

Not published yet

Ollama (ollama): Needs a reference judge run (CARO_EVAL_REFERENCE_JUDGE) so the risk-agreement column is filled before the row is published.
vLLM (vllm): Same as Ollama; token log-probs are measured, the consensus-label pass has not run in CI yet.
Embedded MLX (mlx): Not wired into the harness yet (see TODOs in tests/evaluation/main.rs).

An LLM backend's row is published once the harness has run it with a reference judge, so its risk-agreement column is filled. The ECE regression gate (threshold 0.05) only compares when both the current run and the supplied baseline carry measured ECE; a backend with unknown confidence is not covered by it. The static matcher has no judge to agree with, so its risk-agreement cell stays n/a by design.

Reproduce it

git clone https://github.com/wildcard/caro && cd caro
git checkout 9debe6e
env -u CARO_EVAL_BACKENDS cargo test --test evaluation

The checkout pins the matcher and dataset to the snapshot above; main moves on. The harness runs every backend named in CARO_EVAL_BACKENDS, so the command clears it to get the static-matcher-only table.

The dataset is tests/evaluation/dataset.yaml, the harness is tests/evaluation/main.rs, and the numbers on this page live in website/src/data/evals.ts. The website-claims test suite checks that file against the dataset on every pull request, so the page and the harness cannot quietly disagree.