Evaluating AI Systems

1Evaluation from day one

A model without a metric is a demo, not a decision system. A metric is a number, named before the model is built, that is computed by comparing predictions against answers the model was never shown — and 'named before' is doing real work in that sentence, because a metric chosen after the results are in tends to be the one the results happen to flatter.

Real systems run two evaluations, not one. Offline metrics are computed on a frozen, held-out dataset before anything ships: they are cheap, repeatable, and safe to fail. Online metrics are measured on live traffic after deployment — task success, complaint rates, revenue — and they are the ones the organisation actually cares about.

The offline metric is a proxy for the online one, and the gap between them is where evaluation goes wrong: a system can score beautifully on a fixed test set and still lose its users, because the test set stopped resembling the traffic. Strong teams treat the eval suite as a product of its own — versioned, reviewed, and extended every time a failure reaches production that the suite failed to predict.

The design order is therefore fixed: state the decision the system makes, choose the offline metric that best predicts the online outcome of that decision, freeze the held-out data — and only then fit a model.

Figure. One loop, two measurements. The offline eval on frozen data decides whether a candidate may ship; the online metrics on live traffic decide whether it should stay. Every online failure the offline suite did not predict becomes a new offline test — that dashed edge is how an eval suite grows.

Designing an eval

1. Name the decisionWrite down what the system decides and for whom — flag this transaction, rank this answer, admit this patient.
2. Pick the proxyChoose the offline metric most likely to move with the online outcome, and record why you believe it will.
3. Freeze the holdoutSet aside test data the fit never touches; every reuse of it spends a little of its honesty.
4. Wire the live checkDecide what will be measured after deployment, and what number triggers a rollback.

2Accuracy and the do-nothing floor

Accuracy is the fraction of predictions that are correct; error rate is its complement. It is the first metric everyone learns and the only one many systems ever report, so its failure mode needs to be understood precisely rather than vaguely distrusted.

Accuracy weights every row equally. That is fine when the classes are balanced and the two mistakes cost the same; it is catastrophic when one class is rare, because a model can then buy a high score by ignoring the rare class completely.

Before admiring any accuracy figure, compute the do-nothing floor: the score of the constant model that predicts the majority class for every row. On a dataset of 1,000 payment transactions of which 10 are fraudulent, that floor is 99% — any reported accuracy has to be read against it, not against zero.

The floor is not a straw man. It is the model you get for free, with no data, no training and no failure modes — so it is the price any learned model must beat before it has earned anything.

Figure. The whole dataset as one bar, segments drawn to scale. The sliver at the end is the fraud — 1% of the rows and 100% of the reason the system exists. Accuracy scores the bar by length, so a model that ignores the sliver still scores 99%.

3Class imbalance: when accuracy lies

Class imbalance is the situation where the outcomes you care about are rare: fraud in payments, disease in screening, defects on a production line. It is the normal case in deployed AI, not the exception, and it is exactly where accuracy stops measuring usefulness.

The mechanism deserves a precise statement: accuracy rewards a model for ignoring the rare class entirely, because every rare row sacrificed buys the model 99 easy ones. The optimisation pressure and the purpose of the system point in opposite directions.

The repair is to score the rare class directly, with two questions that have two different denominators: of the real frauds, what fraction did the model catch — and of the rows the model flagged, what fraction were really fraud. The precision-and-recall concept names and formalises both; here the point is only that neither can be bought by ignoring the sliver.

The confusion matrix is the bookkeeping device that makes the failure visible: four cells counting each way a prediction can meet reality. Below, the do-nothing model's 99% collapses into its cells — the entire score lives in one box, and the 10 rows that mattered sit in another.

Figure. The do-nothing model's whole 99% sits in the top-left cell. The 10 rows the system exists for sit beside it, missed; both cells in the calling-fraud row hold zero. Accuracy adds along the diagonal, so it cannot see which cell the correctness came from.

Coding lab. Accuracy vs recall on imbalanced data runs in the app, with checks on your output.

4Precision, recall and F1

Precision and recall put denominators under the two questions that class imbalance forces. Recall — of the rows that are truly positive, what fraction did the model flag — is computed against reality's total. Precision — of the rows the model flagged, what fraction are truly positive — is computed against the model's own flag count. Same confusion cells, different denominators, and confusing the two is the most common metrics error in practice.

The two trade against each other through the flagging behaviour: flag more rows and recall rises while precision falls; flag fewer and the reverse. Neither is 'the' metric — the task's costs decide which failure is dearer, which is the subject of the cost-sensitive lesson that follows this one.

F_1 is the harmonic mean of the pair, F_1 = 2PR/(P+R). The harmonic mean is chosen because it is dragged toward the smaller value: a model cannot hide a terrible precision behind a perfect recall, where an ordinary average would let it.

F1 also treats precision and recall as equally important, which is itself an assumption about costs — a silent one. Report the pair; reach for F1 when you need a single number to rank models by and have no cost model yet.

Figure. The same model as three numbers. The harmonic mean (F1 = 0.32) sits near the weaker of the pair; the arithmetic mean would sit at 0.50 and hide the precision problem entirely.

5Slice metrics: one average hides many systems

Every overall metric is a weighted average over segments of traffic — devices, regions, languages, customer tenures — and the model is not equally good on all of them. Slice metrics are the same metric computed per segment, and they are how real evaluation suites are organised: the overall number decides nothing on its own.

The arithmetic makes the hiding mechanical: a segment that is 10% of the data contributes 10% of the average, so the model can be near-useless there while the headline barely moves. The users in that slice do not experience an average.

A slice regression is also how models fail in production most often: an update improves the majority slice, the overall metric climbs, and a minority slice quietly breaks. An eval that tracks only the overall number will approve exactly that update.

Slicing is also where evaluation meets fairness: subgroup performance gaps are slice metrics by another name — that thread continues in the ethics and governance lesson.

Figure. The overall bar sits comfortably next to desktop while mobile fails at a coin flip — 10% of the traffic moves the average by only 4.5 points. The average is not evidence about any slice.

6Goodhart's law: when the metric becomes the target

Goodhart's law is the observation that a measure that becomes a target stops being a good measure. In ML systems it has two distinct faces, and both will happen to you.

The first face is the model gaming the metric: optimisation finds the cheapest way to the number, not the intended behaviour. Trained on click-through, a recommender learns clickbait; trained to close support tickets fast, a triage system learns to close them unsolved. The metric was a proxy, and optimisation pressure finds the gap between proxy and goal.

The second face is the team gaming the eval: every time a decision is tuned by looking at the test set — a threshold nudged, a feature kept because the held-out score liked it — a little of the set's independence is spent. Enough peeks, and the 'held-out' score is a training score wearing a lab coat.

The defences are structural, not moral: keep a final holdout that nobody tunes against, refresh eval data on a schedule, watch counter-metrics chosen to move if the proxy is being gamed, and treat any metric that improved suspiciously fast as a bug until traced.

The picture this concept wants is two curves parting company: the reported metric climbing steadily while the quantity it was meant to stand for sinks — the gap between them is what optimisation pressure found. The counter-metrics table gives three real instances of that gap.