Probability Calibration

1What a probability promises

A score becomes a probability only by matching frequency: among all the predictions to which the model assigned 0.9, about 90% should come true. That property is calibration, and nothing about a model's architecture grants it automatically — a number between 0 and 1 is not a probability just because of where it lives.

It matters the moment scores price decisions. Expected-cost triage, medical thresholds, an auto-approve-above-0.95 rule — each multiplies someone's cost by the printed number, and a miscalibrated 0.95 that delivers 0.7 misprices every case it touches.

Calibration is checked, never assumed: it is a claim about the world's agreement with the model, so it can only be measured against outcomes — held-out ones.

Figure. Among predictions assigned 0.9, about 90% should come true — that match is calibration. A miscalibrated 0.95 that delivers 0.7 misprices every expected-cost decision it touches.

2The reliability diagram

The calibration check made systematic: collect held-out predictions, group them into buckets by predicted probability, and for each bucket plot the observed frequency of the event against the predicted value. A perfectly calibrated model traces the diagonal — it delivers what it says at every level.

The signature shapes are worth recognising on sight. An overconfident model bows below the diagonal at the high end and above it at the low end: its 0.9s deliver less than 0.9 and its 0.1s more than 0.1. Underconfidence bows the other way. Naive Bayes typically draws an aggressively overconfident curve; a well-regularised logistic model often hugs the diagonal.

Buckets trade resolution against noise — ten equal-width bins is the usual start, and a bucket holding three cases says almost nothing. Scalar summaries such as the Brier score compress the curve into one number, at the usual price of hiding where the failure lives.

Figure. Held-out predictions bucketed by predicted probability. The dashed diagonal is a model that delivers what it says. The solid curve is overconfident: above the diagonal at the low end (its 0.1s come true more than a tenth of the time), crossing at 0.5, and flat at the top — the bucket predicted 0.9 delivers only 0.6.

Building one

1. Score held-out casesCollect predictions the fit never saw, together with their outcomes.
2. Bucket by claimGroup predictions into bins by predicted probability — ten equal bins to start.
3. Compare per bucketPlot observed frequency against claimed probability; the diagonal is the promise kept.

Coding lab. Bucket the promises runs in the app, with checks on your output.

3Calibration is not discrimination

Two separate achievements hide inside 'the model is good'. Discrimination is ranking: do positives get higher scores than negatives? It is what AUC measures. Calibration is truth-telling: do the scores mean their face value? A model can hold either property without the other.

Naive Bayes is the standing example of one side — often an excellent ranker whose probabilities saturate toward 0 and 1. The other side exists too: a model that predicts the base rate for every case is perfectly calibrated and completely useless, because it never separates anyone from anyone.

AUC does not move when scores are monotonically rescaled; calibration does. That asymmetry is the practical hinge: bad calibration is repairable after the fact by a monotone map, while bad ranking is not — no rescaling can reorder what is already ordered.

4Where miscalibration comes from

Miscalibration has causes, and knowing them tells you where to look. Correlated evidence double-counted — the naive Bayes mechanism — pushes scores toward the extremes. Overfitting does the same: a model that has memorised its training set has watched every training case resolve, and its confidence reflects that private certainty rather than the world.

Modern deep networks are famously overconfident even when highly accurate — trained long enough, the loss keeps rewarding sharper probabilities on cases the model already gets right.

And a model calibrated at deployment does not stay calibrated: when the base rate or the population drifts, the prior baked into its scores becomes the wrong prior — exactly the effect the base-rate lesson computed. The common thread is that calibration fails whenever the model's training experience differs from the question's context: too-dependent evidence, too-familiar data, or a shifted world.

Figure. Correlated features re-submit the same signal. Overfit models report private certainty from memorised training cases. Base-rate drift makes yesterday's calibrated score today's mispriced one. All three push scores toward the extremes.

The usual suspects

1. Double-counted evidenceCorrelated features re-submit the same signal; scores saturate toward 0 and 1.
2. Memorised training dataOverfit models report their private certainty, not the world's frequency.
3. A shifted worldBase-rate drift makes yesterday's calibrated score today's mispriced one.

5Repairing the scores

Because ranking survives monotone maps, a miscalibrated model can be repaired without retraining: learn a monotone function from its scores to honest probabilities on a held-out calibration set, and compose it on top. The ranking — and therefore AUC — is untouched; only the numbers change.

Three standard tools. Platt scaling fits a tiny logistic curve on the scores — two parameters, workable with little data. Isotonic regression fits any monotone step function — more flexible, and it needs more data to avoid overfitting the repair itself. Temperature scaling, the neural-network favourite, divides the score's logit by one learned constant T above 1 to soften overconfident outputs.

The non-negotiable: fit the repair on data the model was not trained on. Calibrating against training outcomes measures the model's memory, not its honesty.

6Held-out or it didn't happen

Every claim in this lesson — a reliability curve, a Brier score, a fitted temperature — is only evidence when computed on outcomes the model never trained on. Scores checked against training outcomes measure memory; the promise a probability makes is about cases the model has not seen.

The working discipline: split before anything is fit; fit on the training set; choose settings and fit any repair on validation data; quote final numbers on a test set touched once. The ML course formalises this as the train/validation/test contract, and every serious result you will report lives inside it.

Calibration is also not a one-time certificate. Deployed models drift with the world, so production systems re-check the reliability curve on fresh outcomes — which is monitoring, and it is where this course's final lesson on governance picks up.