Evaluation Metrics

1Accuracy, the obvious first score

Picture a program a bank runs on every card payment. For each payment it makes a prediction — a guess about a label it cannot see yet — and here the label is one of two words: 'legit' (a normal purchase) or 'fraud' (a stolen card being used). Once the truth comes out, each guess is either correct (the model said what the payment really was) or wrong. The obvious question is: how good is this model? The first score everyone reaches for is called accuracy.

Accuracy is simply the number of correct predictions divided by the total number of predictions made. If the model looked at 100 payments and got 95 of them right, its accuracy is 95/100 = 0.95, which people read as 95%. Nothing hidden: right answers over all answers. It feels like the whole story — and that feeling is exactly the trap this concept exists to warn you about.

Figure. One hundred payment predictions laid end to end: 95 came out correct and 5 wrong, so accuracy is 95 out of 100 — read as 95%. Right answers over all answers, nothing hidden.

2The do-nothing model scores 95%

Here is the trap. In real card traffic, fraud is rare: out of every 100 payments, maybe 95 are legit and only 5 are fraud. That 95-to-5 balance is called the base rate — how often each label actually occurs before any model gets involved. Now imagine the laziest possible model: it ignores the payment entirely and says 'legit' every single time. It is right on all 95 legit payments and wrong on all 5 frauds, so its accuracy is 95/100 = 0.95. A model that catches zero fraud — the one job it was built for — reports a shiny 95%.

Why does the number lie like this? Because accuracy counts every correct answer as equally valuable, so a model can rack up a huge score just by always guessing the majority class — the label that appears most often (here, 'legit'). The rare label we actually care about hunting — the fraud — is what evaluators call the positive class, and on skewed data accuracy barely notices whether the positives are found at all. The more lopsided the base rate, the higher a do-nothing model scores.

Figure. On a 95% / 5% split, a model that always calls the majority class scores 0.95 accuracy and finds zero rare positives. The tall bar is the trap; the empty bar is the cost. Quote the base rate before you quote accuracy.

3Base rate and cost beside the number

So before you quote accuracy, write two things next to it. First, the base rate: if a do-nothing majority guesser would already score 95%, then a model reporting 95% has told you nothing. Second, the cost of each kind of mistake: missing a real fraud loses the bank real money, while a false alarm merely means a customer gets a 'was this you?' text. When the mistakes cost different amounts, or the base rate is skewed, you need scores that look at the rare class directly — recall (of the real frauds, how many did we catch?) and precision (of the payments we flagged, how many were really fraud?). Those two get their own lesson next; for now just know they exist and why accuracy alone sent you looking for them.

None of this makes accuracy useless. When the classes are roughly balanced — say, sorting photos into cats and dogs, where both appear about half the time — and both kinds of mistake cost about the same, correct-over-total is a perfectly fair headline. The rule of thumb: balanced classes and equal costs, accuracy can lead; skewed either way, accuracy is only a sanity check, never the decision metric.

1. Count the base rateBefore trusting accuracy, check how often each class appears — a majority vote can look brilliant.
2. Match metric to costFraud and churn need precision/recall or ROC — not a single accuracy headline.
3. Compare train vs testA wide gap signals overfit even when test accuracy still looks acceptable.

4Overfitting signals

Think of a student who prepares for an exam by memorising last year's answer key word for word. Quiz them on last year's paper and they score perfectly; hand them a fresh paper with new questions and they collapse, because they never learned the underlying method — only the specific answers. Machine-learning models can fail in exactly this way, and the failure has a name: overfitting.

To catch it, we split our data into two piles before any learning happens. The training set is the pile of rows the model is allowed to study — it sees both the inputs and the true labels and adjusts itself to fit them. The test set (also called the hold-out set, because we hold it out of reach) is a pile the model never sees during learning, locked away until the very end. The test set plays the role of the fresh exam paper: it measures whether the model learned patterns that generalise — that keep working on new data — rather than memorising the training rows.

Now score the same model on both piles. Suppose it gets 99% of the training rows right but only 71% of the test rows right. Neither number alone is the alarm — the 28-point gap between them is. A big gap with a glittering train score is the classic overfitting signature: the model latched onto quirks and coincidences that exist only in the training pile, exactly like the student who memorised one specific answer key.

Figure. The same model, two scores: 96% on rows it trained on, 71% on rows it has never seen. Neither number alone is the alarm - the 25-point gap is. Close it by simplifying the model, adding data, or regularising before reaching for anything exotic.

5Read the two scores together

Be careful not to confuse this with the opposite problem. If the train score and test score are both poor and close together — say 61% and 60% — the model is not memorising anything; it is too simple, or its input features are too weak, to capture the pattern at all. That is underfitting, and its remedy (make the model richer, or find better features) is the exact opposite of the overfitting remedy. Always read the two scores together: a wide gap says overfit, two low scores in agreement say underfit.

6Fixing an overfit model

So what do you actually do about an overfit model? Start by making it simpler, not fancier: a shallower decision tree means fewer, broader rules with less room to memorise individual rows, and stronger regularisation — a penalty added during training that punishes overly complicated fits — pushes the model toward smoother, simpler patterns. Adding more (and cleaner) training rows also helps, because quirks that appear in 800 rows tend to wash out across 8,000. And one rule is absolute: never tune your settings by re-running against the test set. Every time you adjust the model to raise the test score, the test set quietly turns into another practice paper the model has effectively seen — and it can no longer tell you how the model will do on genuinely new data. Keep a separate validation slice for tuning; touch the test set once, at the end.

Only after those basics are exhausted should anyone reach for a more exotic model. Extra complexity is not a cure for a wide gap — it usually widens it — and it certainly cannot fix data leakage, where an input column secretly contains the answer (imagine a 'refund issued' column leaking into a fraud predictor). If a score looks too good to be true, suspect the data before you celebrate the model.

1. Simplify firstShallower trees and stronger regularisation leave less room to memorise individual rows.
2. Add cleaner rowsQuirks that survive in 800 rows tend to wash out across 8,000.
3. Guard the test setTune on a separate validation slice; touch the test set once, at the end.