Regularisation

1Huge weights chase noise

Picture a student preparing for an exam by memorising last year's paper word for word. On those exact 200 questions they score 100%. On a fresh paper they collapse, because they never learned the subject — they learned the quirks of one specific question set. A model can fail in exactly the same way: it can score brilliantly on the rows it was trained on and badly on new rows. That failure has a name — overfitting.

Training loss is one number that says how wrong the model is on the rows it studied — smaller is better. Overfitting is what happens when the fit is allowed to buy a slightly smaller training loss with a story that will not repeat on fresh rows.

No diagram — the memorising-student table carries the analogy.

2Where overfitting lives: one huge weight

The running example for this whole lesson is a spam filter: a small program that reads an incoming email and answers one question — spam, or not spam. You build it by showing it 200 old emails, each already labelled spam or safe, and letting it learn from them. That learning step is called training. The filter keeps one learned number per clue word — per word, like how often FREE appears. That number is a weight, written w. A weight of +2 on FREE means that word pushes the score firmly toward spam. A weight near 0 means the filter barely listens to that word.

Suppose the word "meeting" happened, by pure coincidence, to appear in three spam emails in the training pile. An ordinary fit — the fit that only looks at training loss — will happily give "meeting" a huge weight, say w = +7, because that buys a slightly better training-loss number on those 200 emails. On next week's inbox, where "meeting" means a meeting, that huge weight is a liability. The training-loss win was a quirk, not a subject.

Figure. The spam filter keeps one learned weight w per clue: +2 on FREE pushes firmly toward spam, a weight near zero is barely heard. An ordinary fit — the fit that only looks at training loss — hands meeting a huge +7 because that word happened to appear in three training spams. The slightly better training-loss number is a coincidence, and the huge weight becomes a liability on next week's inbox.

3An ordinary fit keeps the huge weight

Who chooses the weights? Nobody types them in. Training does. Training tries out weight settings, scores each one, and keeps the setting with the best score. That is the entire game: not a person picking by taste, and not a vote from next week's inbox — a search for the cheapest score on the rows the model is allowed to study.

An ordinary fit is the fit whose bill is training loss alone. Two candidate settings for the spam filter: setting A hangs w = +7 on meeting and scores training loss 0.10 (its "how wrong on the 200 training emails" number). Setting B hangs only w = +0.5 on meeting and scores training loss 0.30. Smaller training loss wins, so 0.10 beats 0.30 and training keeps the monster. Next week's inbox, where that monster embarrasses us, is not on the scoreboard.

Figure. An ordinary fit scores only training loss. Setting A (meeting weight +7) has training loss 0.10; setting B (meeting weight +0.5) has training loss 0.30. Smaller wins, so the ordinary fit keeps the huge weight. Next week's inbox is not on this scoreboard.

4Charge the size of the weights

Regularisation is the fix, and the idea is simple: charge the model for the size of its weights. The extra charge is called a penalty — a second number, added to training loss, that grows as the weights grow. A huge weight on "meeting" is no longer free. Training still tries settings and keeps the cheapest one — that part never changes — but cheapest now means cheapest on the whole bill, loss and penalty together. The fit keeps a large weight only if that weight earns its cost by genuinely improving predictions. Three coincidental emails no longer qualify.

This lesson uses one standard size-bill, called L2 or ridge: add up the squares of the weights. If the two headline weights are w_1 on FREE and w_2 on meeting, the size-bill is w_1^2 + w_2^2. Squaring punishes a 7 much harder than a 0.5 — 7^2 = 49, while 0.5^2 = 0.25. The total bill the fit now minimises is training loss plus \lambda times that size-bill. \lambda (lambda) is a price you choose, not a number the model learns: it says how many points of training loss you are willing to spend per unit of size-bill.

Figure. The same two fits, priced at two λ settings. Numbers on the grid are the total bills: at weak λ=0.001 Fit A wins 0.153 to 0.304; at strong λ=0.01 Fit B wins 0.3425 to 0.63.

5The 0.20 versus 48.75 tug-of-war

Keep the same two spam-filter settings. Both listen to FREE at w_1 = +2.0. Setting A listens to meeting at w_2 = +7.0 and has training loss 0.10. Setting B listens to meeting at w_2 = +0.5 and has training loss 0.30. The L2 size-bill is w_1^2 + w_2^2. For A that is 2.0^2 + 7.0^2 = 4 + 49 = 53. For B that is 2.0^2 + 0.5^2 = 4 + 0.25 = 4.25. A pays 53 - 4.25 = 48.75 extra size-bill for the privilege of the huge meeting weight. What does A buy with that extra size? A training-loss number that is 0.30 - 0.10 = 0.20 smaller. So the tug-of-war is 0.20 of training-loss win versus 48.75 of extra size-bill.

Those two numbers are not yet in the same units. \lambda is the exchange rate: each unit of size-bill costs \lambda points of training loss. At a cheap price \lambda = 0.001, A's extra 48.75 of size costs only 0.001 \times 48.75 = 0.04875 — less than the 0.20 it saves, so A still wins. At a dear price \lambda = 0.01, the same extra size costs 0.01 \times 48.75 = 0.4875 — more than the 0.20 it saves, so B wins. The emails did not change. Only the price of big weights did.

6The same price in scikit-learn

A Python library is a bundle of ready-made tools someone else already wrote, so you do not have to type the maths by hand. scikit-learn is one such library for machine learning. The short name you type in Python is sklearn. The line `import sklearn` loads the library. The everyday line is `from sklearn.linear_model import Ridge` — that reaches into the linear-model drawer inside the library and pulls out one tool, Ridge. Ridge is the name scikit-learn gives the L2 (ridge) fit you just priced by hand: it still tries to make training loss small, and it still adds \lambda times the sum of squared weights.

Two verbs do the work. `fit` means study the training rows and pick the weights. `predict` means use those learned weights on new rows. The penalty knob on Ridge is called `alpha`. It is the same idea as lambda: a bigger `alpha` is a dearer price on big weights. The lab below fits the same four-row table twice — once at `alpha=0.01` (cheap) and once at `alpha=10` (dear) — and prints the two weights. Expect the dear setting to shrink both numbers. Same L2 bill — one line of code. You do not need the rest of the library today.

Figure. scikit-learn is a Python library imported as sklearn. Ridge is the L2 tool inside it. alpha is the same price as lambda. fit studies the training rows and picks weights; predict uses those weights on new rows. At a cheap alpha of 0.01 the two weights stay near 2.00 and 1.98; at a dear alpha of 10 they shrink to about 0.67 and 0.18.

Ridge in four lines

import sklearn
from sklearn.linear_model import Ridge
model = Ridge(alpha=0.01)  # alpha is λ
model.fit(X, y)            # study rows, pick weights
guesses = model.predict(X_new)

Coding lab. Watch alpha shrink the weights runs in the app, with checks on your output.