L1, L2 and the C Dial

1The knob is a price you set — lambda or alpha

Regularisation charges a model for the size of its weights — the learned numbers it hangs on each clue. How hard that charge pushes is a dial you set by hand, not a number the model learns. Textbooks write the dial as \lambda (the Greek letter lambda). A bigger lambda means a stronger charge and therefore smaller weights. scikit-learn is a Python library of ready-made tools; the short name you type is sklearn. Its Ridge tool — the ridge / L2 fit — names the same dial `alpha`. Lambda and alpha are the same kind of knob: bigger means squeeze harder.

Some other tools flip the dial and call it C, defined as one divided by the strength. sklearn's LogisticRegression uses C. So a smaller C means a stronger charge — the opposite direction of lambda and alpha. To regularise harder in LogisticRegression you lower C. Raising C toward 100 or 1000 nearly switches the charge off. Meet alpha first; treat C as the same knob written upside down.

1. Spot overfitTrain accuracy high and test accuracy low — coefficients may be chasing noise.
2. Choose L1 or L2L2 shrinks all weights; L1 can zero some and drop features entirely.
3. Tune C on validationSmaller C means stronger penalty — pick C by validation score, never by peeking at test.

2Score the knob on a held-out pile

You cannot pick the knob by how well the fit does on the rows it studied. Those rows already taught the weights; a tiny lambda will look brilliant there because it was allowed to memorise quirks. The honest score uses a held-out pile: rows you cut aside before any fit starts, that the fit is never allowed to study. That pile is called the validation set. Its only job is to compare the settings you chose by hand — here, which lambda or alpha to keep.

Picture a rent office with 200 already-rented flats. Cut once, up front: 140 flats to study (the training pile), 40 flats the fit must never see while you try knob values (the validation pile), and 20 flats locked for the very end (the test pile). No flat sits in two piles. Do not pick lambda by peeking at the test pile; that headline number would then be a cheat. This lesson uses the validation pile to choose, and leaves the test pile shut.

Figure. One cut of 200 rented flats: 140 to study (train), 40 to score each knob value (validation), 20 locked until the knob is already chosen (test). Each flat lands in exactly one pile.

3Try a few values, keep the winner

The recipe is short. Try a few knob values. For each value, fit on the training flats only. Score that fit on the validation flats using one number: the average miss. A miss on one flat is the absolute difference |\text{actual rent} - \text{guessed rent}| — drop the sign, keep the size. Average miss means add the misses and divide by how many held-out flats you scored. Keep the knob value whose average miss is smallest.

scikit-learn can run that sweep for you. `Ridge` is the tool that fits a rent line and charges for weight size; you pass it one `alpha`. `RidgeCV` is the same tool with a built-in contest: you hand it a list of alphas, it tries each one on held-out rows, and it keeps the winner. You still have to understand the contest — the class is not magic, it is the recipe above typed once.

Figure. Average miss on three held-out flats at three alphas. The middle bar is shortest: alpha 1 wins because Rs 1,000 is smaller than Rs 3,000 and Rs 2,000. Shorter is better.

1. Try a fewPick a short list of alphas — say 0.01 (gentle), 1 (medium), 100 (harsh).
2. Score eachFit on training flats only. Average the absolute misses on the validation flats.
3. Keep the winnerThe alpha with the smallest validation average miss is the one you keep.

One alpha, or a contest of alphas

from sklearn.linear_model import Ridge, RidgeCV

# Ridge: you already chose one alpha.
model = Ridge(alpha=1)

# RidgeCV: try the list, keep the winner.
chooser = RidgeCV(alphas=[0.01, 1, 100])

4Two ways to charge size: L2 and L1

There are two standard ways to charge a model for the size of its weights — the learned numbers it hangs on each clue. L2, nicknamed ridge, adds up the squares. L1, nicknamed lasso, adds up the absolute values instead: |w_1| + |w_2| + \cdots. Absolute value means "drop the sign, keep the size": |-3| = 3, |0.5| = 0.5.

That small change has a striking consequence. L2 shrinks every weight toward zero but almost never lands one exactly on zero — every feature stays in, just tamed. L1 can push some weights to exactly zero. A feature whose weight is exactly zero contributes nothing: it has been deleted from the model. So L1 doubles as automatic feature selection.

No diagram here — the table names the two size-bills; the diamond figure follows.

5Why L1 zeroes and L2 only shrinks

The figure below is the standard picture of why L1 zeroes weights and L2 does not. Its two axes are just two of the model's weights, w_1 and w_2 — every point in the picture is one possible pair of weight values. The penalty acts like a budget: the model must pick its weights from inside a fixed region around the origin. For L1 that region is a diamond, whose sharp corners sit exactly on the axes — and a corner on an axis is a point where one weight is exactly zero. For L2 the region is round, with no corners. The dashed oval is one contour of the training loss with no penalty: points on it fit the training data equally well, and the best unpenalised fit sits at its centre, outside the budget. The chosen weights are where that contour first touches the budget region. A round region gets touched at some generic off-axis point (both weights shrink, neither dies); a diamond very often gets touched at a corner — which is precisely a weight hitting exactly zero.

Figure. Regularisation is a budget on weight size in (w1, w2) space. The diamond is L1 (lasso): its corners sit on the axes, so the first loss contour that meets it often lands on an axis and drives a weight exactly to zero. The rounded constraint is L2 (ridge) - same Euclidean radius, drawn as a regular polygon because this painter has no circle primitive - and it meets the loss off-axis, shrinking both weights without exact zeros. The dashed ellipse is one contour of the unpenalised training loss, drawn around the best no-penalty fit; a stronger penalty shrinks the allowed set toward the origin.

6Lab: compare C on coefficients

Time to watch the shrink happen instead of taking it on faith. The plan: build one classification table, split it once into training rows and held-out test rows, then fit the same LogisticRegression twice on the identical training rows — once with C=1.0 (a mild penalty) and once with C=0.01. Remember the flipped dial from the previous concept: in sklearn, smaller C means a stronger penalty, so C=0.01 is the heavily regularised model, one hundred times stronger than C=1.0. Since everything else is identical, any difference between the two fits is the penalty's doing.

Two numbers get printed for each model, so here is exactly what each one means. First, coef_ is sklearn's name for the array of learned weights — one number per feature, the same weights the penalty is squeezing (in the code, coef_weak holds the C=1.0 weights and coef_strong the C=0.01 weights). Squinting at ten raw weights is awkward, so we summarise each array by its L2 norm, written \|w\|_2: square every weight, add them up, take the square root. It is one number that answers "how big are the weights overall?" — if the penalty is doing its job, the C=0.01 model's norm should be clearly smaller. Second, the hold-out score is plain accuracy measured on the test rows the fit never saw; it answers "did squeezing the weights cost us any real predictive power?".

The lab is two cells that share memory. Cell 1 does the fitting and printing, and deliberately leaves coef_weak and coef_strong alive in the kernel (the Python session running behind the notebook). Cell 2 picks those arrays up and draws a bar chart: for each of the ten features, two bars side by side showing the absolute size of that feature's weight under each C. The chart is where the story gets concrete — you can see which individual features had their weights pulled down, not just that some overall number fell. Two practical settings in Cell 1 are worth noticing: random_state=0 pins the random shuffling, so a re-run gives the same split and the same numbers, and n_samples stays in the hundreds so the in-browser Python stays quick.

Now the reading. Expect the bars to drop sharply at C=0.01 while the hold-out score barely moves. Resist the two tempting wrong conclusions: a big shrink with a flat score does not mean the strong penalty "broke" the model, and it does not prove the small-C model will generalise much better either. It means something quieter and more useful — the model never needed those large weights. It performs essentially as well while leaning on the data far more gently, and a simpler model that matches a complex one is the safer bet on rows nobody has seen yet.

No diagram — the shrink is drawn by the coding lab bar chart, not a static figure.

1. Same split, two C valuesFit LogisticRegression(C=1.0) and C=0.01 on identical train rows.
2. Compare normsPrint L2 norm of coefficients — stronger penalty should pull magnitudes down.
3. Read test scoreCheck whether the shrink cost is worth the generalisation gain on the held-out fold.

Coding lab. Weak vs strong C runs in the app, with checks on your output.