E ExamMaster

Engineering Mathematics · Optimization

Regularization Tradeoff

In Engineering Mathematics because the best fit to training data is not always the best model once complexity and generalization are accounted for.

Optimization in ML rarely minimizes pure training error alone. Regularization rewrites the objective so fit is balanced against simplicity or stability.

  • Engineering Mathematics
  • Medium level
  • 4 concepts

1Regularization adds a penalty to the loss

Regularization changes the objective being minimized. Instead of minimizing training loss alone, the optimizer minimizes training loss plus a penalty for unwanted complexity.

With L2 regularization, large weights cost more because the penalty is proportional to \lVert w\rVert^2.

Figure. Regularization changes the objective by adding a penalty term to the fit loss. The regularized score is fit plus complexity cost.

Build the regularized score

  1. Data fitStart with the ordinary training loss L(w).
  2. PenaltyCompute \lambda\lVert w\rVert^2.
  3. TotalAdd them to get the objective actually minimized.

Compute an L2 penalty

A one-weight model has w=3, training loss L(w)=0.2, and \lambda=0.1. Compute the regularized objective.

  • \lVert w\rVert^23^2=9
  • \lambda\lVert w\rVert^20.1\times9=0.9
  • J_{reg}(w)=L(w)+\lambda\lVert w\rVert^20.2+0.9=1.1

Pro tip. The optimizer sees 1.1, not just the attractive training loss 0.2.

Coding lab. Add an L2 penalty to a loss runs in the app, with checks on your output.

With L=0.4, w=2, \lambda=0.25, ridge-style L+\lambda w^2 equals?
  1. 1.4
  2. 0.65
  3. 0.4
  4. 2

0.4+0.25\cdot4=0.4+1=1.4.

2Regularization can change which model wins

A more flexible model can have lower training loss and still lose after regularization. The penalty makes complexity part of the comparison.

This is not a correction after optimization; it is the objective the optimizer is asked to minimize from the start.

A side-by-side comparison would show candidate A with a small fit bar and large penalty bar, and candidate B with a larger fit bar but small penalty bar.

Compare candidates

  1. Candidate ACompute fit plus penalty for the large-weight model.
  2. Candidate BCompute fit plus penalty for the smaller-weight model.
  3. WinnerChoose the lower total objective, not the lower data loss alone.

Unregularized versus regularized winner

Candidate A has w=3, L=0.2. Candidate B has w=1, L=0.7. With \lambda=0.1, compare unregularized and regularized objectives.

  • Unregularized comparisonA wins because 0.2<0.7
  • J_{reg}(A)0.2+0.1(3^2)=1.1
  • J_{reg}(B)0.7+0.1(1^2)=0.8
  • Regularized comparisonB wins because 0.8<1.1

Pro tip. The regularized objective chooses a slightly worse fit because the large weight was too expensive.

A: L=0.2, w=4. B: L=0.6, w=1. With \lambda=0.2 on w^2, who wins?
  1. B: 0.6+0.2=0.8 beats A's 0.2+3.2=3.4
  2. A still wins on regularized loss
  3. They tie
  4. Regularization ignores w

Penalty flips the ranking toward the smaller weight.

3\lambda sets the fit-versus-simplicity tradeoff

The regularization strength \lambda decides how heavily the penalty counts. When \lambda=0, the objective is pure training loss; as \lambda grows, large weights become more expensive.

Choosing \lambda is therefore a modeling decision. Too little regularization can overfit; too much can underfit by suppressing useful signal.

Figure. As lambda grows, the penalty has more influence, so the chosen model tends to move toward lower complexity.

Vary the penalty

  1. \lambda=0Only the data loss matters.
  2. Moderate \lambdaFit and weight size both influence the winner.
  3. Large \lambdaSmall weights dominate even if fit worsens.

When does the winner switch?

Using A: w=3, L=0.2 and B: w=1, L=0.7, find the \lambda where their regularized objectives tie.

  • Tie condition0.2+9\lambda=0.7+\lambda
  • Collect terms8\lambda=0.5
  • \lambda0.0625
  • At \lambda=0.1B wins because 0.1>0.0625

Pro tip. Below 0.0625, the fit advantage of A is still worth the penalty. Above it, B's smaller weight wins.

Coding lab. Plot fit-versus-penalty vs lambda runs in the app, with checks on your output.

Larger \lambda typically:
  1. Emphasizes the penalty / simpler weights more
  2. Deletes the training loss
  3. Forces \eta=0
  4. Makes all models identical

\lambda dials fit-versus-penalty strength.

4Validation data chooses the tradeoff strength

The training objective uses \lambda, but the usefulness of that choice is judged on held-out data. A regularization value that lowers the regularized training objective can still be too strong or too weak for generalization.

This is why \lambda is a hyperparameter: it is chosen outside the gradient update, usually by comparing validation performance.

A validation comparison would list each \lambda beside its held-out loss and select the smallest validation value.

Choose \lambda

  1. Train candidatesFit models under several \lambda values.
  2. Score validationEvaluate the same held-out metric for each candidate.
  3. SelectKeep the \lambda whose validation score is best.

Pick by validation loss

Three runs use \lambda=0, 0.1, and 1.0. Their validation losses are 1.4, 0.9, and 1.3 respectively. Choose the regularization strength.

  • Best validation loss0.9
  • Corresponding \lambda0.1
  • Gap to no regularization1.4-0.9=0.5
  • Gap to too much regularization1.3-0.9=0.4

Pro tip. The middle value balances fit and restraint best on held-out data in this example.

Why choose \lambda on validation loss, not training loss alone?
  1. We care about generalization under the tradeoff, not just fitting train
  2. Validation ignores labels
  3. Training loss cannot be computed with \lambda
  4. Bayes theorem requires it

Validation estimates held-out risk for each tradeoff setting.

Notes

  • Regularization adds a penalty or constraint so that optimization does not reward complexity without cost.
  • The tuning question is not whether fit matters, but how much fit should be traded for generalization or robustness.
  • An L2 penalty makes large weights expensive by adding \lambda\lVert w\rVert^2 to the data loss.

Formulas

  • J_{reg}(\theta)=J(\theta)+\lambda\Omega(\theta).
  • L2 regularization commonly uses J_{reg}(w)=L(w)+\lambda\lVert w\rVert^2.

Exam traps & shortcuts

  • A lower training loss can still signal a worse model if it was bought with fragile or overly complex parameters.
  • A regularized objective can choose a model with slightly higher data loss because the total objective is lower.

Reference tables

The regularized winner can differ from the pure-loss winner.

Regularization comparison
CandidateTraining loss\lambda\lVert w\rVert^2 at \lambda=0.1Total
A: w=30.20.91.1
B: w=10.70.10.8

Recap

Regularization changes the objective so the model pays for complexity.

Objective
J_{reg}(w)=L(w)+\lambda\lVert w\rVert^2 for L2 regularization.
Penalty
With w=3, L=0.2, and \lambda=0.1, the regularized score is 1.1.
Winner switch
A model with loss 0.7 and w=1 beats a model with loss 0.2 and w=3 once the L2 penalty is included.
Tradeoff
\lambda controls how much fit is traded for smaller, more stable weights.

Practise Regularization Tradeoff

Reading is free and needs no account. Practice, mocks and progress live in the app.

  • A 5-question practice set that ends the chapter
  • 4 quick checks with worked explanations
  • Timed mocks scored with the real marking scheme
  • Readiness tracked per topic, kept on your device
Continue with Google — freeNo card, no trial. Works offline once installed.