Machine Learning · Machine Learning Core
Training Loss
A loss is one number for how wrong a model is on the rows it studied — from a single miss to the training-loss score the fit minimises.
Training loss is the one number that says how wrong a model is on the rows it studied. Picture a rent office: two hundred flats already rented, each with a size, a distance from the centre, and the monthly rent the tenant signed. Two candidate lines, two hundred pairs of guess and true rent, and no way to pick a winner by staring. This lesson names that missing number. We now have one number for how wrong.
- Machine Learning
- Easy level
- 5 concepts
- 6 practice questions
1You need a number for "how wrong"
Picture a rent office. Two hundred flats already rented, each with a size, a distance from the centre, and the monthly rent the tenant signed. Two people each draw a candidate rent line — a recipe that turns those facts into a guessed rent. Which line is better? You cannot answer by staring at 200 pairs of (guess, true rent). You need one number that answers "how wrong is this line, on the flats it studied?"
That number is called a loss. Smaller is better: a loss of 0 would mean every guess matched the true rent exactly. The job of training — the step that picks the numbers inside the line, called weights — is to make this number as small as it can.
Figure. Two candidate rent lines each produce 200 pairs of (guess, true rent). You cannot pick the better line by reading the pairs. A loss is the single number that answers how wrong a line is on the flats it studied.
| Question | What answers it |
|---|---|
| How wrong is this one guess? | That row's miss — true rent minus guess |
| How wrong is this line on the flats it studied? | One loss number for the whole training pile |
| Which of two lines should training keep? | The one with the smaller loss |
Why does a machine learning training loop require a formal scalar loss function?
- It replaces the need to collect feature data from real-world samples
- It automatically prevents the model from ever overfitting on training rows
- It condenses total prediction error across all rows into a single scalar that gradient algorithms can minimise
- It converts multi-class classification problems into single-variable linear equations
Optimization algorithms need a single numeric objective function to calculate gradients and update parameter weights toward lower error.
2One flat, one miss
Start with a single training flat, and name every number. The true rent y is what the tenant actually paid — say ₹8,000 a month, written 8 in thousands of rupees so the arithmetic stays small. The guess is what the candidate line produced for that flat — say ₹7,000, written 7. That guess has a name: \hat{y}, y-hat, the hat meaning estimate. The miss is y - \hat{y} = 8 - 7 = +1. That +1 means the guess was ₹1,000 too low. This one-row miss is a fit-error, also called a residual.
A miss can be positive (guess too low) or negative (guess too high). Both are wrong. A miss of 0 means the guess matched the true rent on that one flat.
Figure. One training flat, rents in thousands of rupees. True rent y is 8 (₹8,000). The line guessed ŷ = 7 (₹7,000). The miss y − ŷ = +1 means the guess was ₹1,000 too low. That one-row miss is a fit-error.
Name the miss on one flat
A 2-bed flat 8 km out has true monthly rent y = 12 (₹12,000). A candidate line guesses \hat{y} = 13. What is the miss, and what does the sign say?
- Miss = true rent minus guess: 12 - 13-1
- The minus sign: the guess 13 is larger than the true rent 12guess was ₹1,000 too high
Pro tip. Always write miss as y - \hat{y}, true minus guess. Swapping the order flips the sign and makes a later sum lie.
A flat's true rent is Rs 8,000 and the model predicts Rs 10,000. In units of thousands, what is the single-row squared error?
- 4, because the residual is +2 thousand rupees and 2 squared is 4
- 2, because the error is simply the raw difference of Rs 2,000
- 80, because the prediction is multiplied by the ground-truth rent
- 0.25, because the error is divided by the target magnitude
In units of thousands of rupees, residual e = 10 - 8 = 2. The squared error is e^2 = 2^2 = 4.
3Many misses become one training-loss number
One miss is not enough. A line can be perfect on one flat and terrible on the next. So you score every training row, then combine those misses into a single number. That single number is the training loss: how wrong the model is on the pile it studied.
If you just add the raw misses, a plus one and a minus one cancel, and a sloppy line can look perfect. Both flats were ₹1,000 wrong — they just missed on opposite sides. Adding them reports zero. The usual fix for a rent line is to square each miss before adding, then divide by the number of training rows. Squaring makes every miss positive, so plus one and minus one both cost one, and a large miss hurts more than a small one.
Figure. Three training misses go in. Each is squared so a too-high guess cannot cancel a too-low one. The average of those squared misses is the training loss — one number for the whole pile the model studied.
| Piece | What it is |
|---|---|
| Fit-error / miss | True rent minus guess, on one training flat |
| Squared miss | That miss times itself — so +1 and −1 both cost 1 |
| Training loss | The average of the squared misses across the training pile |
Why is Mean Squared Error averaged over N examples rather than simply summed?
- Summing errors causes an arithmetic overflow in floating-point hardware
- Averaging forces all individual row residuals to equal zero exactly
- The optimizer cannot compute gradients unless the divisor is N squared
- Averaging keeps the loss scale invariant to dataset size, so adding more training rows does not artificially inflate the error
Dividing by N makes the loss an expectation that remains comparable across datasets of different sizes, preventing sample count from altering the gradient magnitude.
4Three training flats, one score
Work the whole recipe on three training flats from the rent office. Rents are in thousands of rupees, so a miss of 1 means the guess was ₹1,000 off. True rents y = [8, 10, 12]. Line A, a candidate rent line, guesses \hat{y} = [7, 10, 13]. Walk each flat: the miss, the squared miss, then the average — that average is this line's training loss.
A second candidate, Line B, always guesses 10 — the middle rent, ignoring size and distance. Misses -2, 0, +2; squared 4, 0, 4; training loss 8/3. The first line's 2/3 is smaller, so training keeps the first line. That is all the ordinary fit answers to — the fit that only cares about missing the rents: which candidate has the smaller training loss.
| Flat | True y | Guess ŷ | Miss y−ŷ | Squared |
|---|---|---|---|---|
| Studio, 1 km | 8 | 7 | +1 | 1 |
| 1-bed, 3 km | 10 | 10 | 0 | 0 |
| 2-bed, 8 km | 12 | 13 | −1 | 1 |
| Training loss | (1+0+1)/3 = 2/3 |
Two lines, one training-loss verdict
Same three training flats, true rents y = [8, 10, 12] in ₹ thousands. Line A guesses [7, 10, 13]. Line B always guesses 10. Compute each line's training loss — the mean of squared misses — and say which line training keeps.
- Line A misses: 8-7, 10-10, 12-13+1, 0, -1
- Line A squared misses: 1^2 + 0^2 + (-1)^22
- Line A training loss: 2 divided by 3 training rows2/3
- Line B misses: 8-10, 10-10, 12-10-2, 0, +2
- Line B squared misses: (-2)^2 + 0 + 2^2 = 4+0+4, then divide by 38/3
- Smaller training loss wins. 2/3 against 8/3keep line A
Pro tip. Line B is the "always guess the average" baseline. Training loss is how you prove a line beat it — not by staring at the three pairs.
On 3 flats, individual squared errors are 1, 4, and 4. What is the Mean Squared Error for this batch?
- 9.0, obtained by summing the raw squared deviations without dividing
- 3.0, obtained by taking the arithmetic mean (1 + 4 + 4) / 3 = 9 / 3 = 3
- 2.0, obtained by finding the median squared error of the three flats
- 4.0, obtained by taking the maximum squared error in the batch
The MSE is (1 + 4 + 4) / 3 = 9 / 3 = 3.0.
5The spam filter needs a number too
The same job shows up in a spam filter. Each training email is a row whose true label is already known — spam or not spam. The filter outputs a probability, not a rent. How wrong is still a number. A guess of 0.9 spam on a real meeting invite is more wrong than a guess of 0.55, even though both would flip to "spam" at a 0.5 cut. Training loss is the average of those per-email prices across the emails the filter studied.
You do not need the log-loss formula to use the idea. Remember the name and the job: training loss is one number for how wrong the model is on the training pile. The ordinary fit's only job is to make that number small — the line we already drew, the one that only cares about missing the rents. That is the whole idea: we now have one number for how wrong.
| World | One-row miss | Training loss |
|---|---|---|
| Rent line | True rent minus guess | Average of the squared misses on the training flats |
| Spam filter | How far the predicted spam chance was from the true label | Average of those per-email prices on the training emails |
Why is Mean Squared Error rarely used as the training loss for binary classification?
- MSE cannot be mathematically evaluated when targets are 0 and 1
- Log loss always produces an error of zero for any linear model
- Log loss heavily penalises confident wrong predictions and produces non-zero gradients across the probability range
- Classification algorithms cannot run unless the loss is quadratic
When paired with a sigmoid output, log loss (binary cross-entropy) avoids gradient saturation and imposes severe logarithmic penalties on confident misclassifications.
Notes
- A loss is a number that answers "how wrong is this guess?" — smaller is better.
- A one-row miss (true rent minus guess) is a fit-error; training loss is that idea turned into one number for the whole training pile.
- Training's only job is to make training loss small — pick the line that missed the rents the least.
Exam traps & shortcuts
- Name what each number is — true rent, guess, miss, squared miss, average — before adding.
- Training loss is scored only on the rows the model studied, never on the locked test pile.
Recap
This lesson in brief:
- A loss
- A loss is one number that answers "how wrong is this model?" — smaller is better.
- Fit-error vs training loss
- A fit-error is one row's miss. Training loss is those misses combined into one number for the pile the model studied.
- What training does
- Training keeps the candidate with the smaller training loss — the ordinary fit, the one that only cares about missing the rents.
Practise Training Loss
Reading is free and needs no account. Practice, mocks and progress live in the app.
- 6 exam-style questions on this topic, with explanations
- A 5-question practice set that ends the chapter
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device