E ExamMaster

Engineering Mathematics · Linear Algebra

Norms and Projections

In Engineering Mathematics because distance, error, and best-fit ideas in ML all depend on how you measure length and what it means to project onto a simpler subspace.

Once vectors exist, the next question is how to measure size and closeness. This topic introduces norms as length measures and projections as the nearest linear shadow of one vector onto a subspace.

  • Engineering Mathematics
  • Medium level
  • 5 concepts

1Length tells error; projection tells best linear shadow

A norm turns the vague idea of size into a rule. In data work that rule becomes a model of error, because once you can measure the length of a difference vector you can say how far a prediction is from the truth.

Projection then answers a second question: if you are only allowed to stay inside a chosen subspace, which point there is closest to the original vector? That is the geometric heart of approximation and least squares.

Figure. Projection drops a vector onto an allowed line and leaves a residual back to the original vector. This is the geometric core of least squares.

Measure residual size

  1. ResidualSubtract prediction from target or target from prediction, consistently.
  2. Choose normState whether the error is measured by L_2, L_1, or another norm.
  3. Compute sizeApply the norm formula to turn the residual vector into one number.

L2 norm of a residual

A model residual is r=(2,-1,2), using prediction minus target. Compute its Euclidean length.

  • \|r\|_2=\sqrt{2^2+(-1)^2+2^2}\sqrt{4+1+4}
  • 4+1+49
  • \sqrt93

Pro tip. The residual vector has several coordinate errors, but the L_2 norm reports one error magnitude.

Residual r=(3, -4, 0). What is its Euclidean (L_2) norm?
  1. 5
  2. 7
  3. 12
  4. 3

\sqrt{9+16+0}=5. Absolute-sum or max-component norms answer different error questions.

2Different norms ask different error questions

A norm is a measurement rule, not a neutral decoration. The L_1 norm adds absolute errors, the L_2 norm squares before summing, and the L_\infty norm watches only the largest coordinate error.

That choice changes what a model is encouraged to fix. Large single-coordinate errors feel different under L_2 than under L_1, even when the residual vector is the same.

The norm table and ledger show how the same residual becomes 5, 3, or 2 under different rules.

Common residual norms
NormFormula for r
L_1\sum_i |r_i|
L_2\sqrt{\sum_i r_i^2}
L_\infty\max_i |r_i|

One residual, three norm readings

For r=(-2,1,2), compute L_1, L_2, and L_\infty.

  • \|r\|_1=|-2|+|1|+|2|5
  • \|r\|_2=\sqrt{(-2)^2+1^2+2^2}\sqrt9=3
  • \|r\|_\infty=\max(2,1,2)2

Pro tip. The residual did not change; only the measurement rule changed.

Coding lab. Read one residual under three norms runs in the app, with checks on your output.

For r=(1, -2, 2), which triple (L_1, L_2, L_\infty) is correct?
  1. (5, 3, 2)
  2. (5, \sqrt{5}, 2)
  3. (1, 3, 2)
  4. (5, 3, 5)

L_1=1+2+2=5, L_2=\sqrt{1+4+4}=3, L_\infty=\max(1,2,2)=2.

3Projection length onto a unit vector is a dot product

Projection asks how much of a vector points along a chosen direction. When the direction vector has unit length, the signed projection length is simply the dot product with that unit vector.

This is why normalized directions are convenient. The length of the direction no longer contaminates the answer; only alignment with that direction remains.

The projection ledger verifies unit length, computes x\cdot u=5, and reconstructs the projected vector (3,4).

Project onto a unit direction

  1. Check unit lengthVerify \|u\|=1 before using x\cdot u as a length.
  2. DotCompute x\cdot u for the signed length.
  3. Vector projectionMultiply that scalar by u if the projected vector is needed.

Projection length onto a unit vector

Let x=(3,4) and u=(0.6,0.8). Since u is unit length, find the signed projection length and projection vector.

  • \|u\|=\sqrt{0.6^2+0.8^2}\sqrt{0.36+0.64}=1
  • x\cdot u=3\cdot0.6+4\cdot0.81.8+3.2=5
  • Projection length5
  • 5u(3,4)

Pro tip. Here x already points in the unit direction u, so the projection recovers all of x.

Let x=(4, 3) and unit u=(0.8, 0.6). What is the signed projection length x\cdot u?
  1. 5
  2. 4
  3. 3.2
  4. 7

4\cdot0.8 + 3\cdot0.6 = 3.2 + 1.8 = 5. For a unit vector, that dot product is the signed length.

4Projection leaves an orthogonal residual

Projecting onto a line splits a vector into an allowed part and a leftover residual. The leftover is perpendicular to the line direction.

That perpendicular residual is the reason projection solves nearest-point problems. If any allowed direction could still reduce the error, the residual would not be perpendicular yet.

Figure. At the best projection, the leftover error is perpendicular to the allowed subspace. A non-perpendicular residual could still be shortened by sliding along the line.

Split into projection plus residual

  1. CoefficientCompute (x\cdot a)/(a\cdot a) for the line direction a.
  2. ProjectionMultiply the coefficient by a.
  3. ResidualSubtract projection from x and check the residual dot direction is zero.

Project onto the line spanned by (1,2)

Let x=(3,1) and a=(1,2). Project x onto the line through a and check the residual.

  • (x\cdot a)/(a\cdot a)(3\cdot1+1\cdot2)/(1^2+2^2)=5/5=1
  • \operatorname{proj}_a x=1a(1,2)
  • Residual r=x-\operatorname{proj}_a x(3,1)-(1,2)=(2,-1)
  • r\cdot a2\cdot1+(-1)\cdot2=0

Pro tip. The zero dot product is the receipt that the residual is perpendicular to the projection line.

Coding lab. Project a vector and check the residual runs in the app, with checks on your output.

Project x=(2, 2) onto the line spanned by a=(1, 0). What is the residual x - \mathrm{proj}_a x?
  1. (0, 2)
  2. (2, 0)
  3. (1, 1)
  4. (0, 0)

Onto the x-axis, \mathrm{proj}=(2,0), so residual (0,2). It is orthogonal to a=(1,0).

5Least squares chooses the nearest allowed prediction

In least squares, the model cannot usually hit every target value exactly. It chooses the prediction vector inside the column space that gives the smallest residual length.

This is projection language in data form: the allowed subspace is made by model columns, the target is the observed vector, and the residual is the leftover after taking the nearest shadow.

Figure. Least squares compares squared residual lengths. r1 = (1, −2, 2) has squared L2 length 9. r2 = (2, 0, 2) has squared length 8. The smaller squared norm wins.

Read least squares geometrically

  1. Allowed spaceThe columns of the design matrix span the predictions the model can make.
  2. TargetThe observed y vector may sit outside that space.
  3. Nearest predictionThe fitted prediction is the projection of y into the allowed space.

Compare two candidate residuals

Two candidate predictions leave residuals r_1=(1,-2,2) and r_2=(2,0,2). Under L_2, which is closer?

  • \|r_1\|_2^2=1^2+(-2)^2+2^21+4+4=9
  • \|r_2\|_2^2=2^2+0^2+2^24+0+4=8
  • Compare lengths\sqrt8<\sqrt9
  • Closer predictionr_2 is smaller under L_2

Pro tip. Least squares compares squared residual lengths, so the smaller squared norm wins.

Residuals r_1=(2, -1, 2) and r_2=(0, 3, 0). Under L_2, which comparison is correct?
  1. They are equal: both have \|r\|^2=9
  2. r_1 is strictly smaller
  3. r_2 is strictly smaller
  4. Only L_1 can compare them

\|r_1\|^2=4+1+4=9 and \|r_2\|^2=9, so least squares sees them as equally good.

Notes

  • A norm is a rule that assigns a non-negative size to a vector.
  • Projection replaces a vector by the component that lies inside a chosen line or subspace.

Formulas

  • The Euclidean norm is ||x||_2 = sqrt(x_1^2 + ... + x_n^2).
  • The projection of x onto a nonzero vector u is proj_u(x) = ((x . u) / (u . u)) u.
  • If u is a unit vector, the signed projection length of x onto u is x . u.

Exam traps & shortcuts

  • Projection is about closest representation inside an allowed subspace, not about changing the original target.
  • Name the norm before comparing errors; different norms reward different kinds of mistakes.
  • When the projection direction is unit length, the dot product already is the signed length.

Recap

Distance and best fit enter together.

Norm
A norm measures vector size, and error often becomes the norm of a residual.
Projection
Projection picks the closest vector inside an allowed subspace.
Unit direction
For unit u, the projection length of x onto u is x\cdot u.
Residual
A projection residual is orthogonal to the subspace that was allowed.

Practise Norms and Projections

Reading is free and needs no account. Practice, mocks and progress live in the app.

  • A 5-question practice set that ends the chapter
  • 5 quick checks with worked explanations
  • Timed mocks scored with the real marking scheme
  • Readiness tracked per topic, kept on your device
Continue with Google — freeNo card, no trial. Works offline once installed.