E ExamMaster

Engineering Mathematics · Linear Algebra

Linear Algebra in ML

In Engineering Mathematics because datasets, embeddings, and weight matrices are all linear-algebra objects before they become code.

This bridge topic ties the linear-algebra vocabulary back to machine learning. It shows how data matrices, parameter vectors, and embeddings reuse the same structures from the earlier lessons.

  • Engineering Mathematics
  • Medium level
  • 5 concepts

1ML data structures are linear algebra objects

Many machine-learning objects look like software containers at first: arrays, tensors, tables, and weight files. Under the hood they are usually vector-space objects with coordinates, dimensions, and linear operations that tell you what the code is really doing.

Seeing that structure early prevents memorized formulas later. A batch is a matrix, a parameter set is a vector, a layer is often a matrix transform, and similarity often comes from a dot product or projection idea you already know.

The batch-score ledger maps three feature rows to three scalar model scores.

ML object as linear algebra
ML objectLinear algebra reading
one examplefeature vector
mini-batchmatrix of feature rows
weights for one scoreweight vector
dense layerweight matrix

Batch scores from a data matrix

Let X have rows (1,2), (3,1), and (0,4), and let w=(0.5,2). Compute Xw.

  • Row 1 score: 1\cdot0.5+2\cdot20.5+4=4.5
  • Row 2 score: 3\cdot0.5+1\cdot21.5+2=3.5
  • Row 3 score: 0\cdot0.5+4\cdot20+8=8
  • Xw(4.5,3.5,8)

Pro tip. This is the same matrix-vector multiply a model uses to score a whole mini-batch.

Data matrix X has rows (2, 0), (1, 3) and weights w=(0.5, 1). What is Xw?
  1. (1, 3.5)
  2. (1, 3)
  3. (2.5, 3)
  4. (1, 4)

Scores: 2\cdot0.5+0=1 and 1\cdot0.5+3\cdot1=3.5.

2Embeddings compare learned coordinates with dot products

An embedding is a vector whose coordinates were learned rather than named by a human. Even when individual coordinates are hard to interpret, the vector still supports geometric operations.

Similarity search often uses dot product or cosine similarity. The model supplies the embedding coordinates; linear algebra supplies the comparison rule.

Figure. Embedding similarity is geometric alignment. Items A and B point in similar directions, while C points elsewhere, so dot-product similarity separates them.

Compare two embeddings

  1. DotMultiply matching coordinates and add.
  2. LengthsCompute each embedding norm if cosine similarity is needed.
  3. NormalizeDivide the dot product by the product of lengths.

Cosine similarity for tiny embeddings

A query embedding is q=(1,2,0) and a document embedding is d=(2,1,1). Compute cosine similarity.

  • q\cdot d=1\cdot2+2\cdot1+0\cdot14
  • \|q\|=\sqrt{1^2+2^2+0^2}\sqrt5
  • \|d\|=\sqrt{2^2+1^2+1^2}\sqrt6
  • \cos\theta=4/(\sqrt5\sqrt6)4/\sqrt{30}\approx0.730

Pro tip. The coordinates are learned, but the similarity computation is the same normalized dot product as before.

Embeddings q=(2, 0, 1) and d=(1, 2, 2). What is q\cdot d?
  1. 4
  2. 5
  3. 3
  4. 2

2\cdot1 + 0\cdot2 + 1\cdot2 = 4. Dot product is the basic similarity score before optional cosine normalization.

3A weight matrix turns one activation vector into the next

A dense layer can be read as a matrix transform from an input activation vector to an output activation vector. Each row of the weight matrix is the recipe for one output unit.

Biases and nonlinear activations often come next in a real network, but the linear core is this weight-matrix multiply.

Figure. A layer is a matrix path from one activation vector to the next. The figure keeps the data-flow topology visible without pretending to animate the multiply.

Run one dense layer

  1. InputWrite the activation vector in the coordinate order the layer expects.
  2. RowsDot each weight row with the input vector.
  3. OutputCollect one coordinate per output unit before any activation function.

One numeric layer path

Let W=\begin{bmatrix}1&-1&2\\0.5&0&1\end{bmatrix} and input x=(2,3,1). Compute y=Wx.

  • Output 1: 1\cdot2 + (-1)\cdot3 + 2\cdot12-3+2=1
  • Output 2: 0.5\cdot2 + 0\cdot3 + 1\cdot11+0+1=2
  • y=Wx(1,2)

Pro tip. The layer has two output units because the weight matrix has two rows.

Let W=\begin{bmatrix}1&0&-1\\2&1&0\end{bmatrix} and x=(1, 2, 3). What is Wx?
  1. (-2, 4)
  2. (2, 4)
  3. (-2, 5)
  4. (1, 4)

Outputs: 1+0-3=-2 and 2+2+0=4.

4A mini-batch can pass through a weight matrix at once

Frameworks usually run many examples through the same layer together. With rows-as-examples convention, the batch output is another matrix: one output row for each input row.

This is not a new operation. It is the same row-by-row dot-product logic, now with several output columns produced for each example.

Figure. Two example rows pass the same weight matrix at once. Row (1, 0, 2) scores (1, 6). Row (0, 3, 1) scores (6, 0). The output keeps two rows and two columns.

Multiply batch by weights

  1. Batch rowsEach row is one example's input vector.
  2. Weight columnsEach output column is one unit's weight vector.
  3. Output matrixEntry (i,j) is row i dotted with output-unit j.

Two examples through two output units

Let X=\begin{bmatrix}1&0&2\\0&3&1\end{bmatrix} and W=\begin{bmatrix}1&0\\2&-1\\0&3\end{bmatrix}. Compute XW.

  • Row 1, output 1: 1\cdot1+0\cdot2+2\cdot01
  • Row 1, output 2: 1\cdot0+0\cdot(-1)+2\cdot36
  • Row 2, output 1: 0\cdot1+3\cdot2+1\cdot06
  • Row 2, output 2: 0\cdot0+3\cdot(-1)+1\cdot30
  • XW\begin{bmatrix}1&6\\6&0\end{bmatrix}

Pro tip. The output keeps two rows because there are two examples, and two columns because there are two output units.

Mini-batch X=\begin{bmatrix}1&2\\0&1\end{bmatrix} and W=\begin{bmatrix}1\\-1\end{bmatrix}. What is XW?
  1. \begin{bmatrix}-1\\-1\end{bmatrix}
  2. \begin{bmatrix}1\\-1\end{bmatrix}
  3. \begin{bmatrix}3\\-1\end{bmatrix}
  4. \begin{bmatrix}1\\1\end{bmatrix}

Row scores: 1\cdot1+2\cdot(-1)=-1 and 0\cdot1+1\cdot(-1)=-1.

5Projection explains simple recommender scores

A recommender or retrieval model often asks how much a user vector points along an item vector. When the item vector is normalized, the dot product is a projection length.

That makes a score interpretable: it is not just a magic number from code, but the amount of the user representation aligned with the item direction.

Figure. With a unit item direction, the user–item score is a projection length. u = (6, 8) dotted with i = (0.6, 0.8) is 3.6 + 6.4 = 10. The boxes name the lists; they are not drawn at length 10 : 1.

Score by projection

  1. Normalize itemUse a unit item vector when the score should be a projection length.
  2. DotCompute user vector dot item direction.
  3. RankLarger signed projection means stronger alignment in that representation.

User vector projected onto an item direction

A user vector is u=(6,8) and a unit item direction is i=(0.6,0.8). Compute the alignment score.

  • Check i unit\sqrt{0.6^2+0.8^2}=1
  • u\cdot i=6\cdot0.6+8\cdot0.83.6+6.4=10
  • Projection length10

Pro tip. With a unit item vector, the dot product is literally the user's signed length along that item direction.

User u=(3, 4) and unit item direction i=(0.6, 0.8). What is the projection score u\cdot i?
  1. 5
  2. 4
  3. 3.2
  4. 7

3\cdot0.6 + 4\cdot0.8 = 1.8 + 3.2 = 5, the signed length of u along i.

Notes

  • A dataset is commonly arranged as a matrix whose rows are examples and whose columns are features.
  • Weights, embeddings, and latent representations are vectors or matrices with task-specific interpretations.

Formulas

  • If X is an n x d data matrix and w is in R^d, then X w produces one score per example.
  • A dense layer without bias can be written as y = W x or, for a batch, Y = X W.
  • Cosine similarity between embeddings is (a . b) / (||a|| ||b||).

Exam traps & shortcuts

  • When code shape feels abstract, write the object as vector, matrix, or subspace before reasoning about the algorithm.
  • Track whether examples live in rows or columns before reading any matrix multiply in code.
  • An embedding coordinate is learned, not hand-named; comparisons still use dot products, norms, and projections.

Recap

This subject closes by linking notation to ML artefacts.

Data
Rows, columns, vectors, and matrices are not new names in ML; they are the actual objects the code manipulates.
Transfer
Linear algebra becomes useful when you can rename an ML object as a vector, matrix, norm, or projection problem.
Embeddings
Learned coordinates still use dot products, norms, and cosine similarity for comparison.
Layers
A dense layer's linear core is a weight matrix multiplying an activation vector or batch matrix.

Practise Linear Algebra in ML

Reading is free and needs no account. Practice, mocks and progress live in the app.

  • A 5-question practice set that ends the chapter
  • 5 quick checks with worked explanations
  • Timed mocks scored with the real marking scheme
  • Readiness tracked per topic, kept on your device
Continue with Google — freeNo card, no trial. Works offline once installed.