E ExamMaster

Engineering Mathematics · Calculus for ML

Chain Rule

In Engineering Mathematics because layered models are compositions of functions, and their training signal moves backward through those layers by the chain rule.

A composed system changes through its internal links. This topic follows derivatives through those links, from ordinary compositions to the scalar sigmoid chain that foreshadows backpropagation.

  • Engineering Mathematics
  • Medium level
  • 4 concepts

1Local changes multiply through the dependency chain

When one quantity depends on a second, which depends on a third, the total change from the start to the end is built from the local changes along that path. The chain rule is the formal statement of that idea.

This is exactly why it sits at the center of ML training. A neural network is a long composition of simple operations, and the effect of a parameter on the final loss must be traced through every intermediate dependency.

Figure. The chain rule multiplies local rates along a dependency path. The graph is topology only; the ledgers supply the values.

Differentiate a composition

  1. Name insideWrite the inner quantity as its own variable.
  2. Differentiate outsideFind how the output changes with the inner quantity.
  3. Multiply by inside rateMultiply by how the inner quantity changes with the original input.

Two-link composition

Let u=3x-1 and y=u^2. Find dy/dx at x=2.

  • u(2)=3(2)-15
  • dy/du2u
  • du/dx3
  • dy/dx=(dy/du)(du/dx) at u=52(5)(3)=30

Pro tip. The derivative is large because the outside square has slope 10 at u=5 and the inside line changes three units per x.

Let u=2x-3 and y=u^2. What is dy/dx at x=4?
  1. 20
  2. 10
  3. 5
  4. 4

At x=4, u=5. Chain rule gives dy/dx=2u*2=20.

2A scalar sigmoid chain has concrete local factors

A tiny one-neuron chain already shows the ML version of the chain rule. The weight w and input x create a pre-activation z=wx, and the sigmoid turns z into an output y=\sigma(z).

The derivative from w to y is the product of the sigmoid's local slope at z and the derivative of z with respect to w. Every number must match the chosen sigmoid value.

Figure. The scalar chain is w to z = wx to y = σ(z). With x = 2 and w = 0.5, z = 1 and y ≈ 0.731. The weight slope is σ'(1) times x, about 0.197 × 2 ≈ 0.394.

Compute dy/dw

  1. Forward valueCompute z=wx and the concrete y=\sigma(z).
  2. Activation slopeUse \sigma'(z)=\sigma(z)(1-\sigma(z)) at that same z.
  3. Weight factorMultiply by dz/dw=x.

Sigmoid chain with matched numbers

Use x=2, w=0.5, z=wx, and y=\sigma(z). With z=1, take the concrete value \sigma(1)\approx0.731. Compute dy/dw.

  • z=wx0.5(2)=1
  • y=\sigma(1)\approx 0.731
  • \sigma'(1)=0.731(1-0.731)0.731(0.269)\approx0.197
  • dy/dw=\sigma'(1)\,x0.197(2)\approx0.394

Pro tip. The sigmoid derivative is tied to the actual sigmoid value 0.731; changing that value changes the derivative.

Coding lab. Differentiate a scalar sigmoid chain runs in the app, with checks on your output.

If z=wx and y=sigma(z), which factor is dz/dw?
  1. x
  2. w
  3. sigma(z)
  4. 1-z

With respect to w, the input x is constant, so z=wx has derivative x.

3A loss derivative adds one more link

Training usually asks how a loss changes when a parameter changes, not just how a model output changes. That adds another link from output y to loss L.

The same chain rule applies. The loss slope with respect to w is the output-to-loss slope multiplied by the output-to-weight slope.

Figure. Squared-error loss adds one more link. From y ≈ 0.731 and target t = 1, dL/dy = 2(y − t) = −0.538. Times dy/dw ≈ 0.394 gives dL/dw ≈ −0.212, so a small increase in w lowers this loss.

Back up from loss

  1. Loss factorCompute dL/dy from the chosen loss.
  2. Output factorReuse dy/dw from the model chain.
  3. MultiplyCombine them as dL/dw=(dL/dy)(dy/dw).

Squared-error loss through the sigmoid output

Continue the sigmoid chain with y\approx0.731 and target t=1. For L=(y-t)^2, compute dL/dw using dy/dw\approx0.394.

  • dL/dy=2(y-t)2(0.731-1)=-0.538
  • dy/dw\approx0.394
  • dL/dw=(dL/dy)(dy/dw)(-0.538)(0.394)\approx-0.212
  • Descent readingSince dL/dw<0, increasing w slightly lowers this loss locally

Pro tip. The negative sign is not decoration: it says the current prediction is below the target, so increasing the weight helps.

Coding lab. Back a loss gradient through one activation runs in the app, with checks on your output.

For L=(y-t)^2, dL/dy is:
  1. 2(y-t)
  2. y-t
  3. 2t
  4. y(1-y)

Differentiate the squared error with respect to y: 2(y-t).

4Backpropagation reuses intermediate derivatives

Backpropagation is not a new calculus rule. It is an efficient way to store intermediate values from the forward pass and reuse local derivatives while moving backward through the computation.

The practical discipline is bookkeeping: name each intermediate value, compute each local derivative once, and multiply only along real dependency paths.

Figure. Backpropagation reuses the same dependency graph in reverse to pass derivatives back. The figure shows reuse of stored intermediates, not an animation of training.

Keep the chain auditable

  1. Forward cacheStore intermediate values such as z and y.
  2. Local derivativesCompute derivatives for each link using the cached values.
  3. Backward productMultiply local derivatives along the route from loss back to the parameter.

Reuse cached values in the scalar chain

For x=2, w=0.5, cached z=1, y\approx0.731, target t=1, reuse local factors to compute dL/dw.

  • Cached local factorsdL/dy=-0.538, dy/dz\approx0.197, dz/dw=2
  • dL/dz=(dL/dy)(dy/dz)(-0.538)(0.197)\approx-0.106
  • dL/dw=(dL/dz)(dz/dw)(-0.106)(2)\approx-0.212

Pro tip. This gives the same derivative as multiplying all factors at once, but the staged form is easier to audit in larger graphs.

Why cache forward values like z during backprop?
  1. Local derivatives such as sigmoid slope depend on those values
  2. Caching removes the chain rule
  3. It changes the target
  4. It sets gradients to zero

Backprop reuses each forward intermediate to compute the matching local derivative.

Notes

  • The chain rule differentiates a composition by multiplying local rates from each stage.
  • In layered models, each intermediate output becomes the input to the next transformation.
  • Backpropagation is repeated chain-rule bookkeeping over a graph of intermediate values.

Formulas

  • If y = f(g(x)), then dy/dx = f'(g(x)) g'(x).
  • For z = wx and y = sigma(z), dy/dw = sigma'(z) x.
  • For sigmoid, sigma'(z) = sigma(z)(1 - sigma(z)).

Exam traps & shortcuts

  • Trace the dependency path first; the chain rule follows that path one local derivative at a time.
  • Write every intermediate value before multiplying derivatives; it prevents missing a link.

Reference tables

Each chain-rule factor belongs to one dependency arrow.

Scalar chain factors
ArrowLocal derivative
w \rightarrow z=wxdz/dw=x
z \rightarrow y=\sigma(z)dy/dz=\sigma(z)(1-\sigma(z))
y \rightarrow L=(y-t)^2dL/dy=2(y-t)
w \rightarrow LdL/dw=(dL/dy)(dy/dz)(dz/dw)

Recap

Composition is the reason backpropagation works.

Path
The chain rule follows the dependency path from input to output.
ML link
Backpropagation is repeated chain-rule bookkeeping across layers.
Sigmoid
For z=1, \sigma(1)\approx0.731 gives \sigma'(1)\approx0.197, so the derivative values must match that forward value.
Loss
A loss derivative adds the output-to-loss factor to the same chain.

Practise Chain Rule

Reading is free and needs no account. Practice, mocks and progress live in the app.

  • A 5-question practice set that ends the chapter
  • 4 quick checks with worked explanations
  • Timed mocks scored with the real marking scheme
  • Readiness tracked per topic, kept on your device
Continue with Google — freeNo card, no trial. Works offline once installed.