E ExamMaster

Engineering Mathematics · Calculus for ML

Derivatives as Rates of Change

In Engineering Mathematics because learning curves, sensitivity, and optimization all begin with how one quantity changes as another changes.

Calculus starts by quantifying local change. This topic treats a derivative as the slope of the best nearby linear approximation, then reads that slope as the signal an optimizer uses when a parameter moves.

  • Engineering Mathematics
  • Medium level
  • 4 concepts

1Derivative means local responsiveness

A derivative measures how the output of a function responds to a tiny input change near a chosen point. The important word is local: it is a nearby linear view, not a summary of the whole function everywhere.

That local view is why derivatives matter in ML. When a loss changes as a parameter moves, the derivative is the first signal telling you which direction improves or worsens the objective.

Figure. The derivative is the slope of a local tangent, not the average shape of the entire curve. The tangent is checked at one point.

Read a derivative

  1. Name inputIdentify the variable being nudged, such as a parameter w.
  2. Name outputIdentify the quantity that responds, such as a loss L(w).
  3. Stay localInterpret the derivative only near the point where it was computed.

Polynomial loss slope at a parameter value

For the one-parameter loss piece L(w)=w^3-4w^2+4w+3, compute the derivative at w=2.

  • L'(w)3w^2 - 8w + 4
  • L'(2)=3(2)^2-8(2)+412-16+4
  • 12-16+40
  • L(2.1)-L(2)3.021-3=0.021, showing curvature even though the exact local slope at 2 is flat

Pro tip. A zero first derivative means the best linear approximation is flat at that point; it does not prove the whole loss is flat.

For L(w)=w^2-6w+10, what is the local derivative at w=4?
  1. 2
  2. 4
  3. -2
  4. 10

L'(w)=2w-6, so L'(4)=8-6=2. The value is local to w=4.

2The sign of the derivative gives a local direction

Once the derivative is known, its sign says how the output changes for a small positive input move. A positive derivative means the output rises locally; a negative derivative means it falls locally.

For training, this turns into a decision about which way to move a parameter. If increasing w raises loss, a descent step should move w the other way.

The local sign picture is a small parameter step: at w=3, the right side rises and the left side lowers the loss.

Use the sign

  1. Compute slopeFind dL/dw at the current value of w.
  2. Read signPositive means a right-step raises loss; negative means a right-step lowers loss.
  3. Choose descentMove opposite the slope if the goal is to reduce loss.

One local descent decision

Let L(w)=w^2-4w+5. At w=3, decide which way a small descent step should move.

  • L'(w)2w-4
  • L'(3)=2(3)-42
  • L(3)9-12+5=2
  • L(2.9)8.41-11.6+5=1.81, lower than 2

Pro tip. The derivative is positive at w=3, so moving left is the local loss-reducing direction.

At w=-1, a loss has derivative -5. Which small move lowers loss locally?
  1. Increase w
  2. Decrease w
  3. Keep w fixed because derivative is nonzero
  4. Move in either direction equally

A negative derivative means a small positive step lowers the loss; descent moves opposite the uphill sign.

3A derivative builds a nearby linear estimate

The derivative is not only a slope label. It lets you estimate the function a short distance away by replacing the curve with its tangent line near the current point.

This approximation is the mental model behind many optimizer updates. The model says, 'if I move by this tiny amount, the loss should change by slope times step', before curvature corrections appear.

Figure. A linear approximation borrows the tangent line near the base point. It is useful locally and visibly drifts from the curve farther away.

Approximate a nearby value

  1. Start valueRecord the current function value f(a).
  2. Local slopeCompute f'(a) at the same point.
  3. Add slope times stepUse f(a+h) \approx f(a)+f'(a)h for a small h.

Estimate a loss after a small move

For L(w)=w^2+2w, estimate L(1.05) using the derivative at w=1, then compare with the exact value.

  • L(1)1^2+2(1)=3
  • L'(w)=2w+2, so L'(1)4
  • L(1.05) \approx L(1)+L'(1)(0.05)3+4(0.05)=3.20
  • L(1.05) exact1.1025+2.10=3.2025

Pro tip. Small-step predictions are useful because the error is tiny near the point, but they should not be trusted for large jumps.

If f(2)=7 and f'(2)=-3, estimate f(2.1) by linear approximation.
  1. 6.7
  2. 7.3
  3. 4.0
  4. 6.9

Use f(a+h) approx f(a)+f'(a)h = 7 + (-3)(0.1)=6.7.

4Finite differences sanity-check a derivative

A finite difference estimates a derivative by measuring the output change across a small but nonzero input step. It is not the definition's limit, but it is often the easiest way to catch a derivative that was typed or derived incorrectly.

In ML code, gradient checking uses this idea: compare an analytic derivative against a small numerical nudge before trusting a training loop.

Figure. A centered check around w = 2 uses L(2.01) = 0.0101 and L(1.99) = −0.0099. Their difference over 2h = 0.02 recovers the analytic slope L'(2) = 1. The three w marks are a local zoom, not a full number line.

Check a slope numerically

  1. Pick a small stepChoose a small h, such as 0.01, that is not so tiny that rounding dominates.
  2. Evaluate both sidesCompute L(w+h) and L(w-h) around the point.
  3. CompareUse the centered difference and compare it with the analytic derivative.

Centered finite-difference check

For L(w)=w^2-3w+2, check the derivative at w=2 using h=0.01.

  • L'(w)=2w-3, so L'(2)1
  • L(2.01)4.0401-6.03+2=0.0101
  • L(1.99)3.9601-5.97+2=-0.0099
  • (L(2.01)-L(1.99))/(2h)(0.0101-(-0.0099))/0.02=1

Pro tip. A centered finite difference is a practical audit of a derivative formula, not a replacement for understanding the formula.

Coding lab. Check a derivative numerically runs in the app, with checks on your output.

For L(w)=w^2, centered difference with h=0.1 at w=3 equals?
  1. 6
  2. 3
  3. 9
  4. 0.2

(L(3.1)-L(2.9))/0.2=(9.61-8.41)/0.2=6, matching L'(3)=6.

Notes

  • A derivative measures how sensitively one quantity responds to a small change in another.
  • The derivative is local: it describes nearby behavior, not the entire curve at once.
  • For a loss function, the derivative with respect to a parameter estimates how the loss changes under a tiny parameter step.

Formulas

  • f'(x) = lim_{h -> 0} (f(x + h) - f(x)) / h.
  • For a small step Delta x, f(x + Delta x) approx f(x) + f'(x) Delta x.
  • A negative derivative means the function falls as the input increases locally; a positive derivative means it rises locally.

Exam traps & shortcuts

  • Whenever a derivative appears, ask 'change of what with respect to what?' before computing.
  • Do not read one derivative value as the shape of the whole curve; it is a local statement at one point.

Reference tables

The same derivative value supports several ML-facing interpretations.

Derivative readings
ReadingQuestion it answers
RateHow fast is the output changing right here?
SignDoes a tiny positive input move raise or lower the output?
Linear estimateWhat nearby value should I expect for a small step?
Gradient checkDoes the analytic slope match a numerical nudge?

Recap

Derivative language is about local responsiveness.

Rate
A derivative is a local rate of change, not a full-curve description.
Sign
The derivative sign tells whether a small positive parameter move raises or lowers loss.
Estimate
A tangent-line estimate predicts small moves with f(a+h) \approx f(a)+f'(a)h.
Audit
Finite differences are a useful way to sanity-check analytic gradients.

Practise Derivatives as Rates of Change

Reading is free and needs no account. Practice, mocks and progress live in the app.

  • A 5-question practice set that ends the chapter
  • 4 quick checks with worked explanations
  • Timed mocks scored with the real marking scheme
  • Readiness tracked per topic, kept on your device
Continue with Google — freeNo card, no trial. Works offline once installed.