E ExamMaster

Engineering Mathematics · Calculus for ML

Partial Derivatives and Gradients

In Engineering Mathematics because real models depend on many variables, and optimization needs one derivative per coordinate plus a combined direction signal.

Single-variable change is too small for most models. This topic freezes one coordinate at a time to compute partial derivatives, then collects them into the gradient vector used by descent methods.

  • Engineering Mathematics
  • Medium level
  • 4 concepts

1The gradient is the coordinate-wise change summary

When a function depends on many inputs, there is no single slope. Each coordinate can change the output in its own way, so partial derivatives measure those one-at-a-time sensitivities.

The gradient then packages all of those local sensitivities into one vector. That makes it the natural object for optimization, because it tells you how the function reacts across all coordinates at once.

Figure. The gradient packages coordinate-wise partial changes into one vector. Its components still come from the separate coordinate directions.

Build a gradient

  1. Freeze othersDifferentiate with respect to w_1 while treating w_2 as constant.
  2. RepeatDifferentiate with respect to w_2 while treating w_1 as constant.
  3. StackWrite the results in coordinate order as one vector.

Gradient of a two-parameter bowl

Compute the gradient of f(w_1,w_2)=(w_1-1)^2+(w_2+2)^2 at (w_1,w_2)=(3,-1).

  • \partial f/\partial w_12(w_1-1)
  • \partial f/\partial w_22(w_2+2)
  • \nabla f(3,-1)(2(3-1), 2(-1+2))
  • (2(3-1), 2(-1+2))(4, 2)

Pro tip. The two numbers mean loss is locally more sensitive to w_1 than to w_2 at this point.

For f(a,b)=a^2+3b^2, what is the gradient at (2,-1)?
  1. (4,-6)
  2. (2,-3)
  3. (4,6)
  4. (7,7)

The partials are 2a and 6b; substituting (2,-1) gives (4,-6).

2A partial derivative freezes the other variables

A partial derivative answers a deliberately narrow question: what happens if this coordinate changes and the other coordinates do not? That freeze is what separates a partial derivative from an arbitrary path through the surface.

The rule is practical in models with many parameters. You can inspect one weight's sensitivity without letting every other weight drift during the calculation.

The freeze rule would show two one-axis probes through the same surface point: one in the a direction and one in the b direction.

Freeze and differentiate

  1. Choose coordinatePick the variable whose sensitivity you want.
  2. Hold constantsTreat every other variable as a number during that derivative.
  3. Evaluate pointSubstitute the full point only after the derivative expression is ready.

One-coordinate sensitivity

For g(a,b)=a^2b+3b, find \partial g/\partial a and \partial g/\partial b at (a,b)=(2,5).

  • \partial g/\partial a2ab
  • \partial g/\partial a at (2,5)2(2)(5)=20
  • \partial g/\partial ba^2+3
  • \partial g/\partial b at (2,5)2^2+3=7

Pro tip. The a-sensitivity and b-sensitivity are different because each derivative freezes a different coordinate.

For g(a,b)=a^2b+5b, what is partial with respect to a?
  1. 2ab
  2. a^2+5
  3. 2ab+5
  4. a^2b

While differentiating with respect to a, treat b as constant, so a^2b becomes 2ab.

3Negative gradient is the local descent direction

The gradient points toward the fastest local increase of a scalar function. If the function is a loss, the direction that locally lowers it fastest is the negative gradient.

A gradient descent step uses this fact by subtracting a small multiple of the gradient from the current parameter vector. The step size decides how far to trust the local linear picture.

Figure. For a loss, the gradient points uphill locally, so a descent step goes in the opposite direction. The two arrows share one point to show the sign flip.

Take a descent step

  1. Compute gradientFind \nabla f at the current parameter vector.
  2. ScaleMultiply it by the learning rate \eta.
  3. SubtractUpdate w_{new}=w-\eta\nabla f(w) for descent.

One descent step on the bowl

For f(w_1,w_2)=(w_1-1)^2+(w_2+2)^2, start at (3,-1) with gradient (4,2) and use \eta=0.1.

  • \eta\nabla f0.1(4,2)=(0.4,0.2)
  • (w_1,w_2) - \eta\nabla f(3,-1)-(0.4,0.2)
  • New point(2.6,-1.2)
  • f(3,-1) vs f(2.6,-1.2)5 vs (1.6)^2+(0.8)^2=3.2

Pro tip. The step lowers the loss here because it moves opposite the local increase direction.

Coding lab. Take one NumPy gradient step runs in the app, with checks on your output.

At w=(1,2), gradient is (3,-4) and eta is 0.1. What is one descent step?
  1. (0.7,2.4)
  2. (1.3,1.6)
  3. (0.3,-0.4)
  4. (0.9,1.8)

Subtract eta times gradient: (1,2)-0.1(3,-4)=(1,2)-(0.3,-0.4)=(0.7,2.4).

4Dotting the gradient with a step predicts change

The gradient does not only point somewhere. Its dot product with a proposed small step estimates the function change caused by that step.

This lets you compare candidate updates before taking them. A negative dot product with the gradient predicts a local decrease; a positive dot product predicts an increase.

Figure. The same gradient (4, 2) is dotted with two candidate steps. ΔwA = (−0.2, −0.1) scores −1.0. ΔwB = (0.1, −0.4) scores −0.4. The more negative local prediction is A.

Predict a step

  1. Propose stepWrite the candidate change vector \Delta w.
  2. Dot with gradientCompute \nabla f(w)\cdot \Delta w.
  3. Read signNegative predicts decrease, positive predicts increase.

Compare two candidate updates

At a point where \nabla f=(4,2), compare updates \Delta w_A=(-0.2,-0.1) and \Delta w_B=(0.1,-0.4).

  • \nabla f\cdot\Delta w_A4(-0.2)+2(-0.1)=-1.0
  • \nabla f\cdot\Delta w_B4(0.1)+2(-0.4)=-0.4
  • Predicted changeA lowers the loss more locally because -1.0<-0.4

Pro tip. A step can have one coordinate moving the 'wrong' way and still lower loss if the combined dot product is negative.

If gradient is (2,-1) and proposed step is (-0.5,0), what change is predicted?
  1. -1
  2. 1
  3. 0.5
  4. -0.5

Dot product gives the first-order change: (2,-1) dot (-0.5,0)=-1.

Notes

  • A partial derivative changes one variable at a time while the others are held fixed.
  • The gradient collects all first partial derivatives into one vector.
  • For a scalar loss, the negative gradient is the steepest local descent direction under the usual Euclidean distance.

Formulas

  • grad f(x) = [df/dx_1, ..., df/dx_n]^T.
  • For f(w_1,w_2), grad f = (df/dw_1, df/dw_2).
  • A small step Delta w changes f by about grad f . Delta w.

Exam traps & shortcuts

  • Read each partial derivative as a one-coordinate sensitivity test inside a multivariable system.
  • Do not change two coordinates while computing one partial derivative; freeze the rest first.

Reference tables

These objects are related, but they answer different questions.

Gradient objects
ObjectMeaning
Partial derivativeSensitivity along one coordinate while the rest are fixed
GradientVector of all first partial derivatives
Negative gradientSteepest local descent direction for a loss
Gradient dot stepLocal estimate of the change from a proposed update

Recap

Move from one slope to one slope per coordinate.

Partial
A partial derivative changes one variable while freezing the rest.
Gradient
The gradient bundles all first partial derivatives into one direction-bearing vector.
Descent
Subtracting a small multiple of the gradient moves against local increase.
Step test
\nabla f\cdot\Delta w predicts whether a small proposed update raises or lowers the function.

Practise Partial Derivatives and Gradients

Reading is free and needs no account. Practice, mocks and progress live in the app.

  • A 5-question practice set that ends the chapter
  • 4 quick checks with worked explanations
  • Timed mocks scored with the real marking scheme
  • Readiness tracked per topic, kept on your device
Continue with Google — freeNo card, no trial. Works offline once installed.