E ExamMaster

Deep Learning · Deep Learning

Activations and Optimizers

In Deep Learning because activations and optimizers (ReLU, Adam) shape neural training paths.

Activations shape representation; optimizers shape the path.

  • Deep Learning
  • Medium level
  • 4 concepts

1Common activations

ReLU is the default hidden activation; sigmoid and softmax appear at outputs.

Sigmoid saturates — poor for deep hidden stacks, fine for binary outputs.

Figure. ReLU is zero for negative input, then climbs forever at slope 1 - its gradient never fades on the positive side, which is why it is the default hidden activation. The sigmoid flattens toward 0 and 1 at both ends: in those saturated tails the gradient is nearly zero, starving deep stacks - keep it for outputs that must read as probabilities.

Sigmoid is a fine output activation for a binary classifier and a poor hidden activation deep in a stack. What is the difference?
  1. The output layer is smaller, so saturating there costs less
  2. Sigmoid cannot produce negative values, which hidden layers require
  3. A saturated sigmoid has a near-zero slope, which is harmless at the output but starves every layer below it of gradient
  4. Sigmoid evaluates more slowly than ReLU, and hidden layers run more often

At the output the squashing is the point, since you want a probability. In the middle of a stack that same flatness multiplies into the backward pass and stalls training beneath it.

2Dying ReLUs

A large negative bias can keep a ReLU unit off for every sample.

If many units die, lower the learning rate or check initialisation.

Figure. A large negative bias can keep a ReLU unit off for every sample. If many units die, lower the learning rate or check initialisation.

A hidden unit emits zero for every row of every batch, epoch after epoch. Why does it never come back?
  1. ReLU has zero gradient wherever its input is negative, so no update ever arrives to push the unit back into range
  2. ReLU units are reinitialised only at the very start of training
  3. The following layer discards zero outputs before they reach it
  4. Regularisation has driven the unit's weights to exactly zero

The unit is not merely quiet, it is unreachable: with no gradient there is no correction, so the state is self-sustaining. Prevention through the learning rate and initialisation is the only cure.

3SGD

Stochastic gradient descent updates from minibatches with a learning rate.

Too large a rate diverges; too small stalls.

Animation: a loss curve shaped like a valley with its minimum labelled. A ball starts high on the steep left slope and takes gradient steps downhill, each drawn as a gold arrow: big steps where the slope is steep, visibly shrinking steps as the slope flattens, settling at the minimum. The closing frame reads step equals learning rate times slope, too big diverges, too small stalls
Gradient descent walks downhill on the loss surface. The step is learning rate times slope, so steps shrink as the valley flattens - a rate too large overshoots and diverges, one too small crawls.
Your loss leaps to a huge value within a few steps and then to infinity. Which knob is the first suspect?
  1. The batch size, which has been set too small
  2. The learning rate, which is overshooting the minimum and landing further out on each step
  3. The epoch count, which has been set too high
  4. The activation function, which must be saturating

Divergence is the signature of too large a step, not of too much training. A gradient tells you a direction, and only a modest step keeps the promise that direction makes.

4Adam

Adam tracks moment estimates of gradients for adaptive steps.

It is a strong default for small teaching nets; still tune the learning rate.

Figure. Adam tracks moment estimates of gradients and turns them into adaptive steps. It is a strong default for small teaching nets; the learning rate is still chosen by you.

Adam adapts its step size for each parameter. Why is there still a learning rate for you to set?
  1. It is a legacy argument that has no effect when Adam is used
  2. It applies only to the bias terms, which Adam does not adapt
  3. The adaptation rescales a base step, so that base still sets the overall size of every update
  4. It controls how many gradient moments Adam keeps track of

Per-parameter adaptation handles the relative scaling between weights. The absolute scale is still yours, which is why a badly chosen rate breaks Adam much as it breaks plain SGD.

Notes

  • ReLU, sigmoid, SGD and Adam intuition.
  • ReLU is the default hidden activation; sigmoid and softmax appear at outputs.
  • A large negative bias can keep a ReLU unit off for every sample.

Exam traps & shortcuts

  • Keep lab datasets under 2000 rows in the browser runtime.
  • Split train and test before fitting any model that sees labels.

Recap

This lesson in brief:

Common activations
ReLU is the default hidden activation; sigmoid and softmax appear at outputs.
Dying ReLUs
A large negative bias can keep a ReLU unit off for every sample.
SGD
Stochastic gradient descent updates from minibatches with a learning rate.
Adam
Adam tracks moment estimates of gradients for adaptive steps.

Practise Activations and Optimizers

Reading is free and needs no account. Practice, mocks and progress live in the app.

  • A 1-question practice set that ends the chapter
  • 4 quick checks with worked explanations
  • Timed mocks scored with the real marking scheme
  • Readiness tracked per topic, kept on your device
Continue with Google — freeNo card, no trial. Works offline once installed.