E ExamMaster

Deep Learning · Deep Learning

Neural Network Fundamentals

In Deep Learning because layered neural nets (perceptron → MLP) are the core DL representation.

Deep learning starts with layered linear maps and non-linearities.

  • Deep Learning
  • Medium level
  • 5 concepts

1Perceptron

A perceptron computes a weighted sum and applies a threshold or activation.

One layer of linear scores cannot solve XOR — depth or non-linearity is required.

Figure. The whole unit is one line of arithmetic: multiply each input by its weight, add the bias, pass the total through an activation. Everything the unit can learn lives in w1, w2, w3 and b - and a single such linear score is exactly what cannot solve XOR.

Why can one perceptron never learn XOR, however long you train it?
  1. XOR offers too few distinct training examples to fit
  2. XOR requires a sigmoid where the perceptron uses a threshold
  3. The learning rate cannot be tuned low enough to converge
  4. It can only cut the input space with a single straight boundary, and no straight line puts both XOR classes on their own side

This is a limit of the shape of the model, not of the training. XOR is the smallest problem whose two classes interleave, which is why it became the standard demonstration.

2Multilayer perceptron

An MLP stacks linear layers with activations such as ReLU between them.

Width and depth add capacity; they also add overfit risk on tiny data.

Figure. Every input reaches every hidden unit, and every hidden unit reaches every output: this 3-4-2 net already carries 3x4 + 4x2 = 20 weights plus biases. Width and depth buy flexibility, and on tiny data the same wires are how noise gets memorised.

You stack three Linear layers with no activation between them. What have you built?
  1. A three-layer network with roughly three times the capacity
  2. A network that cannot train because its gradients vanish
  3. A single linear map wearing three layers' worth of parameters
  4. A convolutional network without the weight sharing

Composing linear maps yields a linear map. The extra parameters buy no extra expressiveness, which is precisely the hole the non-linearity fills.

3Parameters versus hyperparameters

Weights are learned; learning rate, width, and epochs are chosen by you.

Keep parameter counts tiny in the browser — thousands, not millions.

Figure. Weights are learned. Learning rate, width, and epochs are chosen by you. Keep parameter counts tiny in the browser — thousands, not millions.

Which of these does gradient descent not adjust on your behalf?
  1. The weights in the first Linear layer
  2. The learning rate used to apply each update
  3. The bias terms in the output layer
  4. The weights in the hidden layer

Gradient descent moves whatever the loss has a gradient with respect to. The step size governs that movement from outside it, which is why it is yours to choose and to tune.

4Why depth helps

Depth composes simple functions into more flexible decision surfaces.

On tiny data, that flexibility is also how you memorise noise.

Figure. Depth composes simple functions into more flexible decision surfaces. On tiny data, that same flexibility is how you memorise noise.

With 120 training rows your two-layer net underperforms, so you add six more layers. What is the likeliest outcome?
  1. A better test score, since depth composes richer functions
  2. The training score improves and the test score falls, because 120 rows cannot pin down that much flexibility
  3. No change at all, because depth only matters for image data
  4. A crash during training from having too many parameters

Depth buys flexibility, and flexibility has to be paid for in data. Below that price the extra capacity is spent memorising the particular 120 rows you happen to hold.

5Lab: tiny torch MLP

Build a two-layer network on a toy regression and print a score-like loss.

The runtime provides a teaching torch shim safe for small tensors.

Figure. The lab is a two-layer torch MLP: Linear 4 to 8, ReLU, Linear 8 to 1, on 32 rows of 4 features. A forward pass prints an MSE score. The runtime is a teaching torch shim.

Coding lab. Tiny torch MLP runs in the app, with checks on your output.

This lab prints an MSE loss and labels it 'score'. What does a lower number mean here, and why is that the reverse of the sklearn labs?
  1. Lower is better, because MSE counts error while an accuracy score counts success
  2. Lower is worse, because both MSE and accuracy rise with quality
  3. Lower is meaningless until the loss has been normalised
  4. Lower means the model has not begun training yet

The word score is doing two jobs across these labs. Always ask whether the printed quantity is something you want more of or less of before reading a run as good.

Notes

  • Perceptron to MLP on small tensors.
  • A perceptron computes a weighted sum and applies a threshold or activation.
  • An MLP stacks linear layers with activations such as ReLU between them.

Exam traps & shortcuts

  • Keep lab datasets under 2000 rows in the browser runtime.
  • Split train and test before fitting any model that sees labels.

Recap

This lesson in brief:

Perceptron
A perceptron computes a weighted sum and applies a threshold or activation.
Multilayer perceptron
An MLP stacks linear layers with activations such as ReLU between them.
Parameters versus hyperparameters
Weights are learned; learning rate, width, and epochs are chosen by you.
Why depth helps
Depth composes simple functions into more flexible decision surfaces.

Practise Neural Network Fundamentals

Reading is free and needs no account. Practice, mocks and progress live in the app.

  • A 1-question practice set that ends the chapter
  • 5 quick checks with worked explanations
  • Timed mocks scored with the real marking scheme
  • Readiness tracked per topic, kept on your device
Continue with Google — freeNo card, no trial. Works offline once installed.