Deep Learning · Deep Learning
CNNs on Tiny Images
In Deep Learning because convolutions exploit spatial structure — a neural, not tabular, inductive bias.
Convolutions share weights across local patches.
- Deep Learning
- Medium level
- 3 concepts
1Why convolution
Nearby pixels interact; a shared filter detects the same pattern everywhere.
Parameter count stays lower than a giant dense layer on flattened pixels.
A dense layer on flattened pixels must learn 'edge in the top-left' and 'edge in the bottom-right' as two separate things. What does a convolution change?
- It shrinks the image before the layer ever sees it
- One set of filter weights is applied at every position, so the pattern is learned once and can then be found anywhere
- It removes the need for an activation between layers
- It learns a separate filter for each pixel, which is more precise
Weight sharing is both the parameter saving and the inductive bias. It encodes the assumption that a pattern means the same thing wherever in the image it turns up.
2Tiny stacks
Conv → ReLU → pool patterns extract edges then parts.
For labs, use 8×8 or 16×16 synthetic grids — not ImageNet.
Figure. A tiny stack is conv, then ReLU, then pool: edges first, then parts. Teaching labs stay on 8×8 or 16×16 synthetic grids, not ImageNet.
Why do the later layers of a convolutional stack respond to parts rather than to edges?
- Later layers use larger filters by definition
- Each layer sees a wider patch of the original image through the layers beneath it, so it can combine their edges into something larger
- Pooling substitutes parts for edges directly
- The edges are discarded after the first ReLU
The filters stay small; what grows is how much of the input each position can see. That widening view is what lets a stack build edges into corners and corners into parts.
3Lab: tensorflow dense on grids
Flatten tiny grids and fit a small Sequential model — a stepping stone before true conv layers.
Print a score loss after a few epochs on synthetic data.
Figure. The stepping-stone lab flattens 200 rows of 8 features through Dense 16 ReLU, then Dense 1 sigmoid, fits 5 epochs, and prints a score. It is a Sequential dress rehearsal before true conv layers.
Coding lab. Tiny tf Sequential runs in the app, with checks on your output.
This lab flattens the grid and uses Dense layers. What does flattening throw away?
- The pixel values, which are replaced by their positional indices
- The labels, which have to be reattached afterwards
- Which pixels were neighbours — the very structure a convolution exists to exploit
- Nothing at all, since a reshape preserves every number
Every number does survive the reshape, which is what makes this trap comfortable. What is lost is adjacency: to a dense layer, two neighbouring pixels are no more related than two opposite corners.
Notes
- Convolutional filters for small grids — browser-safe sizes only.
- Nearby pixels interact; a shared filter detects the same pattern everywhere.
- Conv → ReLU → pool patterns extract edges then parts.
Exam traps & shortcuts
- Keep lab datasets under 2000 rows in the browser runtime.
- Split train and test before fitting any model that sees labels.
Recap
This lesson in brief:
- Why convolution
- Nearby pixels interact; a shared filter detects the same pattern everywhere.
- Tiny stacks
- Conv → ReLU → pool patterns extract edges then parts.
- Lab: tensorflow dense on grids
- Flatten tiny grids and fit a small Sequential model — a stepping stone before true conv layers.
Practise CNNs on Tiny Images
Reading is free and needs no account. Practice, mocks and progress live in the app.
- A 1-question practice set that ends the chapter
- 3 quick checks with worked explanations
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device