Deep Learning · Deep Learning
Embeddings and Sequences
In Deep Learning because embeddings turn discrete tokens into vectors learned inside neural models.
Tokens become vectors; keep vocabularies tiny in-browser.
- Deep Learning
- Medium level
- 3 concepts
1Embedding tables
An embedding maps a discrete id to a short dense vector.
Do not download large language-model weights in these labs — author tiny tables instead.
Figure. An embedding maps a discrete id to a short dense vector. Index tokens as 0…V−1, look up a row of length d, then train the table and the layers above from the task loss. Author tiny tables; do not download large language-model weights.
- Index each tokenMap words or ids to integers 0…V−1 — the embedding table has one row per id.
- Look up vectorsEach id pulls a dense vector of length d; similar ids can sit near each other after training.
- Train end-to-endGradients update both the embedding rows and the layers above them from the task loss.
Why prefer an embedding table to one-hot encoding over a 20,000-word vocabulary?
- One-hot vectors cannot be fed into a neural network at all
- One-hot encoding needs a pretrained model in order to be computed
- Embeddings are exact where one-hot vectors are approximations
- One-hot gives every word its own unrelated slot, while a short learned vector can put related words near each other
Two one-hot vectors are equally far apart whichever words they stand for, so the representation asserts that no word resembles any other. A learned table can carry similarity, and it is far smaller.
2Sequence intuition
Order matters for text and time series; bag-of-words ignores order.
RNNs and attention models exist; start with TF-IDF or tiny embeddings first.
Figure. Order matters for text and time series. A bag-of-words view ignores that order. Start with TF-IDF or tiny embeddings before RNNs or attention.
'Not good, actually bad' and 'not bad, actually good' produce the same bag-of-words vector. What does that tell you?
- Bag-of-words is broken and should never be used
- The two sentences really do mean the same thing, so nothing was lost
- The vocabulary is too small and needs to be expanded
- The representation has discarded order, so no model built on top of it can tell the two sentences apart
A model can never recover information its input representation dropped. That is a ceiling on the representation, not a fault in the classifier sitting above it.
3Transfer learning concepts
Transfer learning reuses features trained elsewhere.
Here we only discuss the idea — no Hugging Face downloads at runtime.
Figure. Transfer learning reuses features trained elsewhere. This lesson names the idea only; there are no Hugging Face downloads at runtime.
What does transfer learning actually carry over from the earlier task?
- The earlier labels, relabelled to suit the new task
- The earlier training loop and its hyperparameters
- The earlier dataset, which has to ship alongside the model
- The representation it learned — the features training discovered — rather than its predictions
The useful residue of training is the intermediate features. They are general enough to be reused, which is why the new task needs far fewer labels than the original did.
Notes
- Tiny vocab embeddings without large pretrained downloads.
- An embedding maps a discrete id to a short dense vector.
- Order matters for text and time series; bag-of-words ignores order.
Exam traps & shortcuts
- Keep lab datasets under 2000 rows in the browser runtime.
- Split train and test before fitting any model that sees labels.
Recap
This lesson in brief:
- Embedding tables
- An embedding maps a discrete id to a short dense vector.
- Sequence intuition
- Order matters for text and time series; bag-of-words ignores order.
- Transfer learning concepts
- Transfer learning reuses features trained elsewhere.
Practise Embeddings and Sequences
Reading is free and needs no account. Practice, mocks and progress live in the app.
- A 1-question practice set that ends the chapter
- 3 quick checks with worked explanations
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device