Deep Learning · Deep Learning
How Nets Read Text
In Deep Learning because a neural net reads text as tokens, embeddings and a sequence decision — not as pixels.
A customer left a product review: not bad, actually good. A computer cannot multiply those letters. This lesson walks that one sentence through a reading pipeline — cut it into words, look up a short list of numbers for each word, then turn the stack into a yes-or-no decision. Embeddings and Sequences already taught the lookup table; here you see the rest of the path, and only afterwards how it differs from a tiny vision CNN.
- Deep Learning
- Medium level
- 5 concepts
1Text in, decision out
Start with the review not bad, actually good. That is a string a person typed. A layer in a net only multiplies numbers, so the letters themselves cannot go in. The path is: cut the string into words (tokens), look up a short vector for each word, then produce a decision — here, whether the review is positive.
Hold that path on this one sentence before any other course. Tokenising will turn the comma-stripped words into ids. The embedding table will replace each id with a row of numbers. A later card will average those rows and show why order can vanish.
A tiny vision CNN already receives a grid of numbers: each pixel is a brightness. Text is discrete symbols, so it needs this extra reading pipeline. The Machine Learning Review Sentiment lesson does a related job with TF-IDF word counts and a linear classifier. This lesson is the neural version: same kind of text decision, learned vectors rather than counted words. A neural net cannot multiply letters — that is why the pipeline exists.
Figure. The review not bad, actually good enters as letters and leaves as one decision. The four chips are the tokens the net actually reads; each becomes an id, then a vector, before the head answers. A tiny vision CNN starts already on the number side of this picture.
How a net reads text
- TokeniseCut the string into tokens and map each one to an integer id from a fixed vocabulary.
- EmbedLook up one row of the embedding table per id — the table taught in Embeddings and Sequences.
- DecideTurn the sequence of vectors into one text decision: sentiment, spam, or the next token.
A 16×16 greyscale photo is already a grid of numbers. A product review is a string of letters. What extra job does the review force on a neural net?
- It must flatten the review into a 16×16 grid so a convolution can run
- It must first turn the string into token ids and then into vectors, because a layer cannot multiply letters
- It must download a pretrained language model before any layer can run
- Nothing extra: letters are already numbers in Unicode, so the net can multiply them as-is
A layer multiplies numbers. Pixels already are numbers; letters are not, until tokenising and the embedding table make them so. Unicode code points are not a learned representation, and a convolution expects a grid.
2Tokenise the string
Take the review not bad, actually good. Tokenising cuts it into words — not, bad, actually, good — after the comma is stripped so the comma does not become its own token. A fixed vocabulary then maps each word to an integer id. In this lesson the table is tiny: actually = 1, bad = 2, good = 3, not = 4.
Those ids are what the embedding table receives, in order: 4, then 2, then 1, then 3. An unknown word has no row; in these labs every token is known. Subword tokenisers exist for large models; we stay on whole words so the ids stay readable.
Figure. Tokenising cuts the review into not, bad, actually, good — the comma is stripped first — and the vocabulary maps each word to an id. The embedding table receives 4, 2, 1, 3 in that order.
| Token | Id |
|---|---|
| not | 4 |
| bad | 2 |
| actually | 1 |
| good | 3 |
Ids for not bad, actually good
Vocabulary: actually = 1, bad = 2, good = 3, not = 4. Tokenise "not bad, actually good".
- tokens after stripping the commanot, bad, actually, good
- id of not, then bad, actually, good4, 2, 1, 3
Pro tip. The sequence is 4, 2, 1, 3, not a bag. Order is still sitting in the ids; later pooling is what can throw it away.
The review is 'not bad, actually good' and the vocabulary is actually=1, bad=2, good=3, not=4. What should the embedding table receive?
- The id sequence 4, 2, 1, 3 — one integer per token, in order
- The four words themselves, because an embedding table looks up strings
- A single integer that hashes the whole review
- The letters n, o, t, b, a, d, each as its own id
An embedding table is indexed by integer id. The string has to become those ids first, and the order of the tokens is the order of the ids.
3Look up, do not rebuild
Each id selects one row of the embedding table taught in Embeddings and Sequences. Token id 4 (the word not) becomes the short vector in that row — here (0.10, 0.10). Four tokens become a stack of four rows. If the table is two-dimensional, that stack is shape (4, 2). This lesson does not rebuild that table.
The vectors are learned from a task later; here they are a tiny authored table so you can see the lookup. Do not download a language-model checkpoint in the browser runtime. One table serves every review: four lookups, not four new tables.
No diagram — the idea is carried by the prose, table or coding lab.
| Id | Token | Vector |
|---|---|---|
| 1 | actually | (0.00, 0.00) |
| 2 | bad | (-0.40, 0.20) |
| 3 | good | (0.40, -0.20) |
| 4 | not | (0.10, 0.10) |
One lookup, then a stack
Token id 4 sits in a 5 \times 2 table (row 0 unused). The review ids are 4, 2, 1, 3. What does the model read?
- table row 4 (not)(0.10, 0.10)
- stack of four looked-up rowsshape (4, 2)
Pro tip. The table has one row per id, not per review. Four tokens are four row lookups, then a (4, d) tensor.
This lesson needs a vector for token id 4. Where does that vector come from?
- A one-hot vector of length equal to the review
- The Unicode code point of the first letter of the token
- A random vector drawn again every time the token appears, so the net cannot overfit
- Row 4 of the embedding table taught in Embeddings and Sequences — this lesson does not rebuild that table
The reading pipeline reuses the table. Rebuilding it here would retell Embeddings and Sequences; drawing a fresh random vector every time would erase the very similarity that training is meant to store.
4Sequence to a text decision
A classifier head needs one vector per review, not one per token. The smallest move is to mean-pool: add the token vectors and divide by how many tokens you had. For not bad, actually good that is four rows averaged. That is a bag — it keeps what appeared, not in which order.
The two reviews not bad, actually good and not good, actually bad share the same tokens, so they share the same mean-pooled vector. A sentiment head sitting on that pool cannot tell them apart. Convolution on a tiny image keeps neighbouring pixels; mean-pooling text does not keep word order. Recurrent and attention models exist for order; we do not build them here. Mean-pool is the browser-safe baseline, the same role TF-IDF played in Review Sentiment.
Figure. Both rows hold not, bad, actually and good. Mean-pooling adds the four vectors and divides by four, so review A and review B collapse to the same vector and a sentiment head cannot tell them apart.
| Input | What is kept |
|---|---|
| Tiny image grid | Neighbour pixels (convolution) |
| Pooled token vectors | Which tokens appeared, not order |
| Ordered token vectors | Order (RNN / attention) |
The two reviews collide
Using the tiny table, mean-pool "not bad, actually good" and "not good, actually bad".
- sum of rows 4, 2, 1, 3(0.10, 0.10)
- sum of rows 4, 3, 1, 2(0.10, 0.10)
- either sum divided by 4(0.025, 0.025)
Pro tip. Same bag, same pool. The collision is a property of mean-pool, not a bug in the sentiment head.
'not bad, actually good' and 'not good, actually bad' produce the same mean-pooled vector. What does that imply for a sentiment head sitting on the pool?
- The head will still separate them, because Linear layers restore word order
- The head cannot tell the reviews apart, because the pool already threw the order away
- The two reviews really do have the same sentiment, so the collision is correct
- You must widen the embedding dimension until the means separate
A head cannot recover information its input dropped. Widening the table does not restore order, and a Linear layer on a single pooled vector has no sequence left to read.
5Lab: tiny text pipeline
The lab tokenises four short reviews, looks up the same tiny table, mean-pools, and sends the pooled vector through a torch Linear head. It prints a score loss. The starter reviews are good good good and bad bad bad — bags that pooling can separate — not the order-trap pair not bad, actually good versus not good, actually bad, which would collide.
The runtime is the teaching torch shim: Linear, ReLU, MSELoss. There is no Embedding layer; the lookup is a numpy row select, which is exactly what an embedding table does. Run the cells in order so the ids from the first review are still in memory for the lookup.
Figure. The lab tokenises four short reviews, looks up a tiny table, mean-pools, and sends the pooled vector through a Linear 2 to 1 head. The starter bags good good good and bad bad bad are separable by pooling. It prints a score.
Coding lab. Tiny text pipeline runs in the app, with checks on your output.
The lab mean-pools embedding rows, then applies a Linear head. Why are the starter reviews 'good good good' and 'bad bad bad' rather than the 'not bad / not good' pair?
- The torch shim cannot tokenise the word 'actually'
- Positive and negative labels are only allowed on one-word reviews
- Those two pairs share the same tokens, so mean-pool would give them the same vector and the printed score would not be about sentiment
- Mean-pool is more accurate when every review is three copies of one word
The order-trap pair is the same bag. Mean-pool collides them, so a loss printed on that pair would not be measuring a sentiment decision. The starter reviews are bags pooling can actually separate.
Notes
- Text in, tokenise, embed, then a text decision — the neural reading pipeline.
- A neural net cannot multiply letters.
- Take the review not bad, actually good.
Exam traps & shortcuts
- Keep lab datasets under 2000 rows in the browser runtime.
- Split train and test before fitting any model that sees labels.
Recap
This lesson in brief:
- Text in, decision out
- Start with the review not bad, actually good. That is a string a person typed. A layer in a net only multiplies numbers, so the letters themselves cannot go in. The path is: cut the string into words (tokens), look up a short vector for each word, then produce a decision — here, whether the review is positive.
- Tokenise the string
- Take the review not bad, actually good. Tokenising cuts it into words — not, bad, actually, good — after the comma is stripped so the comma does not become its own token. A fixed vocabulary then maps each word to an integer id. In this lesson the table is tiny: actually = 1, bad = 2, good = 3, not = 4.
- Look up, do not rebuild
- Each id selects one row of the embedding table taught in Embeddings and Sequences. Token id 4 (the word not) becomes the short vector in that row — here (0.10, 0.10). Four tokens become a stack of four rows. If the table is two-dimensional, that stack is shape (4, 2). This lesson does not rebuild that table.
- Sequence to a text decision
- A classifier head needs one vector per review, not one per token. The smallest move is to mean-pool: add the token vectors and divide by how many tokens you had. For not bad, actually good that is four rows averaged. That is a bag — it keeps what appeared, not in which order.
Practise How Nets Read Text
Reading is free and needs no account. Practice, mocks and progress live in the app.
- A 1-question practice set that ends the chapter
- 5 quick checks with worked explanations
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device