Reinforcement Learning · Reinforcement Learning Core
A tiny update loop
In Reinforcement Learning because the table of values improves from the walk you just took, one update at a time.
The Q numbers in the last lesson were written by hand from a path we already knew. This lesson grows them from walks. Start every Q at 0. Take a step. Mix the old number with what just happened — the reward, plus a discounted peek at the next cell. After enough walks, C1 Right becomes 10, B1 Right becomes 8, A1 Right becomes 6.2. Then greedy on that table walks Start to Goal.
- Reinforcement Learning
- Medium level
- 5 concepts
1A table of cells times actions, starting at 0
Make a table with one row per cell and one column per action. Twelve cells times four actions is 48 slots. Fill every slot with zero. That is honesty: before any walk, you do not yet know that C1 Right is worth 10.
We will only watch the three entries that matter on the short path: (C1, Right), (B1, Right), (A1, Right). The rest can stay zero for now. The table is the memory. Walks write into it.
Figure. The honest table starts at zero. C1 Right is not yet 10, because no walk has written it.
| Cell, action | Q |
|---|---|
| C1, Right | 0 |
| B1, Right | 0 |
| A1, Right | 0 |
An empty Q-table
The floor has 12 cells and 4 actions. How many Q numbers start at 0, and which three will the short Goal walk write first?
- 12 cells × 4 actions48 zeros
- short-path slots that will move firstQ(C1, R), Q(B1, R), Q(A1, R)
Pro tip. Zero means 'not yet walked', not 'this door is worthless'. The first Goal step is what turns C1 Right into 10.
Coding lab. Zeros to start runs in the app, with checks on your output.
Why start Q at 0 rather than at 10?
- Before a walk you have not yet seen that C1 Right pays plus 10
- Zero is the reward at Goal
- A neural net requires zeros
The table should start ignorant. The walks write the numbers.
2One step writes one new Q
Take C1 Right into Goal. Reward 10. The walk is over, so the next cell has no future Q — treat that peek as 0. With a step-size of 1 (replace the old number outright) and gamma 0.9: new Q(C1, Right) equals 10 plus 0.9 times 0, which is 10.
That is the whole update. Old 0, what happened 10, new 10. One cell-action pair moved. Nothing else in the table changed yet. Alpha, the step-size, is 1 here so the new number fully replaces the old one.
Figure. C1 Right into Goal. Old Q 0, reward 10, next value 0. New Q 10.
First update at C1
Old Q(C1, R) is 0. You take Right, reward 10, episode ends. Alpha 1, gamma 0.9. New Q?
- old Q0
- reward + 0.9 * 010
- new Q(C1, R)10
Pro tip. Terminal next-value is 0. The plus 10 is the reward, now stored as Q.
Coding lab. Update C1 Right runs in the app, with checks on your output.
In the Temporal Difference update rule, what does the term [r + gamma * max Q(s', a') - Q(s, a)] represent?
- The total discounted cumulative return across all past episodes
- The gradient of the policy with respect to environment transition dynamics
- The TD error, measuring the difference between the new target estimate and the prior belief
- The probability that the agent takes an exploratory random action
The TD error (r + gamma * max Q(s', a') - Q(s, a)) is the surprise/discrepancy between the updated estimate and the old Q-value.
3Each cell borrows the next cell's news
Now stand on B1 and take Right to C1. Reward minus 1. C1's best Q is already 10. New Q(B1, Right) equals minus 1 plus 0.9 times 10, which is 8. Then A1 Right to B1: minus 1 plus 0.9 times 8, which is 6.2.
News travels backward along the path. Goal writes 10 into C1. C1 writes 8 into B1. B1 writes 6.2 into A1. That backup is the temporal-difference idea: mix what just happened with the next cell's current guess, instead of waiting to add the whole walk.
Figure. Goal writes 10 into C1. C1 writes 8 into B1. B1 writes 6.2 into A1. News travels backward.
Backup B1, then A1
After Q(C1, R) is 10, update Q(B1, R) then Q(A1, R). Alpha 1, gamma 0.9.
- Q(B1, R) = -1 + 0.9*108
- Q(A1, R) = -1 + 0.9*86.2
Pro tip. These are the same 8 and 6.2 you met as values. Now they were grown from a walk.
Coding lab. Backup two cells runs in the app, with checks on your output.
Q(C1, R) is already 10. You take B1 Right, reward minus 1. New Q(B1, R) is:
- 8 — minus 1 plus 0.9 times 10
- 10 — copy C1
- Minus 1 — only this step's reward
The update mixes this reward with a discounted peek at the next Q.
4Many walks fill the useful cells
One lucky Goal walk writes three numbers. Walks that fall in the Pit write a different story: C2 Right becomes minus 10, and C1 Down becomes minus 1 plus 0.9 times that, which is minus 10. After several walks the table is no longer zeros — Goal-adjacent Rights go up, Pit-adjacent Rights go down.
The GIF fills those numbers on the floor. You are not training AlphaGo. You are watching three or four cells learn which door was expensive.
A Pit update
C2 Right enters Pit, reward -10, terminal. New Q(C2, R)? Then C1 Down to C2: -1 + 0.9 * that.
- Q(C2, R) = -10 + 0.9*0-10
- Q(C1, D) = -1 + 0.9*(-10)-10
Pro tip. The same update rule writes good news at Goal and bad news at the Pit.
Coding lab. A Pit backup runs in the app, with checks on your output.
How do Q-values propagate from a terminal Goal tile back toward the Start tile across multiple training episodes?
- Each episode updates predecessor tiles using the newly established values of their successor tiles
- The environment instantly broadcasts the goal reward to all grid tiles at initialization
- Start tiles are updated first, and values cascade forward toward the Goal
- The agent requires an explicit graph search tree before any Q-updates can occur
TD learning bootstraps: once a goal is found, adjacent cells learn high Q-values, which then get backed up to earlier cells on subsequent visits.
5Greedy on the learned table walks to Goal
After the backups, Q(A1, Right) is 6.2, Q(B1, Right) is 8, Q(C1, Right) is 10. Greedy at each cell picks Right. Following that table from Start is A1-B1-C1-D1 again — now because the numbers say so, not because we wrote the policy by hand.
That is the loop this course exists to show: act, see a reward, update a number, act a little better. No PPO. No AlphaGo. A table on a named floor.
Figure. A1 6.2, B1 8, C1 10. Greedy Rights walk Start to Goal. The numbers came from updates, not from a typed policy table.
Greedy on learned Q
Q Rights are 6.2, 8, 10 on A1, B1, C1. Other actions 0. Which walk does greedy produce?
- A1 picks max, RightB1
- B1 picks RightC1
- C1 picks RightD1
Pro tip. The policy was not typed this time. Argmax of the learned table produced it.
Coding lab. Greedy on learned Q runs in the app, with checks on your output.
After the backups, greedy on the Q-table walks A1-B1-C1-D1. What produced that policy?
- Argmax of numbers grown from walks — not a policy we typed by hand
- A downloaded AlphaGo checkpoint
- A labelled rent table
The loop is act, update, act better. The table is the policy.
Notes
- In Reinforcement Learning because the table of values improves from the walk you just took, one update at a time.
- Make a table with one row per cell and one column per action. Fill it with zeros. That is honesty: before any walk, you do not yet know that C1 Right is worth 10.
- Take C1 Right into Goal. Reward 10. The walk is over, so the next cell has no future Q — treat that peek as 0. With a step-size of 1 (replace the old number outright) and gamma 0.9: new Q(C1, Right) equals 10 plus 0.9 times 0, which is 10.
Exam traps & shortcuts
- Act, then read the number the floor paid. That number is this step, not the whole walk.
- A policy is a rule per cell. A value is how good that cell is. Neither is a neural net.
Recap
A Q-table starts at 0. One update is: new = reward + gamma times the best next Q. After the short path backs up, Q(C1,R)=10, Q(B1,R)=8, Q(A1,R)=6.2. Greedy on that table is the policy you wanted. Not PPO. Not AlphaGo.
- A table of cells times actions, starting at 0
- Make a table with one row per cell and one column per action. Twelve cells times four actions is 48 slots. Fill every slot with zero. That is honesty: before any walk, you do not yet know that C1 Right is worth 10.
- One step writes one new Q
- Take C1 Right into Goal. Reward 10. The walk is over, so the next cell has no future Q — treat that peek as 0. With a step-size of 1 (replace the old number outright) and gamma 0.9: new Q(C1, Right) equals 10 plus 0.9 times 0, which is 10.
- Each cell borrows the next cell's news
- Now stand on B1 and take Right to C1. Reward minus 1. C1's best Q is already 10. New Q(B1, Right) equals minus 1 plus 0.9 times 10, which is 8. Then A1 Right to B1: minus 1 plus 0.9 times 8, which is 6.2.
- Many walks fill the useful cells
- One lucky Goal walk writes three numbers. Walks that fall in the Pit write a different story: C2 Right becomes minus 10, and C1 Down becomes minus 1 plus 0.9 times that, which is minus 10. After several walks the table is no longer zeros — Goal-adjacent Rights go up, Pit-adjacent Rights go down.
Practise A tiny update loop
Reading is free and needs no account. Practice, mocks and progress live in the app.
- A 3-question practice set that ends the chapter
- 5 quick checks with worked explanations
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device