Reinforcement Learning · Reinforcement Learning Core
Value is how good a cell is
In Reinforcement Learning because a value is how good a cell (or a step from a cell) is — a number, not a neural net.
Some tiles are better to stand on than others. C1 is one step from Goal. D2 is the Pit. A number can say that. The value of a cell is how much return you expect from there if you act well. The value of a (cell, action) pair — Q — is how much return you expect if you take that step and then act well. Both are numbers in a table. Neither is a neural net.
- Reinforcement Learning
- Medium level
- 5 concepts
1How good is this cell?
Stand on C1 and act well — Right, into Goal. The return from C1 is plus 10. So the value of C1 is 10. Stand on B1, step Right to C1, then act well: reward minus 1, then 0.9 times 10, which is 8. The value of B1 is 8.
Stand on A1, step Right to B1: minus 1 plus 0.9 times 8, which is 6.2. Those three numbers describe the same greedy path you already walked. They are how good it is to be on each tile, not a new walk.
Figure. A1 shows 6.2, B1 shows 8, C1 shows 10 — how good each cell is under the greedy Right policy with gamma 0.9.
Values on the short path
Gamma 0.9. Value of C1, then B1, then A1, acting Right each time.
- V(C1) = 1010
- V(B1) = -1 + 0.9*108
- V(A1) = -1 + 0.9*86.2
Pro tip. These are values of cells, not rewards. The reward at C1 Right is still +10.
Coding lab. Print V on the top row runs in the app, with checks on your output.
V(B1) is 8. What does that number mean?
- From B1, acting well, the discounted return is 8
- B1's reward is plus 8
- B1 is eight tiles from Start
Value is expected return from that cell, not the one-step reward.
2A number in a table, not a neural net
Here the knowledge is twelve numbers — or fewer, one per cell we care about — written in a table you can read. C1 is 10 because Right is Goal, not because a layer computed a hidden vector. If you can point at the cell and read the number, you have the idea.
Deep Learning stores knowledge in the weights of a net. A net can sit on this loop later. This course does not start there. The value table is the thing that is standing.
No diagram — the idea is carried by the prose, table, code block or coding lab.
| Course | The number lives in |
|---|---|
| This lesson | A table keyed by cell |
| Deep Learning | Weights of a net |
Coding lab. A table, not a net runs in the app, with checks on your output.
In tabular Q-learning on a small grid, where do learned state-action estimates reside?
- In a multi-layer convolutional neural network weight matrix
- In an external relational database requiring SQL join queries
- In an append-only log file of raw trajectory strings
- In a readable 2D array indexed by discrete states and actions
For discrete small environments, Q-values are stored directly in a matrix/table mapping each (state, action) pair to a scalar value.
3Q is how good this step is
Sometimes you want a number per step, not per cell. Q(C1, Right) is the value of taking Right from C1: that step pays plus 10 and ends, so Q(C1, Right) is 10. Q(C1, Down) goes to C2, pays minus 1, and from C2 a Right would be the Pit — a much worse number.
Q is the action-value: how good is this action in this state. V is the cell-value: how good is this cell if you pick well. On a greedy cell, V equals the best Q. From C1, V is 10 because the best Q is the Right that enters Goal.
Figure. V asks how good the cell is. Q asks how good this step is. At C1 they match because Right is best.
| Symbol | Asks |
|---|---|
| V(C1) | How good is standing on C1? |
| Q(C1, Right) | How good is taking Right from C1? |
Q at C1
What is Q(C1, Right)? Why is it 10, not 8?
- C1 --R--> D1, reward10
- walk over, extra return0
- Q(C1, R)10
Pro tip. 8 is V(A1) or the undiscounted path return from Start. Q(C1, Right) is this step from C1.
Coding lab. Write one Q runs in the app, with checks on your output.
Q(C1, Right) is 10. V(C1) is also 10. Why do they match?
- Right is the best action from C1, so the cell value equals that Q
- V and Q are always the same word
- Goal pays 10 and 8 at the same time
On a greedy cell, V is the best Q. A worse action from C1 would have a smaller Q.
4Greedy means pick the bigger Q
Once each action from a cell has a Q, greedy is mechanical: pick the action with the largest number. From C1, if Q(Right) is 10 and Q(Down) is worse, pick Right. From A1, if Q(Right) is 6.2 and Q(Down) is smaller, pick Right.
You do not need a net to take that argmax. You need the table, and a comparison. Argmax just means 'the action whose Q is biggest'.
Figure. Greedy at A1 picks Right, because 6.2 is larger than 0. The comparison is the whole rule.
Argmax at A1
Q(A1, Right) = 6.2 and Q(A1, Down) = 0. Which action is greedy?
- compare 6.2 and 06.2 is larger
- greedy actionRight
Pro tip. Greedy reads the Q row and picks the max. It does not re-walk the path.
Coding lab. Argmax one row runs in the app, with checks on your output.
If Q(C1, Right) = 10.0 and Q(C1, Down) = -10.0, what does the greedy policy select at C1?
- Down, seeking to explore the negative outcome first
- Up, falling back to a default neutral action
- Random choice, because both values are non-zero
- Right, choosing the action with the substantially higher expected return
The greedy policy chooses argmax_a Q(s, a) = Right (10.0 > -10.0).
5The value of Start is the walk you already know
V(A1) = 6.2 is not a new fact. It is the discounted return of the short Goal path, written as a property of the starting cell. If you act well from Start — Right, Right, Right — that is how good Start is.
The next lesson will grow those numbers from walks, instead of writing them by hand. The meaning stays: a better Start value means a better policy from A1.
Figure. A1 shows 6.2 — the discounted return of the short Goal path, written as a property of Start.
Start value is the discounted short path
Show that V(A1) matches the discounted return of A1-B1-C1-D1.
- -1 + 0.9*(-1) + 0.81*106.2
- V(A1)6.2
Pro tip. Same 6.2, two names: discounted return of the walk, and value of Start.
Coding lab. Check Start's value runs in the app, with checks on your output.
What does the state value V(Start) represent under an optimal policy?
- The immediate reward obtained on the very first step out of Start
- The number of total grid tiles visited before reaching the goal
- The expected cumulative discounted return achievable when starting from that tile
- The learning rate multiplier used when initializing the Q-table
V(s) expresses how good it is to be in state s, measured by expected total discounted reward obtained by following the policy.
Notes
- In Reinforcement Learning because a value is how good a cell (or a step from a cell) is — a number, not a neural net.
- Stand on C1 and act well — Right, into Goal. The return from C1 is plus 10. So the value of C1 is 10. Stand on B1, step Right to C1, then act well: reward minus 1, then 0.9 times 10, which is 8. The value of B1 is 8.
- Deep Learning stores knowledge in the weights of a net. Here the knowledge is twelve numbers — or fewer, one per cell we care about — written in a table you can read. C1 is 10 because Right is Goal, not because a layer computed a hidden vector.
Exam traps & shortcuts
- Act, then read the number the floor paid. That number is this step, not the whole walk.
- A policy is a rule per cell. A value is how good that cell is. Neither is a neural net.
Recap
V(C1) is 10, V(B1) is 8, V(A1) is 6.2 under the greedy path with gamma 0.9. Q(C1, Right) is 10. Greedy from Q is: pick the action with the bigger number. Still a table, not a net.
- How good is this cell?
- Stand on C1 and act well — Right, into Goal. The return from C1 is plus 10. So the value of C1 is 10. Stand on B1, step Right to C1, then act well: reward minus 1, then 0.9 times 10, which is 8. The value of B1 is 8.
- A number in a table, not a neural net
- Here the knowledge is twelve numbers — or fewer, one per cell we care about — written in a table you can read. C1 is 10 because Right is Goal, not because a layer computed a hidden vector. If you can point at the cell and read the number, you have the idea.
- Q is how good this step is
- Sometimes you want a number per step, not per cell. Q(C1, Right) is the value of taking Right from C1: that step pays plus 10 and ends, so Q(C1, Right) is 10. Q(C1, Down) goes to C2, pays minus 1, and from C2 a Right would be the Pit — a much worse number.
- Greedy means pick the bigger Q
- Once each action from a cell has a Q, greedy is mechanical: pick the action with the largest number. From C1, if Q(Right) is 10 and Q(Down) is worse, pick Right. From A1, if Q(Right) is 6.2 and Q(Down) is smaller, pick Right.
Practise Value is how good a cell is
Reading is free and needs no account. Practice, mocks and progress live in the app.
- A 3-question practice set that ends the chapter
- 5 quick checks with worked explanations
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device