Reinforcement Learning · Reinforcement Learning Core
A policy is a rule per cell
In Reinforcement Learning because a policy is the rule that picks the next step from the cell you are in.
You can write, before any walk, what to do on each tile. On A1 go Right. On B1 go Right. On C1 go Right. That list is a policy: in this state, do that. Follow it from Start and the short Goal path happens by construction. A different list — sometimes try Down — is a different policy, not a different floor.
- Reinforcement Learning
- Medium level
- 5 concepts
1In this cell, do that
A policy is a lookup: given the cell, name the action. It is not a feeling and not a neural net. On paper it is a short table — A1: Right, B1: Right, C1: Right, A2: Up, and so on. Each row is one state you already know how to name.
Once the table is written, a walk is just following it. The agent does not invent a new rule mid-step. It reads the cell, reads the row, takes that action. Change the table and you have a new policy on the same twelve tiles.
Figure. A1, B1 and C1 each say Right. A policy is that lookup — in this cell, do that.
| Cell | Action |
|---|---|
| A1 | Right |
| B1 | Right |
| C1 | Right |
Coding lab. Read the rule runs in the app, with checks on your output.
What is a policy on this floor?
- A rule that names an action for each cell
- The plus-10 payment at Goal
- A labelled rent column
In this state, do that. The payment is a reward, not a policy.
2A greedy rule on the top row
Greedy here means: from this cell, take the step that looks best right now. On the top row that is Right, Right, Right — each step moves closer to Goal, and from C1 it is the plus-10 step.
Follow that rule from A1 and you walk A1-B1-C1-D1. Return 8, as already computed. Greedy is a kind of policy, not a second idea. It is one filled-in table, not a new floor.
Figure. A1, B1, C1 highlighted. The written rule is Right on each. The walk is the short Goal path.
Follow greedy from A1
Policy: A1/B1/C1 all Right. Which cells do you visit, and what is the return?
- cellsA1, B1, C1, D1
- return8
Pro tip. The policy named the actions. The return is still the sum of rewards.
Coding lab. Follow greedy runs in the app, with checks on your output.
What does a greedy policy do when presented with a state where action values are Left: 2.1, Right: 7.5, Down: 4.0?
- Explores Left with 75% probability to discover alternative paths
- Averages all three action values together to form a composite step
- Terminates the episode because multiple actions have positive values
- Deterministically selects Right because it has the highest estimated action value
A greedy decision rule always picks the action with the largest expected value without exploring alternatives.
3Sometimes try the other door
If the rule is always Right on the top row, you will never stand on A2. You will never learn what Down from Start is like — the door that leads toward rank 3, and also toward the long path. Explore means: most of the time follow the greedy rule, and sometimes pick a different legal action on purpose.
Exploration is not a second floor. It is a way of using the same policy table — follow it, but not every single step. Without it, a bad first rule can freeze and never meet Goal.
Figure. Greedy from Start is always Right. Explore keeps that rule and sometimes tries Down — the other door.
| Style | From A1 |
|---|---|
| Greedy | Always Right |
| Explore | Usually Right; sometimes Down |
Coding lab. Force one explore step runs in the app, with checks on your output.
A policy that is always Right on the top row never visits A2. What does explore add?
- Sometimes pick a different legal action on purpose
- A new floor with more tiles
- A neural net that invents actions
Explore uses the same floor and tries a door the greedy rule would skip.
4A walk is just following the table
Write a policy that is Right on A1, B1, C1 and also Up on A2 (so a wanderer who explored Down can climb back). Following it is a loop: read cell, read action, step, repeat until Goal or Pit.
The lab does that loop and prints the cells. If the table says Right, Right, Right, you will see A1 B1 C1 D1 again — the same short path, now as a policy being executed, not as a path you picked by hand.
Figure. Executing Right on A1, B1, C1 walks A1-B1-C1-D1. The walk is the policy applied.
Execute the table
Policy Right on A1, B1, C1. Start at A1. List cells until done.
- A1 reads RB1
- B1 reads RC1
- C1 reads RD1 done
Pro tip. The walk is the policy applied. You did not choose mid-step.
Coding lab. Execute until done runs in the app, with checks on your output.
In tabular reinforcement learning, what is a deterministic policy table?
- A neural network that outputs continuous probability distributions over states
- A lookup table that specifies exactly one action to take from each visited state
- A random number generator that selects steps without inspecting coordinates
- A list of past trajectories recorded during previous training episodes
A deterministic policy pi(s) is a mapping from state space to action space: given a state, it outputs the action to execute.
5Random is a policy too — a poor one
Picking U, D, L, R at random on every tile is still a policy. It will sometimes fall in the Pit at D2, sometimes bump Wall at B2, and only sometimes thread A1-B1-C1-D1. Its average return is worse than the directed top-row rule that scores 8.
You do not need to compute that average here. The point is the category: random is a rule ('pick any legal action'), just not a useful one on a floor that has a Goal and a Pit.
Figure. Both are policies. Directed Right threads Goal. Uniform random wanders and may hit the Pit.
| Policy | Typical walk |
|---|---|
| Top-row Right | A1-B1-C1-D1, return 8 |
| Uniform random | Wanders; may hit Pit |
Coding lab. One random action runs in the app, with checks on your output.
Why is a purely random policy inefficient for discovering goal states in complex environments?
- It causes mathematical divergence in Bellman equation calculations
- It is forbidden by the Markov Decision Process specification
- It overwrites existing Q-table values with negative infinity
- It wanders diffusely and takes exponentially long to reach distant reward states
Random walk exploration diffuses slowly and fails to exploit learned directional gradients, making discovery of distant rewards very slow.
Notes
- In Reinforcement Learning because a policy is the rule that picks the next step from the cell you are in.
- A policy is a lookup: given the cell, name the action. It is not a feeling and not a neural net. On paper it is a short table — A1: Right, B1: Right, C1: Right, A2: Up, and so on.
- Greedy here means: from this cell, take the step that looks best right now. On the top row that is Right, Right, Right — each step moves closer to Goal, and from C1 it is the plus-10 step.
Exam traps & shortcuts
- Act, then read the number the floor paid. That number is this step, not the whole walk.
- A policy is a rule per cell. A value is how good that cell is. Neither is a neural net.
Recap
A policy is a rule per cell. Greedy on this floor means the step that looks best right now. Explore means sometimes try the other door. The floor stays the same.
- In this cell, do that
- A policy is a lookup: given the cell, name the action. It is not a feeling and not a neural net. On paper it is a short table — A1: Right, B1: Right, C1: Right, A2: Up, and so on. Each row is one state you already know how to name.
- A greedy rule on the top row
- Greedy here means: from this cell, take the step that looks best right now. On the top row that is Right, Right, Right — each step moves closer to Goal, and from C1 it is the plus-10 step.
- Sometimes try the other door
- If the rule is always Right on the top row, you will never stand on A2. You will never learn what Down from Start is like — the door that leads toward rank 3, and also toward the long path. Explore means: most of the time follow the greedy rule, and sometimes pick a different legal action on purpose.
- A walk is just following the table
- Write a policy that is Right on A1, B1, C1 and also Up on A2 (so a wanderer who explored Down can climb back). Following it is a loop: read cell, read action, step, repeat until Goal or Pit.
Practise A policy is a rule per cell
Reading is free and needs no account. Practice, mocks and progress live in the app.
- A 3-question practice set that ends the chapter
- 5 quick checks with worked explanations
- Timed mocks scored with the real marking scheme
- Readiness tracked per topic, kept on your device