Reinforcement learning gridworld

An agent moves around a grid one cell at a time: up, right, down or left. Each move earns a reward — usually a small cost — and the episode ends at a goal or in a pit. The question reinforcement learning answers is which move to make in each cell to collect the most reward in the long run. This page works it out two ways: by planning with the full model of the world (dynamic programming), and by trial and error from the agent’s own experience (temporal-difference learning). The worlds include the two classics from Sutton and Barto’s Reinforcement Learning: An Introduction: cliff walking (example 6.6) and the windy gridworld (example 6.5).

How to use it

Pick a world from the list at the top, or paint your own: choose a brush (Wall, Pit, Goal, Start, Wind, Clear) and tap or drag across the grid. Inspect (the default) selects a cell instead. The size, the rewards and the slipperiness are under The world.
Pick an algorithm and press Step (one sweep, or one move of the agent), Run (animated, at the chosen speed), Converge / Episode, or +100 episodes. Reset forgets what was learned.
The grid shows each cell’s value as colour (green higher, red lower) and number, and the greedy policy as arrows (several arrows mean a tie). Q triangles splits every cell into the four action values instead. The orange dot is the learning agent and the line its current episode.
Tap a cell with Inspect to see its Bellman backup with the numbers in play, in the panel on the right.
The chart shows the reward of each episode with a moving average for the learning methods, and the largest change per sweep (Δ, log scale) for the planning methods.

The algorithms

Value iteration repeats the Bellman optimality backup for every cell: V(s) ← max over a of Σ p(s′, r | s, a) [r + γ V(s′)]. Each sweep, the reward at the goal spreads one cell further; Δ shrinks by about γ per sweep once it has. The panel lists, for the chosen cell, every action’s possible outcomes with their probabilities, so on slippery ice you can see why the best move is sometimes away from the goal.

Policy iteration alternates two steps: evaluate the current policy (sweeps of V(s) ← Σ p [r + γ V(s′)] using only the policy’s action) until it settles, then improve it by switching each cell to the greedy action. It usually needs only a handful of improvements.

Q-learning, SARSA and Expected SARSA don’t know the model: the agent moves, sees a reward and the next cell, and nudges one action value towards a target:

Q-learning: target = r + γ max over a′ of Q(s′, a′) — the value of acting greedily from now on, whatever the agent actually does next (off-policy).
SARSA: target = r + γ Q(s′, a′) for the action a′ it really will take next, exploration included (on-policy).
Expected SARSA: target = r + γ times the ε-greedy average of Q(s′, ·).

Q ← Q + α (target − Q). Exploration is ε-greedy: with probability ε a random move, otherwise the best known one; ε decay multiplies ε after each episode.

Cliff walking: Q-learning against SARSA

Load the cliff world and press Q-learning vs SARSA. Both learn for 500 episodes with ε = 0.1 (averaged over three runs). Q-learning learns the shortest path along the cliff edge — the optimal one — but while it keeps exploring it falls off now and then, so its reward per episode is worse. SARSA learns the value of the policy it is actually following, falls included, and settles on a path a row or two up: worse in theory, better in practice while ε stays at 0.1. Set ε decay below 1 and both end up on the edge.

The world

Rewards: per move, for arriving at a goal, and for falling in a pit. A pit either ends the episode or (as in cliff walking) sends you back to the start.
Slip: the chance that a move goes sideways instead — split evenly between the two perpendicular directions. Frozen lake uses 2/3, so you go where you meant only a third of the time.
Wind: each column can push you up 1–3 cells after your move, using the wind of the column you move from, as in the book.
Bumping into a wall or the edge leaves you where you are (and still costs the move).

Related: the maths of machine learning (gradient descent, entropy, softmax), classic machine learning step by step (decision trees to optimisers), the neural network designer, and train a tiny transformer in your browser.

Browser support

Any current browser, on a phone too. Painting uses pointer events, so a finger works like a mouse.

Privacy

Everything runs in your browser. Your grid and settings are kept in this browser’s local storage and go into a link only when you press Share.