Distributed patterns on Redis

The problem, What we want and How Redis does it, with the commands (each links to its page in the Redis client’s reference) and the Lua scripts, with Copy.

Run it — the labs. Chips across the top choose one: race (without the pattern: watch it go wrong), fix (with it), failure (a crash, pause, failover or partition the pattern does not survive) and walk-through. The diagram is a sequence chart with a column per client, one for Redis, one for a replica when the lab has one and one for the world outside Redis (a database, a payment provider, a file). Each row is one event: a command with its reply (an arrow to Redis, a heavier one for a script, whose inner commands open under inside the script), a reply that arrives later to a blocked client, a pushed message, a pause, a wait, a failover. Play steps through on its own (½×, 1×, 2×); Step, Back, Reset and End move by hand (→, ← and space work when the lab has focus). Beside it, every key after the current event with its type, TTL and value; the replica’s keys; the state outside Redis; clients that are paused, blocked or cut off; and at the end the verdict — what went wrong, or why it held.

Open in the Redis client opens the lab’s commands in the Redis client with each line in console pane A or B (A> …, B> …; a third client goes in pane B), so the interleaving is replayed over two real connections — on its built-in Redis, or on a server you connected there. Pauses and waits become comments: wait that long yourself.

Failure modes (crash, failover, restart, clock drift, process pauses, partitions, eviction, hot and big keys, duplicates), It guarantees / It does not guarantee, Variants, See also (the Postgres lab for the same pattern, related patterns) and Sources.

For nine patterns the earlier widgets are kept underneath: the four rate limiters on the same traffic (Fixed window), the lock and fencing timeline (Simple lock), Redlock over five masters, stampede protection compared, and command-by-command walk-throughs (cache-aside, leaderboard, streams, reliable queue, sessions).

Cluster. At the top the client — with its cached slot map drawn as a strip of all 16,384 slots coloured by owner — and redis-cli, which runs reshardings. Below, one card per master and, under it, its replicas: role, slot ranges, number of keys, configEpoch and currentEpoch, replication offset, and the node’s own slot map (its slots bright, the rest faded; a red dashed outline is a slot it is MIGRATING, green one it is IMPORTING). PFAIL and FAIL flags a node holds, and a replica’s election, appear under it. Commands, replies, MOVED and ASK, replication, FAIL broadcasts, vote requests and votes, SETSLOT and MIGRATE are labelled packets; the cluster-bus pings and pongs are faint dots. Tap the client to send commands, a node to crash, restart or isolate it, redis-cli to reshard. Do something takes a key (or several, comma-separated, for MGET and MSET), gives the client a stale map for that key’s slot, reshards any slot range between two masters, and crashes, restarts or isolates nodes. The Slot map panel lists each node’s view; Client requests shows each command’s path, such as M1 → ASKING+M3.

Sentinel. A client, the sentinels and a master with its replicas. Each sentinel shows which nodes it thinks are subjectively down, how many sentinels agree, whether the master is objectively down, its epoch and vote, and whether it is leading a failover. The data nodes show a counter the client increments, their replication offset, replica priority and run id.

Hash slots. Type keys, one per line: each one’s hashed part (the hash tag) is marked, with its CRC16, its slot and which master owns it for 3–10 masters; with several keys it says whether a multi-key command over them would be allowed or refused with CROSSSLOT.

Mutual exclusion & coordination — Simple lock (SET NX PX), Lock renewal (watchdog), Reentrant lock, Read-write lock, Fair (queued) lock, Counting semaphore, Fencing tokens, Redlock and its critique, Leader election with leases, Distributed barrier, Countdown latch, Run once (SET NX on a job id).

Optimistic concurrency & transactions — Check-and-set with WATCH / MULTI / EXEC, MULTI/EXEC semantics (no rollback), Atomic read-modify-write with Lua, Redis Functions, Compare-and-swap on versions.

Queues & messaging — List queue (LPUSH / BRPOP), Reliable queue (BLMOVE + processing list), Priority queue, Delayed and scheduled jobs, Retries with backoff and a dead-letter queue, Pub/sub fan-out (fire-and-forget), Sharded pub/sub (SSUBSCRIBE / SPUBLISH), Streams with consumer groups, Recovering pending entries (XAUTOCLAIM), Trimming streams, Exactly-once effect from at-least-once delivery, Stream as an event log (event sourcing), Work stealing.

Watching & change notification — Keyspace notifications, Client-side caching with CLIENT TRACKING, Blocking reads as notification, Replication acknowledgement (WAIT / WAITAOF).

Rate limiting & admission — Fixed-window rate limit, Sliding-log rate limit, Sliding-window counter, Token bucket, Leaky bucket, GCRA (generic cell rate algorithm), Concurrency limiter (in-flight requests), Distributed circuit breaker.

Caching — Cache-aside (lazy loading), Read-through with single-flight loading, Write-through, Write-behind (write-back), Cache stampede protection, Stale-while-revalidate, Tag-based invalidation, Negative caching, Near cache with client-side caching.

Idempotency & deduplication — Idempotency keys with stored responses, Deduplication window (SET NX EX, Bloom filter), Exactly-once effect.

Counting & analytics — Atomic and sharded counters, Unique counts with HyperLogLog, Daily actives with bitmaps, Bloom, Cuckoo, Count-Min and Top-K, Time-series rollups.

Data-structure patterns — Leaderboards, Sliding-window top-N, Secondary indexes, Autocomplete with lexicographic ranges, Nearby search with GEO (drivers, shops), Session store, Presence: who is online, Feature flags and percentage rollouts, Unique ID generation.

Service coordination — Service discovery with TTL heartbeats, Distributed configuration with change notification, Job scheduler (cron) across many servers, Saga and workflow state.

Cluster-aware design — Hash tags for multi-key atomicity, Avoiding hot keys, Big keys and how to split them.

Lua. EVAL, EVALSHA, SCRIPT, FUNCTION LOAD and FCALL run on the Lua interpreter, with redis.call / redis.pcall, the conversions between Lua and Redis replies, read-only scripts, #!lua flags=…, and Redis 7.2’s error messages. A script runs atomically: nothing else happens until it returns.

Replication and failover. A lab can have a replica. It receives the primary’s data only when the lab says so (replicate) or a client calls WAIT — which is what asynchronous replication means: an acknowledged write can still be lost. A failover promotes the replica, the clients reconnect to it, and the writes it never received are listed as lost. Functions replicate; the EVAL script cache does not.

Blocking commands (BLPOP, BLMOVE, BZPOPMIN, XREAD … BLOCK, XREADGROUP … BLOCK) block the client until another client’s write lets them return, in the order they blocked, as Redis serves them.

Pushes — pub/sub messages, keyspace notifications and CLIENT TRACKING invalidations — arrive at the subscribed client as rows of their own, right after the command that caused them. A disconnected client misses them, as on a real server.

Process pauses, clock skew, partitions, crashes, eviction and restarts are steps: a paused client does nothing while time passes; a skewed client computes with its own clock while Redis keeps its own; a client cut off gets a timeout; eviction deletes a key as maxmemory would.

The world outside Redis is plain state the app steps change (rows in a database, charges at a payment provider), so a lab can end with a verdict on whether the outcome was right.

Hash slots. HASH_SLOT = CRC16(key) mod 16384, with the specification’s CRC16 (XMODEM: polynomial 0x1021, initial value 0, not reflected — CRC16("123456789") = 0x31C3) over the key’s bytes, and its hash-tag rule: if the key contains a { and a later } with at least one character between them, only that part is hashed. Slot ranges are divided as redis-cli --cluster create does. The tests check the specification’s test vector and its hash-tag examples.

Routing. A node answers a command whose keys are all in one slot it owns. Keys in different slots: -CROSSSLOT. A slot it does not own: -MOVED <slot> <node>. A slot it is MIGRATING: it serves the command if all the keys are still there, answers -ASK <slot> <target> if none are, and -TRYAGAIN if only some are. A node IMPORTING a slot serves a command only after ASKING. A replica answers MOVED to its master. The client updates its map on MOVED (and, by default, refreshes the whole map as most clients do), follows ASK once without changing its map, and on a connection timeout refreshes its map from another node and retries.

Resharding follows the steps in the specification and redis-cli: SETSLOT IMPORTING on the target, SETSLOT MIGRATING on the source, MIGRATE key by key (the key is deleted from the source only once the target has it), then SETSLOT NODE on the target — which takes a new configEpoch without agreement — and on the source.

Failure detection and failover. Nodes ping each other on the cluster bus. No reply for node-timeout → PFAIL; a node with PFAIL and reports from a majority of the masters that serve slots (within 2 × node-timeout, counting itself if it is a master) marks FAIL and broadcasts it. With cluster-require-full-coverage yes a node answers CLUSTERDOWN while a FAIL master serves slots, and a master that cannot reach a majority of masters does the same. A replica of a FAIL master waits 500 ms + random(0–500 ms) + rank × 1000 ms, increments currentEpoch and asks every master for a vote; a master votes once per epoch, only if it also sees the master as FAIL, and not twice for replicas of one master within 2 × node-timeout. With a majority the replica takes its master’s slots with the new epoch as configEpoch. A node accepts a claim to a slot only with a higher configEpoch than the current owner’s; a master that loses all its slots becomes a replica of the node that took them and resynchronises, discarding writes the new master never received. Replication is asynchronous: a master answers before its replicas have the write.

Sentinel. Each sentinel pings the master and replicas; no valid reply for down-after-milliseconds → SDOWN. A sentinel with the master SDOWN asks the others (is-master-down-by-addr); with quorum agreeing it is ODOWN. It then waits a random part of a second, starts a new epoch, votes for itself and asks for votes; every sentinel votes once per epoch, for the first that asks. The winner needs max(quorum, majority of sentinels) votes. It picks a replica — not down, lowest replica-priority (0 is never), largest replication offset, smallest run id — sends REPLICAOF NO ONE, then REPLICAOF to the other replicas, and the new configuration spreads to the other sentinels with its epoch. A failover that finds no leader in failover-timeout is abandoned and retried later. A returning old master is converted to a replica. With min-replicas-to-write N a master refuses writes (NOREPLICAS) unless N replicas have acknowledged within min-replicas-max-lag.

The labs are models of the interleavings that matter, not load tests: one command at a time, in the order shown. The real-server replay keeps that order; it does not race the clients.

One Redis. Redlock’s five masters and a cluster’s shards are modelled with separate keys and app steps, which say so; the Cluster tab is the place for real multi-node behaviour.

The page’s Lua is Lua 5.1 with the libraries Redis scripts use (string, table, math, cjson, bit); not cmsgpack, struct or the debugger. A script that runs more than five million steps is stopped with an error.

Keyspace notification event names follow Redis for the common commands. RedisBloom and RedisTimeSeries commands are not available (their labs use core commands).

Cluster and Sentinel timeouts are shortened so failures play out in seconds: node-timeout 1.5 s instead of 15 s, down-after-milliseconds 1 s instead of the sample configuration’s 30 s, failover-timeout 3 s instead of 3 minutes. Network latency is fixed apart from a little jitter.

The cluster bus is simplified: every node pings every other at a third of node-timeout, and a ping carries the sender’s whole slot claim and its PFAIL list (real nodes ping a few random nodes each second and gossip about a sample of others). Not modelled: replica migration between masters, manual CLUSTER FAILOVER, READONLY reads from replicas, multiple databases, cluster-bus authentication, persistence and partial resynchronisation (a demoted node always does a full sync). Keys are user:N strings.

Sentinel: one monitored master; sentinels find each other and the replicas from the start rather than through hello messages and INFO; the client is a simple Sentinel-aware client.

Redlock’s widget acquires one node after another with a fixed latency; the critique of its timing assumptions is described and shown as labs, not simulated in time.