Kafka, simulated

The diagram. An architecture diagram of the whole system, drawn from the simulator’s state: producers on the left, the brokers in the middle as tall cards, the consumer group on the right, and the KRaft controller quorum above (★ marks the active controller). Inside each broker every partition replica is drawn as its log — one cell per offset, coloured by key, with offsets underneath; faded cells are above the high watermark (written but not yet committed), the black bar is the high watermark and the dashed slot the log-end offset. The leader’s row is outlined, an in-sync follower’s plain and a replica out of the ISR dashed red. The same partition sits on the same lane in every broker, so replication runs straight across. Wires show who talks to whom: produce requests from each producer to the partition leaders (orange), followers fetching from their leader (blue), leaders feeding the consumer that owns each partition (green), brokers and controllers exchanging metadata (violet), consumers and the group coordinator (dashed yellow, with __consumer_offsets shown in its card) and transactional producers and the transaction coordinator (dashed pink). Messages travel the wires as labelled packets — a record’s key, the offsets a fetch carries, ack @offset, a commit — and a cell lights up as it is appended. A dead broker is crossed out and its wires go grey; a network partition cuts wires (✕); a new leader is announced by a badge travelling from the old leader’s row; records truncated away flash red; a rebalance moves the consumer wires and shows revoked partitions struck through. Consumer cards show each assigned partition’s position and lag. With reduced motion set in your system, messages jump to where they are going instead of travelling. Tap a broker, controller, producer or consumer to inspect it and act on it. On a phone the diagram stacks top to bottom.

Delivery counters. Sent, acknowledged, in flight, failed, acked but lost (acknowledged to the producer but no longer in the current leader’s log), duplicates written by retries, processed, processed twice and skipped.

Partition logs. Every replica of every partition, one cell per offset, coloured by key. Cells above the high watermark are faded (written but not committed); the bar is the HW and the number on the right the log-end offset. Segment boundaries, transaction markers (C/A), aborted transactions, tombstones (∅), compacted gaps and deleted segments are drawn too.

Consumer group. The protocol, generation (or group epoch), state and coordinator; each member’s partitions; and per partition the end offset, the committed offset and the lag.

Do something. Send records (with a key, a value, or as a tombstone), steady traffic, change acks and idempotence, lose the next request or ack on the network; kill, restart, slow the disk of, or cut off any broker; kill the active controller; add, close, crash or restart consumers; begin, commit and abort transactions.

Event log and internal logs: the metadata log (cluster_metadata), committed offsets (consumer_offsets) and transaction state (__transaction_state).

Settings rebuild the cluster: brokers, partitions, replication factor, min.insync.replicas, acks, idempotence, transactions, batching, unclean election, consumers, protocol, assignment strategy, commit timing, isolation level, retention and compaction.

Topics and logs. Partitions are append-only logs of offsets, stored in segments that roll at segment.bytes / segment.ms. Replicas are placed as Kafka’s rack-unaware assignment does (with its random start fixed at 0).

Producers. Keys go to toPositive(murmur2(key)) % partitions, the same murmur2 as the Java client (checked against Kafka’s own test vectors); null keys use the uniform sticky partitioner (KIP-794). Records are batched per partition by linger.ms and batch.size, sent one batch per partition per request, retried with exponential backoff (KIP-580) until delivery.timeout.ms.

acks. acks=0 is never answered; acks=1 is answered after the leader’s write; acks=all checks the ISR against min.insync.replicas before writing (NOT_ENOUGH_REPLICAS), and answers once the high watermark passes the batch (NOT_ENOUGH_REPLICAS_AFTER_APPEND if the ISR shrank meanwhile) — see acks.

Idempotence (on by default since Kafka 3.0, KIP-679): producer id, epoch and per-partition sequence numbers; the leader answers a duplicate batch with its original offset and rejects a gap with OUT_OF_ORDER_SEQUENCE_NUMBER; it remembers the last five batches per producer.

Transactions (KIP-98): the transaction coordinator is the leader of __transaction_state partition abs(transactionalId.hashCode()) % 50; InitProducerId bumps the epoch and fences zombies; AddPartitionsToTxn, EndTxn, PrepareCommit/PrepareAbort, COMMIT/ABORT control markers, the last stable offset (LSO) and the aborted-transaction index that read_committed consumers use.

Replication. Followers fetch from the leader; the leader tracks each follower’s position and "caught up" time exactly as Kafka’s Replica.updateFetchState does; the high watermark is the smallest log-end offset in the ISR. A follower not caught up within replica.lag.time.max.ms is removed from the ISR, and added back once it reaches the HW, through AlterPartition requests to the controller (KIP-497); the controller never adds a fenced broker (KIP-841). Followers detect divergence through their last leader epoch and truncate (KIP-101, KIP-320).

KRaft. Brokers heartbeat to the active controller; after broker.session.timeout.ms it fences a silent broker, removes it from every ISR and elects new leaders from the ISR at a new leader epoch, writing PartitionChangeRecords to the metadata log that every broker replays (KIP-500, KIP-631). If the last ISR member is lost the ISR keeps it and the partition goes offline — unless unclean.leader.election.enable lets an out-of-sync replica lead. The controllers form a Raft quorum (KIP-595): without a majority nothing changes. A restarted broker registers again and stays fenced until it has caught up with the metadata.

Consumer groups. The group coordinator is the leader of __consumer_offsets partition abs(groupId.hashCode()) % 50. The classic protocol runs JoinGroup / SyncGroup with a client side assignor: eager (range, round-robin, sticky — every partition revoked) or cooperative (cooperative-sticky, KIP-429 — only moved partitions, then a follow-up rebalance). The next-generation consumer protocol (KIP-848, generally available in Kafka 4.0, group.protocol=consumer) computes the assignment on the coordinator (uniform or range) and lets each member reconcile through heartbeats with no group-wide barrier. Commit timing decides delivery: before processing is at-most-once, after processing and auto-commit are at-least-once, and transactional (offsets committed atomically with the results, as with sendOffsetsToTransaction, KIP-447) is exactly-once.

Retention and compaction. Whole segments below the high watermark are deleted by retention.ms (largest timestamp) and retention.bytes; the log start offset moves up and consumers below it reset (auto.offset.reset). With cleanup.policy=compact the cleaner keeps the newest record per key outside the active segment once the dirty ratio reaches min.cleanable.dirty.ratio; tombstones survive one cleaning and go after delete.retention.ms; offsets never change (log compaction).

One topic per cluster, one consumer group, up to three producers. Internal topics (consumer_offsets, transaction_state) are not replicated logs here: their coordinators move with broker availability and their state survives.

Timeouts are about a tenth of Kafka’s defaults so failures play out in seconds (each setting shows Kafka’s default); latency is a fixed one-way delay, disks a fixed write time.

A "kill" is a process crash: the log files survive (Kafka does not fsync each write by default, so a machine crash could lose more). No rack awareness, no preferred-leader election, no tiered storage, no quotas, no compression, no follower fetching (KIP-392), and no Eligible Leader Replicas (KIP-966).

The sticky assignors follow Kafka’s algorithm for equal subscriptions; where Kafka iterates a hash map, members are taken in sorted order. KIP-848’s uniform assignor is approximated by the same sticky, balanced algorithm.

The throughput tab is a back-of-the-envelope model, not a benchmark.