Transactions across services, simulated

The diagram. The client(s) on the left; in the middle the coordinator (2PC/3PC), the saga orchestrator or the broker; on the right the participants or services. In the outbox world the chain runs left to right: client, orders service, outbox relay (below), broker, consumer. Each card shows the node’s database in bold — a locked row has 🔒 and the transaction holding it, with → the value it will have if it commits — its in-memory state (transactions collecting votes, in doubt, saga progress as ✓ … ✗ ↺ ↩ ⏸), and, in the shaded strip, the tail of its durable log: ■ a forced write (the node waits for the disk), □ a lazy one. Down nodes are crossed out; cut links are red. Tap a node to crash or restart it, cut it off, make it vote NO or fail its next request, or drop its next vote; tap a client to start a transaction, a saga or an order, or (the other transaction) to read a row.

Counters. Committed and aborted transactions, forced writes, messages, reads blocked by locks; completed and compensated sagas, compensations, retries and duplicates; orders, published events, lost events, duplicates in the broker and effects applied twice — and safety violations.

Do something. Start a transfer (choose the amount; 200 is more than alice has), a saga (one that fails at shipping), or orders; crash, restart or isolate any node; drop the next message of a kind; heal; 20% random loss.

Transactions. Per transaction its decision and each participant’s state; per saga each step; per order whether it is in the database, the outbox, the broker (missing, once, duplicated) and how many times it was consumed.

Durable logs. The last ten records of every node’s log.

Invariants. Atomicity (never committed at one participant and aborted at another), a YES voter never aborting alone, money conserved (alice + bob = 150), what is in doubt and locked, and the cost per transaction of the 2PC variant (forced writes and messages, from Mohan et al.); for sagas money and stock conserved and every saga ending approved or compensated; for the outbox no lost events and exactly-once effects.

Two-phase commit (Gray 1978, Notes on Data Base Operating Systems). The coordinator sends PREPARE; each participant does its work, locks the rows, force-writes PREPARED with the new values and votes YES — or votes NO and aborts at once (insufficient funds, or a row locked by another transaction: no waiting, so no distributed deadlock). All YES → the coordinator force-writes COMMIT (the commit point) and sends it; any NO or a vote timeout → ABORT. Participants write the outcome, install or discard the values, release their locks and acknowledge; the coordinator writes END and forgets. A participant that voted YES and hears nothing is in doubt: it keeps its locks and sends STATUS? to the coordinator; a crashed participant re-reads its log, takes its locks again and asks. A coordinator that crashes after the votes leaves everyone blocked until it recovers from its log.

Presumed abort and presumed commit (Mohan, Lindsay & Obermarck, Transaction Management in the R* Distributed Database Management System, ACM TODS 1986). Presumed abort: no record before the decision, aborts are not logged at the coordinator and not acknowledged, and a question about an unknown transaction is answered ABORT. Presumed commit: a forced COLLECTING record with the participants before PREPARE, commits not acknowledged (and not forced at participants), aborts forced and acknowledged, and an unknown transaction is answered COMMIT; a coordinator recovering with COLLECTING and no decision must actively abort.

Three-phase commit (Skeen, Nonblocking Commit Protocols, SIGMOD 1981). CanCommit? / YES, PreCommit / ACK, DoCommit. A participant that times out runs a termination protocol with the participants it can reach: if any is pre-committed (or committed) it commits, otherwise it aborts. Non-blocking under crash failures with reliable timeouts; under a partition the two sides decide differently. Consensus-based commit — Gray & Lamport’s Paxos Commit (2006), or 2PC between replicated groups as in Spanner — is what avoids both problems; see Raft and Paxos, simulated.

Sagas (Garcia-Molina & Salem, Sagas, SIGMOD 1987; Richardson, microservices.io). An order saga of local transactions — create the order, charge, reserve stock, book a courier, confirm the reservation, approve — each with a compensation (reject, refund, release, cancel). Orchestration: a coordinator sends commands, logs every reply, retries a command with the same id until it is answered, and on a failure runs the compensations of the completed steps in reverse order. Choreography: services publish events to a broker that redelivers until acknowledged, and each service reacts to the events it subscribes to — failures travel back as events. Services keep the ids of processed commands and events, so a duplicate is answered without being applied twice. The semantic lock countermeasure marks reservations of unfinished sagas, so another saga is told to retry instead of being refused.

The outbox (transactional outbox, idempotent consumer). Without it the orders service commits and then publishes — two writes, one of which a crash can lose. With it the order and an outbox row commit together; a relay publishes unsent rows and marks them sent (a crash in between publishes again); a consumer applies each message and, in the same local transaction, records its id, then commits its offset (a crash in between redelivers). At-least-once delivery plus idempotent processing gives exactly-once effects. Try the runnable versions in Postgres: the outbox lab and idempotency keys; for transactions inside Kafka (KIP-98) see the Kafka simulator’s exactly-once scenario. For the wider picture, Kleppmann’s Designing Data-Intensive Applications, chapter 9.

One kind of transaction per world: a money transfer across two banks plus audit and notification participants; one order saga; one orders topic. Participants vote NO instead of waiting for locks, so there are no deadlocks to detect.

Coordinator logs are never truncated; recovery is instant; the coordinator’s own failure is modelled but not its replication. XA, heuristic decisions and read-only optimisations are not modelled.

The 3PC termination protocol is simplified to "ask whoever you can reach, commit if anyone is pre-committed"; it is there to show why 3PC is not used, not as a reference implementation.

The saga orchestrator’s saga log survives crashes as a whole; compensations always succeed eventually (as the saga model requires). Choreography has no time-outs of its own beyond broker redelivery.

The broker is a single durable log with one consumer; no partitions, no rebalancing — see the Kafka simulator for those.

The network is a fixed latency with jitter, optional loss, cut links and injected drops; a message is lost if its sender or receiver is down when it arrives.