Chris Collins

Tools · distributed

Offline-first sync and replication

How offline-first apps sync, animated: CouchDB/PouchDB-style replication between a phone, a laptop and a server that go offline — revision trees, the changes feed, checkpoints, _revs_diff and _bulk_docs, conflicts and the deterministic winning revision, three ways to resolve them, compaction and _revs_limit's false conflicts — against a Firestore-style single-leader cache with pending writes, server timestamps and rolled-back writes.

Added

Works with

How to use it

Two ways to build an app that keeps working without a network, side by side in one page. In the first, every device holds a full database and any two can replicate — the CouchDB and PouchDB model, where concurrent edits become branches of a document’s revision tree. In the second, one server holds the truth and each device keeps a cache with a queue of pending writes — the model of Cloud Firestore and similar mobile backends. Nothing here talks to a real database.

How to use it

The Multi-master replication and Single leader + cache tabs are simulators. Pick a scenario and press Start. Each step does something — takes a device offline, edits, reconnects, resolves a conflict — plays until it has happened, and pauses so you can read what happened and why, with links to the documentation. Next step moves on, Back rewinds (the simulator is deterministic, so it replays the earlier steps), ⏭ jumps to the end of the step and Step ⇥ advances to the next single event. Space plays and pauses; → is the next step. Each tab has a playground with no script. Speed scales everything; Hop time is how long a request takes to cross the diagram at 1× — while one is on its way the simulated clock slows down — and Sim clock turns that off. Neither changes what happens. ⛶ Full screen (or F) gives the diagram the whole window.

Multi-master replication. A phone, the server and a laptop, side by side (stacked on a phone screen). Each box shows the replica’s update sequence and checkpoints, what its push and pull replications are doing, and every document with its winning revision (green), a +n badge for other live leaves (_conflicts) and its body. Requests are labelled as they travel: GET _local checkpoints, _changes (a longpoll while waiting), _revs_diff, _bulk_docs and _bulk_get, and their answers. Tap a replica, or use Do something, to take a device offline or bring it back, edit, add list items, delete, create documents, resolve a conflict with a chosen strategy, compact, or drop the next message of a kind. Panels: Revision trees (one document on every replica, drawn as a tree: winner, conflicting and deleted leaves, stubs without bodies, and … where _revs_limit forgot older history), Documents (what GET ?conflicts=true answers on each replica), Changes feeds and checkpoints and Last conflict resolution (the leaves, their timestamps, the common ancestor, the merged body and the revisions written).

Single leader + cache. The same three boxes, but the server is the authority. Each client shows what its listener sees — its cache with its own pending writes laid over it, marked pending — and the server shows each document’s version and update time. Write from either client, take them offline, reconnect them in either order, stamp a server timestamp, add list items as a whole-array write or with arrayUnion, or break a rule. Panels: what each client sees next to the server, the queues of pending writes and rolled-back writes, and a comparison of the two models with CRDTs.

Revision tree. Write a revision tree as paths, one per line (1-a1 2-b7 3-c4, a line may branch from a revision named earlier, * marks a deleted leaf), or pick an example. It draws the tree, ranks the leaves with the reason each beats the next, shows what GET returns with _conflicts and _deleted_conflicts, and previews what a given _revs_limit would forget.

Scenarios

Multi-master replication:

  1. How replication works — revision ids, the push and pull replications, checkpoints in _local documents, the changes feed, _revs_diff, _bulk_docs with new_edits: false, the longpoll that wakes the other device, and why a replicator’s own changes come back as a no-op.

  2. Offline edits on two devices — both devices change the same note offline; nothing is rejected on reconnect, the server keeps both revisions as branches, and every replica picks the same winner by the same rule; the same edit made twice gets the same revision id and is no conflict.

  3. Resolving conflicts — a field-level three-way merge of edits to different fields; a real clash settled by last-writer-wins, which the phone’s slow clock fools; and a list merged as a set by an application merge on the server.

  4. Delete against edit — a deletion with the longer history loses to an edit, so the document comes back; deleting the remaining leaf makes the deletion stick.

  5. Interrupted replication resumes — six documents sent two at a time with a checkpoint per batch; a lost _bulk_docs answer, a timeout, a retry from the last checkpoint, and _revs_diff skipping what had already arrived.

  6. Compaction and _revs_limit — compaction keeps revision ids and drops old bodies; with _revs_limit 3 a laptop away for four edits gets a false conflict when it returns.

  7. Playground.

Single leader + cache:

  1. Pending writes and latency compensation — a write shown at once with pending writes, an offline queue, the queue flushed in order on reconnect.

  2. Server timestamps — the placeholder and its estimate, and an offline write stamped when it reaches the server.

  3. Last write wins (by arrival) — two offline edits: the later arrival wins field by field, nobody is told; whole-list writes lose an item; arrayUnion keeps both.

  4. A rejected write is rolled back — an offline write that breaks a rule is shown until the device reconnects, then refused and taken back.

  5. Single-leader playground.

What it models

  • Revisions. Every change to a document makes a new revision N-hash: N is one more than the parent’s, the hash is derived from the parent revision, the deleted flag and the body — CouchDB computes an MD5 over these and the attachments; this page uses a 128-bit hash built from four seeded 32-bit hashes, which behaves the same way (deterministic, so the same edit to the same parent always gets the same id). A deletion is a revision with _deleted: true, a tombstone. A PUT must name a leaf as its _rev, or it gets 409 Conflict.

  • Revision trees and the winner. A document is a tree of revisions; concurrent edits are branches. Every replica chooses the same winning revision from the leaves, without talking to anyone: a leaf that is not deleted beats a deleted one; then the leaf with the longer history (the higher N); then the leaf whose revision hash sorts higher as a string (Replication and conflict model; in CouchDB’s source the leaves are compared by {not Deleted, RevPos, RevId}). The other live leaves are _conflicts, the deleted ones _deleted_conflicts. When every leaf is deleted the document is deleted.

  • Replication follows the CouchDB replication protocol: read the checkpoint (a _local document named after the replication) from both ends and start from it only if both have one; read the source’s changes feed since then, in batches, with style=all_docs (each changed document once, at its latest sequence, with all its leaves); ask the target _revs_diff which revisions it lacks; fetch those from the source with their _revisions history (open_revs / _bulk_get); write them to the target with _bulk_docs and new_edits: false, which grafts the history onto the target’s tree without making new revisions; save the checkpoint on both ends. Each device runs a push and a pull replication, as PouchDB’s sync does (PouchDB: replication); the pull waits on a longpoll _changes request. A request without an answer fails the session, which restarts from the checkpoint.

  • Conflict resolution follows the pattern in the CouchDB and PouchDB guides: read the conflicting leaves, compute one body, write it as a new revision on the winning branch and write a deletion on every other live leaf. The strategies: keep the winner; last writer wins by the time each revision was written on its device (an application’s updatedAt field — kept beside the body here); a field-level three-way merge against the lowest common ancestor in the tree (two-way when the ancestor’s body is gone or the leaves share no ancestor); and an application merge that also treats lists as sets.

  • Compaction and stemming. Compaction removes the bodies of revisions that are not leaves; the ids stay. _revs_limit (default 1000) keeps only that many revision ids along each branch; a replica whose newest revision has been forgotten everywhere else cannot connect its history and gets a second root — a conflict between a revision and its own descendant.

  • The single-leader cache follows Cloud Firestore’s documented behaviour (offline data): writes apply to the local view at once and listeners see them with hasPendingWrites (listening); offline writes queue and are sent in order; the server applies each write to its current document — update merges fields, so the last arrival wins per field; serverTimestamp() is filled in at commit, and until then a listener sees an estimate from the local clock, the previous value or null (adding data); arrayUnion is applied at the server; a write the rules refuse is removed from the queue and the local view rolls back (Security Rules).

  • For sync built on CRDTs instead — data types whose replicas merge without conflicts — see Clocks, CRDTs and gossip.

Limits

  • Two documents to start with and three replicas in a star (no device-to-device replication, although CouchDB allows it). Bodies are small flat JSON objects; no attachments, no design documents, no validate_doc_update, no filtered replication, no _purge.

  • Revision hashes are not CouchDB’s MD5, so the ids differ from a real server’s for the same edits (the rules that use them are the same). The checkpoint comparison is simplified to "both ends have one, take the older"; CouchDB compares session histories. Batches are counted in changes; timeouts and retries are fixed and much shorter than a real replicator’s back-off.

  • PouchDB’s own revision ids and defaults may differ from CouchDB’s between versions; this page follows CouchDB.

  • The single-leader model is reduced to what the scenarios need: one document store, two rules standing in for Security Rules, one write in flight per client, no queries, no transactions or batched writes (which need the server), no multi-tab persistence and no cache eviction. Atlas Device Sync (Realm), another single-leader design with its own merge rules, is not modelled.

Linking to an example

Links can be written by hand. Add a tab after #: sim (the default), leader or tree. A link’s settings replace this page’s saved settings (a button restores yours); a scenario brings its own settings, and parameters given with it override them.

Parameter Meaning Example

example

A scenario from either list below (add #leader for the single-leader ones)

example=offline-conflict

scenario

The multi-master scenario, without its own settings

scenario=resolve

step

Open the multi-master scenario at this step

step=2

latency

One-way network latency, ms

latency=150

batch

Changes per replication batch

batch=1

revsLimit

_revs_limit on every replica, 1–1000

revsLimit=2

skew

The phone’s clock offset in ms (negative = slow), used by last-writer-wins

skew=-5000

strategy

Default resolution: winner, lww, field or app

strategy=lww

live

Continuous replication (false = only when you press Sync)

live=false

lscenario

The single-leader scenario (with #leader)

lscenario=rejected#leader

lstep

Open the single-leader scenario at this step

lstep=1

llatency

One-way latency in the single-leader pane, ms

llatency=200

lskew

The phone’s clock offset in the single-leader pane, ms

lskew=-4000

maxItems

The rule: most items in a list

maxItems=3

ts

Pending server timestamps show as estimate, previous or none

ts=none

tree

A revision tree for the calculator (with #tree), lines separated by %0A

tree=1-a%202-b%0A1-a%202-c#tree

seed

Random seed

seed=7

speed

Playback speed: 0.25, 0.5, 1, 2, 4 or 8

speed=2

hop

Hop time in seconds (0.25, 0.5, 1, 2, anything from 0.1 to 5) or clock

hop=1

Multi-master scenarios:

example= Shows

replication

The replication protocol step by step (open)

offline-conflict

Offline edits on two devices become a conflict with one winner everywhere (open)

resolve

Three-way merge, last-writer-wins with a slow clock, lists merged as sets (open)

delete-vs-edit

An edit beats a longer deletion; making the deletion stick (open)

interrupted

A lost answer mid-batch; the retry resumes from the checkpoint (open)

revs-limit

Compaction, and the false conflict of a too-small _revs_limit (open)

playground

Three replicas, no script (open)

Single-leader scenarios:

example= Shows

pending-writes

Latency compensation, the offline queue, the flush in order (open)

server-timestamp

serverTimestamp() estimated locally and filled in at commit (open)

last-write-wins

The later arrival wins per field; whole-list writes against arrayUnion (open)

rejected

An offline write refused by the rules and rolled back (open)

leader-playground

A server and two clients, no script (open)

Sources: the CouchDB documentation — Replication and conflict model, CouchDB Replication Protocol, /db/_changes, /db/_bulk_docs (new_edits), /db/_revs_limit, Compaction; CouchDB source (couch_key_tree, couch_doc) for the leaf ordering; PouchDB guides on replication and conflicts; Cloud Firestore documentation on offline data, realtime updates, adding and updating data, transactions and Security Rules.

Runs entirely in your browser; nothing you type leaves this page.