MongoDB replica sets and sharding, simulated

Replication. The primary applies a write, appends it to its oplog with an optime (term, position) and its wall-clock time, and journals it (at once with j: true, otherwise at the next group commit, 100 ms). Each secondary keeps a getMore open on its sync source’s oplog (a tailable, awaiting cursor), applies the batches it receives and reports its applied and journaled positions back (replica set data synchronisation). The primary writes a no-op every 10 s when idle, as periodicNoopIntervalSecs does, so that staleness can be measured.

Write concern (manual). w: 0 is not acknowledged; w: 1 when the primary has applied it (and journaled it with j: true); w: n when n data-bearing members have applied it; w: "majority" when a majority of voting data-bearing members have journaled it — the count is the smaller of a majority of all voting members and the number of voting data-bearing members, so an arbiter votes but never acknowledges. wtimeout returns an error but leaves the write in place. A primary that steps down answers its waiting writes with an error; retryable writes (on by default) retry once on the new primary.

The commit point. The highest oplog entry of the primary’s own term that a write-majority has journaled; everything up to it is majority-committed and can never be rolled back. Members learn it from oplog batches and heartbeats.

Elections (protocol version 1; replica set elections). Heartbeats every 2 s. A secondary that is electable (data-bearing, priority above 0, not hidden or delayed) starts an election when it has not heard from a primary for electionTimeoutMillis (10 s) plus a random offset of up to 15 %. It first runs a dry run (asking whether voters would vote for it in the next term, without incrementing its own), then increments its term, votes for itself and asks for votes. A voter refuses a candidate whose term is older than its own, one it when it has already voted for another candidate in that term, or one whose last applied optime is older than its own. A majority of votes makes it primary; it catches up from any member it can see that is further ahead, then writes a no-op in the new term and accepts writes. A primary steps down when it sees a higher term or cannot see a majority of voting members for an election timeout. A caught-up secondary with a higher priority than the primary calls a priority takeover election.

Rollback (rollbacks during failover). A member whose oplog diverges from its sync source’s finds the last common entry, undoes everything after it (saving the documents to a rollback file) and fetches from there. Writes acknowledged with w:1 that the majority never had are counted as acked but lost. A crash loses what was applied but not yet journaled.

Read concern. local (and available) reads the member’s latest applied data; majority and snapshot read the member’s majority-committed data; linearizable runs only on the primary, which writes a no-op and waits for a majority to acknowledge it before answering (read concern).

Read preference and staleness. The driver checks every member it can see every 10 s (heartbeatFrequencyMS; hidden members and arbiters are never in its list). Server selection follows the drivers' Server Selection specification: primary, primaryPreferred, secondary, secondaryPreferred, nearest; with maxStalenessSeconds each secondary’s staleness is estimated as (S.lastUpdateTime − S.lastWriteDate) − (P.lastUpdateTime − P.lastWriteDate) + heartbeatFrequencyMS (or from the freshest secondary when there is no primary), and values below 90 s or below heartbeatFrequencyMS + idleWritePeriodMS are refused (Max Staleness specification). Among eligible members, those within localThresholdMS (15 ms) of the fastest round trip are chosen from at random.

Sharding (manual). Documents belong to ranges [min, max) of the shard key, compared in BSON order (MinKey, numbers, strings, ObjectIds, dates, MaxKey); compound keys compare field by field; a hashed field is replaced by its hash. A new collection with a ranged key is one range on the primary shard; with a hashed key it is pre-split over every shard (two ranges each here). mongos routes with a cached copy of the routing table; a shard that sees an older shard version answers StaleConfig and mongos refreshes from the config servers and retries. Zones split ranges at their boundaries and keep them on their shards.

The balancer (sharded cluster balancer), as in 6.1 and later: ranges are not split on insert; each round, a range in the wrong zone moves first, then if the fullest and emptiest shards differ by at least three times the range size, a range of about the range size is split off the fullest shard and moved to the emptiest — clone, catch-up, commit on the config servers (the collection’s major version goes up), then the donor deletes its copy in the background. A range above the range size whose documents all share one key value cannot be split, is marked jumbo and is never moved.

Query targeting (query routing). The filter’s bounds on the shard key: equality or $in on a prefix narrows to points, the first range ends the narrowing (later fields are MinKey…MaxKey), a filter without the first field is sent to every shard, a hashed field is narrowed only by equality, $or is targeted clause by clause. Range operators bracket by type, as MongoDB’s do. The shards whose ranges overlap the bounds are asked; explain on mongos shows SINGLE_SHARD or SHARD_MERGE.

The shard-key evaluator (choose a shard key, analyzeShardKey). Cardinality is the number of distinct key values; frequency the share of the most common ones; monotonicity the correlation between the order of documents by key and by insertion, called monotonic at |r| ≥ 0.7. The dataset is split into ranges of the chosen size (never between equal keys) over the shards, the next 300 generated orders are placed to see where inserts go, and five typical queries are checked for targeting.

Real timings (heartbeats, election timeout, the driver’s 10 s checks) but played faster; the network latency is fixed and small. A few members, three keys (x, y, z) and a handful of values; documents are not stored in the replica set, only the oplog that sets them.

Not modelled in the replica set: initial sync, chained replication (every secondary syncs from the primary), votes: 0 members, reconfiguration, catch-up takeover, read concern available’s difference on sharded clusters, `afterClusterTime and causal consistency, transactions, the oplog’s size and window, flow control, and the WiredTiger cache pressure in a PSA set (described, not simulated). snapshot is shown as the member’s majority-committed state.

In the sharded cluster each shard is one box, not a replica set; there is one mongos and one config server box. Range size is counted in documents (standing in for 128 MB). Not modelled: auto-merging, moveChunk throttling and _waitForDelete, orphaned documents, resharding, refineCollectionShardKey (described), sharded transactions, aggregation pipelines across shards, sorts (SHARD_MERGE_SORT) and updates or deletes without the shard key.

The hash is a 32-bit stand-in for MongoDB’s 64-bit hash of the value (an MD5-based hash), so the ranges a hashed value falls into are not the ones a real cluster would use; the spread is the same.

The evaluator’s thresholds (when cardinality is "low", a value "hot", inserts "concentrated") are this tool’s judgment, not MongoDB’s.