Elasticsearch, simulated

The diagram. At the top, the client, the coordinating-only node (it routes writes and scatters and gathers searches) and the master-eligible nodes, with ★ on the elected master and the current term, cluster-state version and voting configuration. Below, one card per data node (tier, zone, and pills for down, slow, cut off and no master), with a box for every shard copy: P primary or R replica, its sequence number and local and global checkpoints, and three rows — the indexing buffer (one chip per document not yet in a segment), the translog (one cell per operation since the last flush; filled = fsynced, outlined = not yet) and the segments (width by document count; blue = live documents, red hatching = deleted documents, dashed outline = not committed by a flush, faded = written but not yet searchable). Packets are labelled: an index request and the shard it was routed to, seq N replication and its ack, query / top k / fetch for search, v12 / commit for cluster-state publication, votes between master nodes, recovery transfers. Inside a shard, refresh, fsync, flush and merge _1+_2+_3 show where they act. A promoted replica flashes. On a phone the diagram stacks top to bottom and busy nodes switch to one line per shard. Tap a node or a shard to inspect it and act on it (kill, power cut, restart, slow down, cut off the network; refresh, flush or force-merge one shard copy).

Counters. Cluster health, the master, the cluster-state version and term, writes acknowledged and failed, acked but lost (acknowledged to the client but no longer in the current primary), searchable and not-yet-searchable documents, segments, deleted documents, unassigned shards and average search time.

Do something. Index documents (with an id, a text, a count and refresh=false|wait_for|true), delete or GET by id, steady traffic; search with a query and from, take the next page with search_after, open a point in time; _refresh, _flush, _forcemerge, change the number of replicas; kill, power-cut, restart, slow down or cut off any node; add a data node; heal the network.

REST console. Type requests as you would with curl or Kibana’s Dev Tools; responses come back when the simulated cluster answers, in Elasticsearch’s format. help lists what it understands: PUT /idx (with number_of_shards, number_of_replicas, refresh_interval, index.translog.durability), POST /idx/_doc, PUT /idx/_doc/<id>?routing=&refresh=, GET /idx/_doc/<id>?realtime=false, DELETE, GET /idx/_search?q=&size=&from= or a match query body with search_after and pit, POST /idx/_refresh, POST /idx/_flush, POST /idx/_forcemerge?max_num_segments=1, PUT /idx/_settings, POST /idx/_pit, POST /alias/_rollover, GET _cluster/health, GET _cluster/state, GET _cluster/allocation/explain, and _cat/health, _cat/nodes, _cat/master, _cat/indices, _cat/shards, _cat/segments, _cat/allocation (add ?v for headers).

Last search. The query, its phase, the time each shard took and on which copy, how many entries the coordinating node merged and kept, the top hits, and the adaptive replica selection statistics the coordinating node keeps for each data node.

Shard copies (like _cat/shards with the engine inside) and the event log, with a filter for cluster-state publications and elections.

Settings rebuild the cluster: Elasticsearch or OpenSearch, master-eligible and data nodes, a coordinating-only node, data tiers, zones and awareness, shards, replicas, the routing function, routing_partition_size, starting documents, refresh interval, translog durability and sync interval, wait_for_active_shards, delayed allocation timeout, OpenSearch segment replication, segments_per_tier, adaptive replica selection, legacy minimum_master_nodes, latency and seed.

Routing. shard = hash(_routing) % number_of_primary_shards is how the documentation explains it. Exactly: the hash is Elasticsearch’s Murmur3HashFunction — Lucene’s murmurhash3_x86_32 with seed 0 over the routing string’s UTF-16 code units, low byte first — and indices created by 7.0–9.3 compute floorMod(hash, routing_num_shards) / routing_factor, where routing_num_shards is the largest number_of_shards × 2^x ≤ 1024 so the index can be split; indices created by 9.4 use floorMod(hash, number_of_shards). routing_partition_size adds floorMod(hash(_id), size). The tests check these against Elasticsearch’s own Murmur3HashFunctionTests and IndexRoutingTests (all 99 values of testBWC).

Writes. The coordinating node resolves the shard and forwards to the primary; the primary checks wait_for_active_shards, assigns the next sequence number under its primary term, applies the operation, replicates it to every copy in its routing table, waits for all of them, fsyncs the translog when index.translog.durability: request, and answers with _shards, _seq_no, _primary_term and _version. A replica that does not answer is reported to the master and removed from the in-sync set before the write is acknowledged; a primary that cannot reach a master does not acknowledge (data replication model). Local and global checkpoints are tracked per copy. Updates are a delete of the old version plus an add; optimistic concurrency with if_seq_no returns 409 on a conflict. A write to an index that does not exist creates it (1 shard, 1 replica).

Inside a shard. New documents go to the indexing buffer and the translog, and their id into the live version map. A refresh (every index.refresh_interval, 1 s by default) writes the buffer as a new segment, applies pending deletes and opens it for search; a realtime GET of an id still in the version map refreshes only the internal reader. A flush is a Lucene commit: the buffer becomes a segment, every segment is fsynced, the commit point records the local checkpoint and the translog rolls to a new generation; it happens when the translog grows past a threshold or ages past flush_threshold_age, or on _flush. Merges follow Lucene’s TieredMergePolicy: the allowed segment count per tier from segments_per_tier, candidate merges of up to max_merge_at_once segments scored by skew, size and reclaimed deletes, too-large segments left alone unless they are mostly deletes; force-merge rewrites to max_num_segments. Search uses BM25 (k1 = 1.2, b = 0.75) with shard-local statistics, in which deleted documents still count until merged.

Crashes. Kill is a process crash: what reached the operating system survives (the whole translog). Power cut keeps only what was fsynced: the last commit and the fsynced part of the translog. On restart the master allocates a primary only to a node holding an in-sync copy on disk (store recovery: open the last commit, replay the translog above its checkpoint); replicas recover from the primary — ops-based (replaying only the missing operations, as retention leases allow) when the node had the shard, file-based (copying segments, then the operations since) otherwise.

Allocation. The master assigns primaries, then replicas once their primary is active, with the deciders that matter here: never two copies of a shard on one node (same_shard), _tier_preference (data_hot, data_warm, data_cold), zone awareness, and at most two incoming recoveries per node. Among allowed nodes it picks the lowest weight of the BalancedShardsAllocator: 0.45 × (shards − average) + 0.55 × (index shards − average); it rebalances (relocating) when that improves the weight by more than 1, and moves shards off a tier they no longer belong to. When a node leaves, its primaries are replaced by promoting an in-sync replica (primary term + 1) and its replicas wait index.unassigned.node_left.delayed_timeout (delayed allocation). Health is red with an unassigned primary, yellow with an unassigned replica, green otherwise.

Cluster coordination (the 7.0+ "Zen2" design, discovery and cluster formation). Master-eligible nodes elect a master by a quorum of the voting configuration, after a pre-vote, in ever-increasing terms. The master publishes every cluster-state change to all nodes, commits it once a quorum of master-eligible nodes has acknowledged and steps down if it cannot. Leader and follower checks detect failures (interval, retry count); a restarted node is a new node to the master. The voting configuration is kept odd-sized and never shrinks below three nodes on its own (as Reconfigurator does). With minMasters set the simulator instead behaves like legacy Zen discovery with discovery.zen.minimum_master_nodes: any that many visible master-eligible nodes may elect a master, and when two masters meet the one with the older cluster state yields.

Search. The coordinating node sends the query to one copy of each shard — chosen by adaptive replica selection (on by default since 7.0, C3: rank = R − 1/µ + q̂³/µ over EWMAs with α = 0.3, as ResponseCollectorService computes it) or round-robin — each shard returns its top from + size ids and scores, the coordinating node merges them and fetches the page’s documents from the shards that hold them. from + size > 10000 is refused (index.max_result_window). search_after and points in time are supported; a point in time keeps its segment readers. Each data node has two search threads; a slow node multiplies its service time.

The master is called the cluster manager; the coordination protocol is the same.

Routing keeps routing_num_shards (there is no 9.4-style modulo routing).

index.replication.type: SEGMENT (segment replication) is available: replicas write the translog but do not index; the primary copies its segments to them after each refresh, so replicas use less CPU but see new documents a little later. Realtime GETs go to the primary (segment replication).

There is no _tier_preference or ILM: nodes are tagged with an attribute and indices move with Index State Management (ISM) policies.

A handful of nodes and shards; documents are a short title field; only match-style queries (BM25 on that field), no aggregations, mappings, analyzers beyond lower-cased words, or scroll.

Timeouts are shorter than Elasticsearch’s so failures play out in seconds (each setting shows the real default). The flush threshold is a number of operations standing in for flush_threshold_size. Merge sizes are counted in documents, and the merge policy is Lucene’s algorithm reduced to those sizes. Network latency is fixed.

Not modelled: the write-load and disk-usage terms of the allocator, disk watermarks, shard request caching, search_idle (described, not simulated), translog retention for recovery (retained history is assumed), primary-replica resync after promotion, cross-cluster features, snapshots, stateless/serverless Elasticsearch, and the 8.x/9.x merge policy for time-series data (LogByteSizeMergePolicy).

The shard-sizing tab is guidance arithmetic, not a benchmark; the fan-out tab assumes independent lognormal shard latencies.