Chris Collins

Tools · infra

NoSQL data modelling

Model one application four ways — document (embed or reference, with the subset, bucket, extended-reference, computed and outlier patterns), wide column (one table per query), key-value (keys and hand-made indexes) and graph — and see for each which access patterns are one read, several or scatter-gather, how many copies every write touches and what can go out of step; then a "Which database?" questionnaire with a reasoned shortlist and a CAP/PACELC explainer, and exports as CQL, MongoDB $jsonSchema, Cypher and JSON.

Added

Works with

How to use it

A NoSQL schema starts from the questions the application asks, not from the data. The same application — the same entities, relationships and access patterns — comes out very differently in a document store, a wide-column store, a key-value store and a graph, and each version makes some reads cheap by making some writes expensive. This page takes one application and models it all four ways side by side, says which rule produced every choice, and counts the cost.

How to use it

Application. Pick one of six examples — e-commerce orders, a chat app, IoT telemetry, a social graph, a content management system or a multi-tenant SaaS product. The Application tab lists its entities (fields, typical size, how many), its relationships (cardinality, how far they grow, whether they are read together) and its access patterns with how often each runs. Change a frequency and every model follows; Edit the application as JSON lets you describe your own (the format is in the comment at the top of nosql-modeling-data.js), checked before it is applied. Sharded / partitioned cluster decides whether the document model is sharded, which is what turns some queries into scatter-gather.

Each model tab opens with five numbers: the share of read traffic answered by a single request, the mean number of round trips per read, how many reads are scatter-gather, the mean number of copies written per logical write, and the consistency risk. Below them:

  • Document — every relationship’s embed-or-reference decision with the reasons (cardinality, growth, read together, update frequency, size against the 16 MB document limit) and the pattern it uses; each collection as an example document with its shard key; the reads (as find queries), the writes (which documents each touches and whether it needs a multi-document transaction) and the risks. Export: one db.createCollection with a $jsonSchema validator per collection.

  • Wide column — one table per query: partition key, clustering columns with their order, the columns copied in, static columns, rows and bytes per partition (flagged past Cassandra’s guidance), plus counter, latest-value, rollup and fan-out timeline tables and a source-of-truth table for anything no query reads. Export: CQL.

  • Key-value — the key schema: one key per aggregate, list and set keys, and idx: keys that are secondary indexes kept by hand; each read as the commands it takes. Export: JSON.

  • Graph — nodes and relationships drawn (link entities such as an order line or a follow become relationships with properties, dashed); each read as a Cypher query. Export: uniqueness constraints.

  • Compare — every read pattern under each model, and the totals, in one table. Export: all four models as JSON.

  • Which database? — answer eight questions (the shape of the hot reads, how well the queries are known, consistency, scale, regions, latency, who runs it, memory). Each answer adds or takes away points from eight candidates with the reason shown, optionally with signals from the application itself (a multi-hop traversal favours a graph, a heavy time series a wide-column store). Answers for this example fills in a sensible set for the chosen application. Below the shortlist, CAP and PACELC are explained with a table of where common systems sit.

What it models

  • One-to-one or one-to-few, read together and bounded → embed. One-to-many is embedded too while the worst case stays under 1 MB — this page’s rule of thumb, well inside MongoDB’s 16 MB BSON document limit, because a growing document is rewritten and sent whole.

  • Read together but too large or growing without bound → subset: the newest N in the parent ($push with $slice), all of them in their own collection.

  • Read together, usually small, but a few parents have vastly more → outlier: embed up to a threshold, flag the parent and put the rest in overflow documents.

  • Unbounded time series → bucket: one document per parent per hour or day (MongoDB 5.0+ time series collections do this internally).

  • Many-to-one with fields the hot read needs → extended reference: keep the id and copy those fields; updating them fans out to every copy unless the copy is a deliberate snapshot (an order line’s product name at the time of purchase).

  • Many-to-many, bounded → an array of ids (or extended references). Unbounded or not read together → reference: the child stores the parent’s id.

  • An aggregate read more often than its inputs change → computed: kept in the document and updated on every write.

  • An entity is embedded in at most one parent; a second relationship to it is a reference.

  • Shard keys: the top-level field of each collection’s busiest read; a read without it is scatter-gather.

Wide column (Apache Cassandra, Data Modeling): a table per read pattern; the partition key is the pattern’s equality fields (plus a time bucket under a bucketed parent), clustering columns its sort order and then the entity’s key; included children of a single parent become rows of its partition with the parent’s columns STATIC; other included data is copied in. Aggregates become counter tables (counters cannot share a table with other columns), latest-value or rollup tables. A timeline through a link (posts of the people I follow) becomes fan-out on write. Partitions are flagged past about 100 MB or 100 000 rows, and low-cardinality partition keys are called out. Writes count every table that holds the data.

Key-value (Redis, or DynamoDB without indexes): the document model’s aggregates as values; a read by the key is one GET; a parent’s children are a sorted set of ids, read and then MGET (two round trips); any other lookup needs an idx: key maintained by hand; edges are sets; a time series is a list or stream per bucket. Multi-key writes are atomic only within one node (Redis hash tags) or a transaction (DynamoDB TransactWriteItems).

Graph (Neo4j, data modelling): entities are nodes, relationships edges, link entities edges with properties; every read is one Cypher query, and a traversal’s cost is the number of relationships walked. Huge time series and supernodes are flagged.

Which database? The scores are this page’s judgement, written down so you can disagree with each one. PACELC is from Daniel Abadi, Consistency Tradeoffs in Modern Distributed Database System Design (IEEE Computer, 2012); CAP from Eric Brewer’s PODC 2000 keynote, proved by Gilbert and Lynch (2002).

Limits

  • The models are derived by rules from a short description, not designed by hand: a real schema would also weigh things the description does not hold (which fields change together, privacy, retention, how the application handles stale data). Treat each model as a well-argued first draft.

  • Sizes and counts are illustrative orders of magnitude for a mid-sized service. Fan-out counts (how many copies an update must reach) are estimated as count(referencing) ÷ count(referenced).

  • "One read" means one request to one place; it says nothing about how many bytes come back or how hot the partition is — see row key design and NoSQL sizing for that.

  • MongoDB secondary indexes and Cypher property indexes are assumed to exist for every field a pattern looks up by; Cassandra secondary indexes and materialised views are not used — each query gets its own table, as the Cassandra guidance recommends.

  • The questionnaire ranks eight well-known systems and leaves out many good ones (Couchbase, HBase, Bigtable, FoundationDB, ArangoDB, TiDB…). Classes in the PACELC table describe default configurations.

Linking to an example

Links can be written by hand. Add a tab after #: app (the default), document, wide, kv, graph, compare or choose. A link’s settings replace this page’s saved settings (a button restores yours); an edited application travels packed in the link.

Parameter Meaning Example

example

An example application with its questionnaire answers (list below)

example=chat

app

The same, keeping your questionnaire answers

app=iot

sharded

Shard the document model (true/false)

sharded=false

shape

Hot reads: aggregates, relationships, timeseries, keylookup or mixed

shape=relationships

flex

Queries: known, some or adhoc

flex=adhoc

cons

Consistency: eventual, session, strong or transactions

cons=strong

scale

Writes and data: small, medium or large

scale=large

regions

one, multi-read or multi-write

regions=multi-write

lat

Read latency: ms, low or relaxed

lat=ms

ops

Who runs it: managed, either or self

ops=self

mem

Working data fits in memory or is bigger

mem=fits

useApp

Score the application’s own patterns too (true/false)

useApp=false

example= Shows

ecommerce

Embedded order lines, a snapshot copy of the product name, the subset pattern for reviews and a computed rating (open)

chat

Messages bucketed per channel per day, memberships embedded in users, and a display-name change that reaches 4 000 copies (open)

iot

50 000 readings a second: bucket documents, an hour in the partition key, latest-value and rollup tables (open)

social

The home timeline: fan-out on write in a table store against two hops in a graph (open)

cms

The outlier pattern for comments and a tag table with one row per tag (open)

saas

Tenant-scoped queries, a shard key that makes "my tasks" scatter, and an audit log in day buckets (open)

Runs entirely in your browser; nothing you type leaves this page.