Chris Collins

Tools · infra

Row key design

Build a row key from parts — fields, reversed timestamps, salts, hash prefixes, padding, separators — run a workload against Bigtable, HBase, Cassandra or DynamoDB, and see where the writes land over time (region splits included), which queries become one range scan and which a fan-out or scatter-gather, and what the key costs in bytes.

Added

Works with

How to use it

In a wide-column or key-value store the row key decides two things at once: where each write goes, and which reads are cheap. Get it wrong and one server takes every write while the rest idle, or the query you run most becomes a scan of the whole table. This page lets you try a key on a workload and see both.

How to use it

Store and key. Choose the store: Bigtable and HBase keep rows sorted by key and split the key space into tablets or regions; Cassandra and DynamoDB hash the partition key to place it. Start from a preset, or build the key from parts:

  • Field — a value from the workload, as text, zero-padded decimal, an 8-byte big-endian number, a raw signed long (as Bytes.toBytes(long) writes it) or one with the sign bit flipped.

  • Timestamp — 13-digit milliseconds, 8 bytes, Long.MAX_VALUE − ts (8 bytes or 19 digits, newest first) or an hour bucket.

  • Salt — a hash of a field mod N buckets (the reader can work it out), or a round-robin salt per write (the reader cannot).

  • Hash prefix — the first hex digits of a hash of a field.

  • Constant and a separator between parts.

For Cassandra and DynamoDB, say how many leading parts form the partition key; the rest is the clustering or sort key. The sample keys are shown the way HBase prints them (\xNN for bytes that are not printable), with checks from the Bigtable and HBase guides underneath.

Workload and cluster. Sequential order IDs, a time series of devices × metrics, users drawn from a Zipf distribution (a few users do most of the writing), or tenants with one big customer. Set the number of writes, the time span, and the cluster: nodes or region servers, the HBase split policy and sizes (in rows, standing in for the real 128 MB flush size and 10 GB maximum), pre-splitting, Cassandra nodes and tokens, DynamoDB partitions.

Load. The verdict says what share of the writes the busiest server took against an even share. The large map is drawn like Bigtable’s Key Visualizer: time across, the key space down (first key at the top), brightness for writes, and lines for the region boundaries as they split. A timestamp-first key draws a bright band at the bottom that splitting never fixes — the newest region always takes the writes. Below it, writes per server or node.

Queries. For the workload’s typical reads — one row, one entity over a time range, the latest ten rows, everything in one minute — the page plans the scan from the key (which parts the query fixes, where a range can start, whether a salt makes it a fan-out, whether the partition key is known) and runs it on the generated rows: regions or partitions touched, rows read and rows returned.

Key size, Byte order and What the guides say explain the costs and the encodings: why a raw long puts −1 after 255, why "10" sorts before "2", and why Long.MAX_VALUE − ts puts the newest row first.

What it models

  • Range-partitioned stores. Rows are kept sorted. A region splits at its middle key when it passes the split size: HBase’s SteppingSplitPolicy (the default since 2.0: twice the flush size while the table has one region on that server, then hbase.hregion.max.filesize), IncreasingToUpperBoundRegionSplitPolicy (min(max size, 2 × flush × regions³)) or a constant size. Daughters stay on their parent’s server and a balancer evens out region counts each minute. The Bigtable model also splits a tablet that took a large share of a minute’s writes and places tablets on nodes by load — a simplification of Bigtable’s internal rebalancing, which is not published.

  • Hashed stores. Cassandra places partitions on a ring of nodes with num_tokens tokens each; DynamoDB on partitions owning equal slices of the hash space. The hash is a stand-in (FNV-1a with a murmur finaliser), not Cassandra’s Murmur3 token, so it shows placement and skew, not real tokens.

  • Queries are counted on the generated rows, not timed.

Checked against

The tests check the byte encodings against HBase’s Bytes (toBytes(long), unsigned compareTo, toStringBinary): negative longs sort after positive ones, flipping the sign bit gives numeric order over 500 random values and the extremes, zero padding keeps decimal order, and reverse timestamps sort newest first in both forms. The split-size formulas follow the HBase reference guide. The simulation is checked for contiguous, non-overlapping regions that hold every key, for a monotonic key leaving all writes on the last region, and for random keys spreading. Every query plan is checked against a brute-force filter of the data: the scan ranges contain every matching row and the counts agree.

Sources: Google Cloud, Bigtable schema design best practices, schema design for time series data and Key Visualizer; Apache HBase reference guide, Rowkey design and Region splits; AWS, Designing partition keys and write sharding; Apache Cassandra, data modelling.

Related: DynamoDB single-table modeller and Cassandra lab go much deeper into those two stores; Partitioning and replication animates consistent hashing and vnodes; Time-series database lab; Storage engines.

Privacy

Everything runs in this page; your designs and settings stay in this browser’s local storage. Nothing is loaded from the network.

Runs entirely in your browser; nothing you type leaves this page.