Time-series database lab

Time-series databases get their speed and their size from a few ideas: a series is a name plus a set of tags, samples arrive in time order at a steady rate, and neighbouring values are alike. This page takes those ideas apart on data you paste or generate. Everything runs in the page.

How to use it

Line protocol. Paste lines you would write to InfluxDB, or switch to Prometheus exposition and paste the output of a /metrics endpoint. Each line is checked the way the server checks it: escaping (a backslash escapes commas and spaces in measurements; commas, equals signs and spaces in tag keys, tag values and field keys; quotes and backslashes in string fields), field types (1.5 and 1 are floats, 1i an int64, 1u a uint64, "…" a string, t/true/False… a boolean), ranges, timestamps in the chosen precision, and type conflicts between lines. The table shows each point and its typed fields; Series keys lists the series (measurement plus tags sorted by key) and the TSM keys InfluxDB stores (series key#!~#field). For exposition text it checks names, label escapes, TYPE/HELP order, duplicate series and histogram buckets (increasing le, cumulative counts, a +Inf bucket equal to _count). To query metrics, take them to the PromQL and LogQL playground.

Compression. Pick a generated signal (a constant, a counter with and without a restart, a gauge, temperatures to one decimal place, CPU percentages, integers…), a series from the first tab, or paste timestamp_ms,value lines. The table compares bytes per sample: raw (16), varints, Gorilla as in the paper, Prometheus’s XOR and XOR2 chunks of 120 samples, InfluxDB’s TSM blocks, and the browser’s deflate of the raw and of pre-differenced columns as a stand-in for a general-purpose compressor such as zstd or Snappy. Bit view writes out the first samples of one codec bit by bit — header, timestamp and value fields in different colours, every field labelled (hover, or open Field by field) — with the hex and a table of deltas, deltas of deltas and XORs.

Codec What it does

Codec	What it does
Gorilla (paper)	Pelkonen et al. 2015 §4.1: a 2-hour block with a 64-bit start time, the first delta in 14 bits, then the delta of deltas in `0` / `10`+7 / `110`+9 / `1110`+12 / `1111`+32 bits; each value XORed with the previous: `0` if equal, else `1` and either `0` + the meaningful bits inside the previous window, or `1` + 5 bits of leading zeros + 6 bits of length + the bits. Seconds only.
Prometheus XOR	`tsdb/chunkenc/xor.go`: a 2-byte sample count, t₀ as a varint, v₀ raw, the first delta as a uvarint, then dod buckets of 14/17/20/64 bits (milliseconds) and the same XOR scheme.
Prometheus XOR2	`xor2.go` (Prometheus 3): one joint control code for time and value, so an on-time unchanged sample is a single bit; dod buckets of 13/20/64 bits packed to byte boundaries; stale markers in 3–5 bits.
InfluxDB TSM	`tsdb/engine/tsm1`: timestamps as deltas divided by the largest common power of ten, then run-length or simple-8b; integers as zig-zag deltas in simple-8b or run-length; floats in Gorilla XOR ending with a NaN.
simple-8b	64-bit words: a 4-bit selector and 60 bits of data split 240 or 120 ones, 60×1, 30×2 … 1×60 bits.

Gorilla (paper)

Pelkonen et al. 2015 §4.1: a 2-hour block with a 64-bit start time, the first delta in 14 bits, then the delta of deltas in 0 / 10+7 / 110+9 / 1110+12 / 1111+32 bits; each value XORed with the previous: 0 if equal, else 1 and either 0 + the meaningful bits inside the previous window, or 1 + 5 bits of leading zeros + 6 bits of length + the bits. Seconds only.

Prometheus XOR

tsdb/chunkenc/xor.go: a 2-byte sample count, t₀ as a varint, v₀ raw, the first delta as a uvarint, then dod buckets of 14/17/20/64 bits (milliseconds) and the same XOR scheme.

Prometheus XOR2

xor2.go (Prometheus 3): one joint control code for time and value, so an on-time unchanged sample is a single bit; dod buckets of 13/20/64 bits packed to byte boundaries; stale markers in 3–5 bits.

InfluxDB TSM

tsdb/engine/tsm1: timestamps as deltas divided by the largest common power of ten, then run-length or simple-8b; integers as zig-zag deltas in simple-8b or run-length; floats in Gorilla XOR ending with a NaN.

simple-8b

64-bit words: a 4-bit selector and 60 bits of data split 240 or 120 ones, 60×1, 30×2 … 1×60 bits.

Cardinality. Enter the metrics and the number of distinct values of each tag. The page multiplies them (the worst case, when every combination occurs), flags tags with 10,000 or more values, and estimates memory at a size per series you can change. The usual culprits are user IDs, request IDs, full URLs and email addresses in tags or labels.

Rollups and retention. Downsample a generated series into windows: min, max, average, last, and for counters the increase and per-second rate with restarts handled (a drop means the counter started again from zero). The chart shows why a naive difference of window values goes negative at a restart. Below, add retention tiers — interval, how long to keep it, values per point — and get the storage for your series count.

Storage engines. Drag the slider to move time on a Prometheus server: the head block, a block cut every 2 hours once the head spans 3, compaction into 6 h, 18 h and 54 h blocks, and blocks deleted at the retention. Beside it, the same story for InfluxDB’s TSM engine: WAL and cache, snapshots to level-1 files, compaction levels and shards.

Checked against

The tests encode 33 series with Prometheus’s own chunkenc package (Prometheus v0.315.1 — the 3.x line — NewXORChunk and NewXOR2Chunk) and InfluxDB 1.11.8’s TSM codecs (int.go, timestamp.go, float.go) with jwilder/encoding/simple8b, and require this page’s output to be byte-identical — including stale markers, negative and huge timestamps, every delta bucket edge, and InfluxDB’s quirk of masking leading zeros to five bits before clamping. Every codec also decodes its output back to the same bits. The Gorilla bucket layout is checked field by field against the paper. Line protocol is checked against the escaping examples in InfluxDB’s documentation, and the exposition parser against the example in Prometheus’s. The block planner is a port of Prometheus’s db.go and compact.go (head.compactable, ExponentialBlockRanges, selectDirs, splitByRange).

Sources: T. Pelkonen et al., "Gorilla: A Fast, Scalable, In-Memory Time Series Database", VLDB 2015; Prometheus tsdb/chunkenc and tsdb/docs/format/chunks.md; InfluxData, line protocol reference and In-memory indexing and the TSM storage engine; Prometheus exposition formats and storage; V. N. Anh and A. Moffat, "Index compression using 64-bit words" (simple-8b), 2010.

Limits

XOR2 is shown without start timestamps (the optional ST data) and for up to 127 samples a chunk. Histograms and native histograms are not encoded.
Cardinality multiplies independent tags; real data is usually smaller. Memory per series is a rule of thumb — measure your own server.
The block simulation ingests continuously from time zero with no gaps, out-of-order samples or deletes.

Related: PromQL and LogQL playground, Storage engines: LSM-tree vs B-tree, Postings lab (simple-9/16, PForDelta and other integer codecs), Probabilistic sketches, Row key design (time-series keys in Bigtable and HBase), DuckDB lab.

Privacy

What you paste is parsed and encoded in this page and kept in this browser’s local storage; nothing is uploaded and nothing is loaded from the network. The general-purpose comparison uses the browser’s built-in CompressionStream.