Latency numbers every programmer should know

Table — every operation in ns, µs, and ms, a log-scale bar, and its multiple of an L1 cache reference. Copy it as text, Markdown, or CSV.

Squares diagram — the visual from Colin Scott’s page, redrawn: one black square is 1 ns; a hundred of them make one blue square (100 ns); a hundred blue make one green (10 µs); a hundred green make one red (1 ms). Each operation is drawn in the largest unit that keeps it at one square or more. Use Zoom to change the square size.

Human scale — scale everything so one operation (an L1 cache hit by default) takes one second: a main-memory reference becomes minutes, a disk seek months, and a transatlantic round trip years.

Compare — pick any two operations, including the modern extras, to see the ratio.

Back of the envelope — editable versions of Jeff Dean’s classic estimates: a page of 30 thumbnails read serially or in parallel, quicksorting 1 GB, moving data over a link, serial versus batched versus parallel lookups, and scanning a dataset from memory, SSD, or disk — and, for AI systems, the KV-cache hand-off from a prefill to a decode GPU, writing a training checkpoint, decode speed against human reading speed (4–6 tokens a second), and a voice assistant’s half-second round trip.

Modern extras — L3 hits, system calls, context switches, NVMe reads, cloud availability-zone and cross-region round trips, DNS, TLS, GPU kernel launches, and LLM tokens. AI hardware rows (filter ai-fabric): a GPU L2 hit, an HBM load (~500 ns — HBM is wide, not fast), a CXL-attached DIMM (~200 ns), an InfiniBand switch hop (~130 ns), a store into another GPU’s memory over NVLink through an NVSwitch (~0.8 µs), a small all-reduce across an NVSwitch domain, and a Grace CPU reading Hopper HBM over NVLink-C2C (~0.9 µs). Filter memory-ladder for DRAM → CXL → HBM → C2C → NVMe.

Peter Norvig, Teach Yourself Programming in Ten Years — the original table.

Jeff Dean, Designs, Lessons and Advice from Building Large Distributed Systems (LADIS 2009) and Building Software Systems at Google and Lessons Learned (Stanford, 2010), where the numbers and the thumbnail and quicksort estimates come from; collected in Jonas Bonér’s widely shared gist.

Colin Scott, Numbers Every Programmer Should Know By Year (source, MIT licence). This tool reimplements its trend model formula for formula, so its numbers match that page for every year. The model’s newest data points are from 2012–2014; later years are extrapolated from trend lines, and some (memory bandwidth especially) have drifted far from real hardware — compare with the modern extras.

GPU L2 and HBM: Luo et al., Benchmarking and Dissecting the Nvidia Hopper GPU Architecture (2024), Table IV (H800: L2 263 cycles, global memory 479 cycles at 1.755 GHz); Chips and Cheese, Nvidia’s H100: Funny L2, and Tons of Bandwidth (2023) for large-footprint latency.

NVLink and NVSwitch on GB200 NVL72: Every µs Matters: Achieving Near Speed-of-Light Latency in GPU Collectives (2026) — SM ↔ L2 round trip 0.306 µs, remote store 0.792 µs, all-reduce bound 1.404 µs, 2.37 µs achieved against 11.0 µs for an NCCL ring.

InfiniBand: NVIDIA Quantum HDR switch datasheet, 130 ns port to port.

CXL: The Next Platform, Just How Bad Is CXL Memory Latency? (2022); Sun et al., Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices (MICRO 2023).

NVLink-C2C: Werner, Weisgut & Rabl, Towards Memory Disaggregation via NVLink C2C: Benchmarking CPU-Requested GPU Memory Access (HCDS 2025): 800–1,000 ns.

The envelopes: the KV-cache size 2 × layers × KV heads × head dim × tokens × bytes (Llama 3 70B: 80 layers, 8 KV heads of 128); H100 NVLink 900 GB/s total, 450 GB/s each way; InfiniBand NDR 400 Gb/s per port; 16 bytes per parameter of training state from the ZeRO paper.