Chris Collins

Tools · ai

Failures, checkpoints and goodput

Why a cluster of reliable GPUs fails every few hours (system MTBF = per-GPU MTBF ÷ N, with Llama 3's 419 interruptions in 54 days), the Young/Daly checkpoint interval and the goodput it buys, checkpoint size at 12/14/16 bytes per parameter with synchronous and asynchronous stalls, and inference availability — hot, warm and cold standby and the blast radius of a replica layout.

Added

Works with

How to use it

Four tabs:

  • Failures at scale — from a per-GPU MTBF (≈ 50,000 hours for an H100), or from observed interruptions (Llama 3 405B: 419 unexpected in 54 days on 16,384 GPUs), get the cluster’s mean time between failures, failures per day, the chance a job runs clean, and a log–log chart of MTBF against GPU count.

  • Checkpoint interval — Young’s interval τ = √(2δM) for a system MTBF M and checkpoint stall δ, Daly’s refinement, the numerical optimum, and a goodput curve (useful work ÷ wall time) with an optional restart time and your own interval marked.

  • Checkpoint size — parameters × 12, 14 or 16 bytes, the synchronous stall at your storage bandwidth, the asynchronous stall (each GPU’s shard copied to host memory), and whether the background write drains before the next checkpoint. Use as δ carries a stall to the interval tab.

  • Inference availability — lay a pod (72 GPUs in a GB200 NVL72) out as replicas, see which requests one GPU fault takes down, how often that happens, the availability of k-of-n replicas (with a link into the probability tool’s reliability tab), and the failover time of hot, warm and cold standby for a request of a given context length.

Limits

Failures are modelled as independent and exponential (constant rate); real fleets have infant mortality, wear-out and correlated failures (a switch, a power feed, a bad driver) that make outages lumpier. The observed mode blames every interruption on the GPUs. Goodput counts checkpoint stalls, lost work and restarts only — not slow nodes, data stalls or evaluation. The asynchronous stall assumes the state is evenly sharded over every GPU and that the host copy runs at the given PCIe rate. Failover times are lower bounds: detection, scheduling and routing add to them.

Related: series, parallel and k-of-n reliability, training memory and parallelism planner, AI hardware calculator: training compute, latency numbers.

Companion to AI Acceleration: From Transistors to Trillion-Parameter Models: why large training runs fail, the checkpoint write-storm and fault tolerance at inference time.

Sources

Linking to an example

Links can be written by hand. Add the tab after #: mtbf, interval, size or serve. Parameters you leave out take their defaults. Numbers may be written 100000, 1e5 or 100,000. In a URL + means a space, so write 1e%2B5 — or just 1e5; a 1e+5 typed into the address bar also works.

Parameter Meaning Example

mode

Failures tab: fleet (from a per-GPU MTBF) or observed

mode=observed

gpus, mtbf

GPUs, and MTBF per GPU in hours (also used by the inference tab)

gpus=100000&mtbf=50000

fails, days, on

Observed interruptions, over how many days, on how many GPUs

fails=419&days=54&on=16384

job

Job length in days

job=54

M, delta, R, tau

Interval tab: system MTBF (hours), checkpoint stall (s), restart (s), your interval (minutes)

M=3&delta=60

p, bpp

Size tab: parameters (billions), bytes per parameter (12, 14, 16)

p=405&bpp=16

bw, ngpu, d2h, every

Storage GB/s, GPUs sharing the state, GPU-to-host GB/s, interval (minutes)

bw=500&ngpu=16384

pod, rep, conc, need

Inference tab: GPUs in the pod, GPUs per replica, requests per replica, replicas needed

pod=72&rep=8

ctx, kvt, fab, pfr, lag, mttr

Context tokens, KV MB per token, fabric GB/s, prefill tokens/s, failover lag (ms), replica repair (min)

ctx=131072

example

One of the presets below; parameters given with it override the preset’s

example=p16-young

Presets:

example= Shows

p16-llama3

419 interruptions in 54 days on 16,384 GPUs: one every 3.1 h, ≈ 50,700 h per GPU (open)

p16-llama3-all

All 466 interruptions: one every 2.8 h (open)

p16-few-hours

~16,000 GPUs at 50,000 h: a failure every 3 hours (open)

p15-1e4-gpus

104 GPUs: a failure every 5 hours (open)

p15-1e5-gpus

105 GPUs: a failure every 30 minutes (open)

p16-young

M = 3 h, δ = 60 s: checkpoint every ≈ 19 min, goodput ≈ 90% (open)

p16-restart

The same with a 10-minute restart per failure (open)

p16-async

A 2 s asynchronous stall: every ≈ 3.5 min, ≈ 98% goodput (open)

p16-405b-ckpt

405B × 16 B = 6.5 TB; 13 s synchronous vs 16 ms asynchronous (open)

p20-70b-ckpt

70B: 980 GB at 14 B/param, 1.12 TB at 16 B (open)

p20-175b-ckpt

175B: 2.45 TB at 14 B, 2.8 TB at 16 B (open)

p15-nvl72

72 GPUs as nine 8-GPU replicas: blast radius 1/9 (open)

p15-nvl72-tp72

One model over all 72 GPUs: blast radius the whole pod (open)

p15-postures-128k

Hot, warm and cold failover for a 128K-token request (open)

Sharing

Share link copies a URL that opens this tool with your current settings, written as the parameters above. Opening a link replaces your saved settings for this tool (Restore my settings undoes it).

Runs entirely in your browser; nothing you type leaves this page.