Chris Collins

Tools · ai

LLM serving planner

The arithmetic of serving an LLM: time per token and throughput against batch size (where the KV cache fills HBM and compute starts to bind), static vs continuous batching animated, pod requests per second and cost per million tokens, a per-request latency waterfall, the prefill-to-decode KV hand-off, speculative decoding, reasoning cost, and prefix-cache economics.

Added

How to use it

Eight tabs share one model and GPU (7B, 13B, 70B, 405B or a 671B mixture of experts, on an H100, H200, B200, B300, MI300X or MI355X) with a weight format, a KV-cache format and a tensor-parallel GPU count (auto picks the smallest power of two that leaves half the memory for the KV cache). When the weights do not fit on the GPUs you chose, the tool says so and adds GPUs: the 70B model’s 140 GB of FP16 weights do not fit in one 80 GB H100, so the single-GPU figure of 3.35 TB/s ÷ 140 GB ≈ 24 tokens/s is a ceiling no single H100 can run, and the tool shows it over tensor parallelism.

  • Batch sweep — each decode step reads all the weights once, shared by the batch, plus every sequence’s own KV cache, and computes 2 × active parameters FLOPs per token. Step time is the larger of bytes ÷ bandwidth and FLOPs ÷ peak. Three charts against batch size: time per output token (TPOT) for each request, tokens/s per GPU, and dollars per million tokens. Marked: where the KV cache fills HBM (beyond it the batch does not fit) and where compute starts to bind. A request’s TPOT stays flat or rises as the batch grows; what falls is the cost per token.

  • Scheduler — static and continuous batching animated over the same queue of requests: idle slots are hatched, each tick shows evictions and backfills, and a table counts idle slot-steps, utilisation and steps to finish. Lengths are a list such as 50,2000,50*39.

  • Pod capacity — requests/s = tokens/s/GPU × GPUs ÷ tokens per answer, compared for two answer lengths, with $/M tokens = $/GPU-hour ÷ (tokens/s × 3,600) × 10⁶ and the cost per answer. Below it, the KV-cache ceiling on concurrent sequences for the chosen model and GPU.

  • Latency — gateway → cache lookup → queue → prefill → first token → decode × N → last token, as a share of the wall clock and on a log scale. Toggle a prefix-cache hit (only the uncached tokens are prefilled) and hidden reasoning tokens.

  • KV hand-off — the KV cache a prefill pool sends to a decode pool: bytes = tokens × KV per token, time = bytes ÷ (per-GPU link bandwidth × GPUs sending), for NVLink 5 and 4, InfiniBand / Ethernet at 800 and 400 Gb/s and PCIe Gen5 and Gen4, against the prefill time of the same prompt.

  • Speculative — tokens per target pass (1 − αk+1) ÷ (1 − α), wall-clock speedup ÷ (k·c + 1), a table for α = 0.5…0.9, an animated draft–verify round (the book’s "accept 2 of 4 → 3 tokens", or a random round at your α), the speedup against batch size with the crossover where speculation starts to lose, and a Medusa tree of mK paths.

  • Reasoning cost — C = cdec·(Tthink + Tout) + cpre·Tin per answer, with its breakdown, the decode time and what thinking does to a pod’s requests per second.

  • Prefix cache — cache reads at 10% of the input price and writes at 125%: total cost with and without the cache, break-even, and the prefill time saved.

The GPU and model figures, each with its source, are in the table under the tool.

Limits

Every number is an ideal ceiling from first principles and vendor peak figures: real servers (vLLM, TensorRT-LLM, SGLang) reach a fraction of them — use Achieved % of peak to derate. Tensor parallelism is an ideal split with no all-reduce cost. Prefill uses 2 × active parameters FLOPs per token plus causal attention at the efficiency you set. Mixture-of-experts reads assume uniform routing. The scheduler queues every request at step 0 with prefill already done. The Medusa acceptance assumes independent candidates, an upper bound. Prices are inputs, not quotes.

Related: AI hardware calculator (memory, roofline, training compute), floating-point inspector (FP8, FP4 and block-scaled formats), latency numbers.

Companion to Part XVI of AI Acceleration: From Transistors to Trillion-Parameter Models — the inference serving stack, pod capacity and scale-up, speculative decoding variants, reasoning-time compute and long-context inference.

Sources

Linking to an example

Links can be written by hand. Add the tab after #: batch, scheduler, pod, latency, disagg, spec, reasoning or prefix — for example /tools/llm-serving/?model=70b&gpu=h100&batch=32#batch. Settings you leave out take their defaults.

Parameter Meaning Example

model

7b, 13b, 70b (default), 405b or moe (671B, 37B active)

model=405b

gpu

h100 (default), h200, b200, b300, mi300x, mi355x

gpu=b200

w

Weight format: fp16, fp8, fp4

w=fp8

kv

KV-cache format: fp16, fp8

kv=fp8

tp

Tensor-parallel GPUs, or auto; raised automatically if the weights do not fit

tp=4

ctx

KV-cache tokens per sequence (batch sweep, pod ceiling, speculative)

ctx=8000

batch

Decode batch (concurrent sequences)

batch=32

eff

Achieved % of peak bandwidth and FLOPs

eff=70

price

$ per GPU-hour

price=3

slots, lens

Scheduler: batch slots, and answer lengths in arrival order (len*count repeats)

slots=2&lens=50,2000,50*39

tps, gpus, ans, ans2

Pod: tokens/s per GPU, GPUs, tokens per answer, and the length compared

tps=3000&gpus=72&ans=300&ans2=10000

prompt, out, think

Input, answer and hidden reasoning tokens (latency and reasoning tabs)

prompt=2000&out=300&think=10000

hit, cached

Latency: prefix-cache hit (1 or true) and how many prompt tokens it covers

hit=1&cached=1500

rt

Latency: count the reasoning tokens (1 or true)

rt=1

queue, mfu

Queue wait in ms; prefill efficiency, % of peak

queue=50&mfu=50

seq, link, send

KV hand-off: prompt tokens, link (nvlink5, nvlink4, ib800, ib400, pcie5, pcie4), GPUs sending at once

seq=8000&link=ib400

a, k, c, m

Speculative: acceptance rate α, draft length k, draft cost ratio c, Medusa top-m

a=0.8&k=4&c=0.05

round

book plays the book’s 2-of-4 round when the tab opens

round=book

cdec, cpre

Reasoning: decode and prefill prices, $ per million tokens

cdec=15&cpre=3

pin, pre, suf, n, rd, wr

Prefix cache: input $/M, prefix tokens, new tokens per request, requests, read and write multipliers

pre=100000&n=10&rd=0.1&wr=1.25

example

One of the presets below; parameters given with it override the preset’s

example=p15-pod-capacity

In a URL + means a space, so write a plus sign as %2B (seq=1e%2B6). A number such as 1e+6 typed straight into the address bar also works: the tool reads 1e 6 back as 1e+6. Lists in lens use commas and *, which need no escaping.

Presets:

example= Shows

p04-single-stream

One stream of the 70B model on an H100: 3.35 TB/s ÷ 140 GB ≈ 24 tok/s — but 140 GB does not fit in 80 GB, so it is shown over TP = 2 (open)

p04-fp4-fits

The 70B model in FP4 (35 GB) fits one 80 GB H100 (open)

p15-batch-sweep

TPOT and tokens/s against batch for 70B on 4 × H100: per-request TPOT stays flat then rises; cost per token falls until the KV cache fills HBM (open)

p15-compute-knee

Short contexts with FP8 KV: the batch grows until compute, not memory, binds (open)

p15-static-idle

A 50-token and a 2,000-token request batched statically: one slot idles 1,950 steps; continuous batching backfills it (open)

p15-continuous-mix

A realistic mix of answer lengths on 4 slots: continuous batching finishes far sooner (open)

p15-pod-capacity

72 GPUs × 3,000 tok/s = 216,000 tok/s: 720 req/s at 300 tokens, 21.6 req/s at 10,000 (open)

p15-kv-ceiling

KV-cache capacity caps concurrent sequences: 70B on a 72-GPU B200 rack at 8,000 tokens each (open)

p19-cost-per-token

$3 per GPU-hour at 2,000 tok/s = $0.417 per million tokens (open)

p15-latency

Per-stage latency of one turn on the 70B model: decode dominates (open)

p15-latency-prefix-hit

The same turn with 1,500 of the 2,000 prompt tokens already in the prefix cache (open)

p15-latency-reasoning

10,000 hidden reasoning tokens turn a seconds-long turn into minutes (open)

p15-image-prompt

An image-heavy prompt past 10,000 tokens: time to first token grows with it (open)

p15-kv-handoff

An 8,000-token KV cache (~2.6 GB) handed from prefill to decode: ~3 ms on NVLink, ~52 ms on 400G InfiniBand (open)

p15-kv-handoff-ib

The same hand-off across racks on a 400 Gb/s NIC (open)

p15-kv-128k

A 128K-token context: ~42 GB (40 GiB) of KV cache to move (open)

p15-kv-1m

A 1M-token prompt: ~330 GB (305 GiB) of KV cache — more than one GPU holds (open)

p15-spec-round

One round: accept 2 of 4 draft tokens → 3 real tokens for one target pass (open)

p15-spec-alpha

k = 4: α 0.5 / 0.8 / 0.9 → 1.94 / 3.36 / 4.10 tokens per pass (open)

p15-spec-medusa

Medusa: 4 heads, top-3 each → 3⁴ = 81 candidate paths verified in one pass (open)

p15-spec-batch

Batch pressure: at large batch the verify pass is no longer free and speculation loses (open)

p15-spec-moe

MoE: draft tokens route to different experts, so the verify pass reads more weights (open)

p15-reasoning-cost

Cost per answer = decode price × (thinking + answer tokens) + prefill price × input tokens, with 10,000 thinking tokens (open)

p15-prefix-cache

A 100K-token cached document: reads at 10%, the first write at 125% (open)

Runs entirely in your browser; nothing you type leaves this page.