AI hardware calculator
How much GPU memory an LLM needs and how many accelerators that takes; the roofline model with a batch slider and overlaid chips; training compute, MFU, isoFLOP loss valleys and the EU AI Act line; inference speed, batching, KV hand-off, energy and cost per token; and a sourced table of accelerators from H100 to Groq, Cerebras, Apple silicon and Jetson.
Added · updated 14 October 2026
How to use it
-
LLM memory — pick a model (Llama 3 8B/70B, Llama 3.1 405B, Llama 4 Scout and Maverick, gpt-oss-120b, DeepSeek-V3 and R1, the book’s 70B worked example, and more — or your own numbers) and an accelerator. For serving it adds up the weights (parameters × bits per value, with block scales for MXFP4 and NVFP4; As released uses the checkpoint’s own mix, such as gpt-oss’s MXFP4 experts or DeepSeek’s FP8) and the KV cache (2 × layers × KV heads × head dimension × bytes, per token of context, per sequence — DeepSeek’s multi-head latent attention stores a compressed latent instead, and gpt-oss’s sliding-window layers keep only 128 tokens), shows whether it fits, how many GPUs it needs, and the memory per GPU with tensor parallelism. For training it uses mixed-precision Adam (16 bytes per parameter) plus activations.
-
Roofline — attainable throughput = min(peak, arithmetic intensity × bandwidth), plotted for the accelerator and precision you choose, with a matrix multiply, one attention decode step, or an elementwise operation marked on it. The batch slider grows the matrix multiply’s M from 1 to 8,192 — the dot climbs the slope as batching does — and Overlay other accelerators draws up to five more roofs (H100 against B200, MI355X, Ironwood, the Apple M1 Neural Engine …).
-
Training compute — C ≈ 6·N·D FLOPs on a log scale against GPT-3, Chinchilla, Llama 3.1 405B and the EU AI Act’s 1025 FLOP line; time, GPU-hours and cost at a given MFU and hourly price; MFU from one measured step (tokens per step and step time) or from a run’s reported GPU-hours; isoFLOP loss valleys and the compute-optimal frontier; and, given the tokens a model will serve in its lifetime, the smaller, over-trained model that reaches the same loss for the least training + inference compute.
-
Inference speed & cost — time to first token (prefill, compute-bound), time per output token (decode, usually bandwidth-bound), a chart of tokens/s per sequence and in total against the batch with the largest batch the KV cache allows, the single-stream bandwidth ceiling, the time to hand a prompt’s KV cache from a prefill GPU to a decode GPU over NVLink or a 400 Gb/s NIC, requests per second, dollars per million tokens, and energy per token (watts ÷ tokens/s) and per answer. Measured tokens/s per GPU replaces the estimate for cost, energy and request rate.
-
Accelerators — one table of every chip the tool knows: memory, bandwidth, dense peaks by precision (marked where the vendor’s headline is 2:4 sparse), power, ridge point, the bandwidth-bound decode ceiling for a model, the chips needed just to hold it (the question that decides SRAM machines such as Groq and Cerebras), and a source and as-of date for every row.
Every result is an estimate from first principles and the vendors' peak numbers — real kernels and serving stacks reach a fraction of them. Throughput figures are dense: NVIDIA’s headline tensor-core numbers (and AMD’s, Jetson’s and DGX Spark’s “with sparsity” figures) assume 2:4 sparsity and are twice these. The number formats are explained bit by bit in the floating-point inspector.
Limits
-
Peaks are datasheet ceilings; delivered performance is typically a half to a third of them (MFU). The decode model reads weights and KV cache once per step and ignores kernel launch, communication and scheduling overheads; tensor-parallel figures assume a perfect split.
-
MoE routing is assumed uniform. Llama 4’s chunked-attention layers are treated as full attention (an overestimate of their KV cache).
-
The loss curves use one published fit of the Chinchilla data (Besiroglu et al. 2024); the absolute loss values belong to that dataset and tokenizer, not to any model you name.
-
Energy counts accelerator board power only — not the host, networking or cooling (multiply by the facility’s PUE).
-
Groq LP30 figures come from NVIDIA’s GTC 2026 disclosure as quoted in the book; Cerebras publishes no dense peak, and Apple publishes no GPU TFLOPS.
Related: floating-point inspector (FP8, MXFP4 and NVFP4 bit by bit), latency numbers.
Companion to the roofline model, sizing a 70B model, the MFU worked calculation, GQA, MQA and MLA, accelerators at a glance, scaling laws, cost per token, edge silicon and the EU AI Act’s compute trigger in _AI Acceleration: From Transistors to Trillion-Parameter Models.
Sources
-
Accelerators: each row of the Accelerators tab links its source and the month it was checked — NVIDIA’s H100, H200, HGX B200 and B300 (180 GB and 2.1 TB per 8 GPUs; FP4 144 | 72 and 144 | 108 PF sparse | dense), GB200 NVL72 and GB300 NVL72 pages, the A100, V100 and L40S datasheets and the Ada and Blackwell GeForce whitepapers; AMD’s MI300X, MI325X and MI355X data; Google Cloud’s TPU v6e and TPU7x (Ironwood) pages; AWS Trainium; Intel Gaudi 3; Groq’s LPU papers; Cerebras WSE-3; Apple’s newsroom and tech-spec pages; NVIDIA Jetson Thor, Orin and DGX Spark; Hailo-10H.
-
Model configurations: the models' published
config.jsonfiles and papers — Meta, The Llama 3 Herd of Models (Table 3) and the Llama 4 model cards; Jiang et al., Mistral 7B; Mistral’s Mixtral 8x7B, Qwen2.5, Gemma 2, DeepSeek-V3 and DeepSeek-R1 configurations; OpenAI’s gpt-oss model card. -
Williams, Waterman & Patterson, Roofline: an insightful visual performance model for multicore architectures (2009).
-
Kaplan et al., Scaling Laws for Neural Language Models (2020) — C ≈ 6·N·D.
-
Hoffmann et al., Training Compute-Optimal Large Language Models (2022) — Chinchilla; Besiroglu et al., Chinchilla Scaling: A replication attempt (2024) — the loss fit used for the valleys; Sardana et al., Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (2023).
-
Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models (2019) — 16 bytes per parameter with mixed-precision Adam.
-
Korthikanti et al., Reducing Activation Recomputation in Large Transformer Models (2022) — activation memory.
-
Regulation (EU) 2024/1689 (the AI Act), Article 51 — the 1025 FLOP presumption.
Sharing
Share link copies a URL that opens this tool with your current settings — every value it remembers between visits, packed into ?s=. Opening a link replaces your saved settings for this tool. Share links made with earlier versions of the tool still open as they did.
Linking to an example
Links can be written by hand. Add the tab after #: memory, roofline, training,
inference or chips. The first group of parameters sets the same setting on every tab that
has it (gpu=b200 picks the B200 on all four calculators).
| Parameter | Meaning | Example |
|---|---|---|
|
Model id: |
|
|
A custom model (with |
|
|
Accelerator id, as in the Accelerators table: |
|
|
Weight format: |
|
|
KV-cache format (same ids) |
|
|
Compute precision: |
|
|
Context length in tokens (memory) or attention context (roofline) |
|
|
Concurrent sequences |
|
|
Number of accelerators (training and inference) |
|
|
Dollars per GPU-hour |
|
|
Memory tab: |
|
|
Roofline: operation |
|
|
Roofline overlays: accelerator ids separated by commas |
|
|
Training: tokens in billions (blank = 20 per parameter), MFU %, or a FLOP budget |
|
|
Training: tokens in one measured step and its seconds (gives MFU), reported GPU-hours (gives MFU), lifetime inference tokens (over-training) |
|
|
Inference: prompt and output tokens, prefill efficiency % |
|
|
Inference: measured tokens/s per GPU and power per GPU in watts (blank = the table’s board power) |
|
|
Accelerators tab filter: |
|
|
One of the presets below; parameters given with it override the preset’s |
|
Numbers may be written as 4e6 or 4000000. In a URL + means a space: write 1e24 (or
1e%2B24); a 1e+24 typed straight into the address bar also works, because the number fields
ignore the space it becomes.
Presets (names carry the book’s part number):
example= |
Shows |
|---|---|
|
70B dense weights in FP16: 140 GB, more than one 80 GB H100 holds (open) |
|
The same weights in FP8: 70 GB (open) |
|
In FP4 (0.5 byte each): 35 GB, on one 80 GB GPU (open) |
|
KV cache of one 8,000-token conversation on the 70B model: 320 KiB/token, 2.6 GB (open) |
|
The same at 128,000 tokens: 42 GB for one sequence (open) |
|
The 70B model with 64 KV heads (full multi-head attention) instead of GQA’s 8: 8× the cache, 21 GB at 8,000 tokens (open) |
|
Training the 70B model with mixed-precision Adam: 16 bytes/parameter, 1.12 TB before activations (open) |
|
Decode ceiling: 3.35 TB/s ÷ 140 GB ≈ 24 tok/s per H100 of bandwidth; the 140 GB model is split over 4 H100s (tensor parallel), whose combined bandwidth gives ≈ 96 tok/s (open) |
|
Batching: total tokens/s climbs with the batch while each sequence slows a little, until the KV cache fills memory (open) |
|
Prefill → decode hand-off: an 8,000-token prompt’s 2.6 GB KV cache over NVLink vs a 400 Gb/s NIC (open) |
|
$3 per GPU-hour at 2,000 tokens/s: $0.42 per million tokens (open) |
|
A 72-GPU rack at 3,000 tok/s per GPU: ≈ 720 requests/s on 300-token answers (open) |
|
The same rack on 10,000-token reasoning turns: ≈ 22 requests/s (open) |
|
An 8B model in 4 bits on an M5 (153 GB/s): bandwidth-bound decode; switch to the M4 (120 GB/s) for the 1.27× gap (open) |
|
Hailo-10H: a 7B model at the claimed 10 tok/s under 5 W is 0.5 J per token — and above its ≈17 GB/s bandwidth ceiling (open) |
|
Jetson AGX Thor: an 8B model in NVFP4 at 130 W — tokens/s, joules per token and Wh per answer (open) |
|
H100 roofline: 989.5 TFLOPS ÷ 3.35 TB/s = a ridge at ≈ 295 FLOP/byte; one decoding token (M = 1) sits at ≈ 1 (open) |
|
Batching climbs the roofline: drag the batch slider from M = 1 towards the ridge (open) |
|
H100, B200, MI355X and TPU Ironwood rooflines overlaid (BF16, dense) (open) |
|
The Apple M1 Neural Engine’s roofline (11 TFLOPS FP16 over 68 GB/s), against an H100 (open) |
|
MFU of one step: 70B model, 4M tokens, 512 H100s, 7 s → 47.4% (open) |
|
GPT-3: 175B parameters × 300B tokens × 6 ≈ 3.15 × 10²³ FLOPs (open) |
|
Chinchilla: 70B on 1.4T tokens (20 per parameter), 5.9 × 10²³ FLOPs (open) |
|
Llama 3.1 405B on 15.6T tokens: 3.8 × 10²⁵ FLOPs on 16,384 H100s, over the EU AI Act line (open) |
|
Llama 3.1 405B’s reported 30.84M GPU-hours imply ≈ 35% MFU; $92M at $3 per GPU-hour ($62–123M at $2–4) (open) |
|
Over-training: an 8B model on 15T tokens, and the size that minimises training + 10¹⁴ lifetime inference tokens (open) |
|
EU AI Act: 400B parameters on 4T tokens is 9.6 × 10²⁴ FLOPs, just under the 10²⁵ presumption (open) |
|
Every accelerator, with dense peaks, bandwidth, power, ridge and sources; chips to hold the 70B model in FP8 (open) |
|
SRAM machines: Groq chips (230 MB each) and Cerebras wafers (44 GB) needed to hold the 70B model in FP16 (open) |
|
Apple silicon M1–M5: unified-memory capacity and bandwidth, and decode ceilings (open) |
|
Edge modules (Jetson Thor and Orin, DGX Spark, Hailo-10H) with dense peaks and power (open) |
|
gpt-oss-120b as released (MXFP4 experts): 65 GB of weights, on one 80 GB H100 (open) |
Feedback is a gift. Something to improve, a feature to add, or a bug? Tell me.
Runs entirely in your browser; nothing you type leaves this page.