Collectives and cluster topology

⛶ Full screen (or F) gives the tool the whole screen — the browser’s full screen where it is allowed, otherwise a full-window layer (iPhone). The ring or tree in the step view and the NVL72 wiring diagram grow to fill it and follow the window; on a wide screen the NVL72 figures sit in a column beside the wiring, and ◫ Panels hides them. Escape leaves.

Ring & tree, step by step — pick 2 to 16 GPUs and step through an all-reduce. In the ring, every GPU’s tensor is cut into n chunks (a, b, c, …); the grid shows which GPUs' contributions each chunk already holds (a0+a1+a2), shading darker as it fills and outlined green when it is fully reduced. Reduce-scatter takes n − 1 steps, all-gather another n − 1, and every ring link is busy on every step. Chunk indexing runs either the book’s Python exactly as printed (rank r ends reduce-scatter holding chunk (r + 1) mod n) or the shifted version where rank r owns chunk r. The tree reduces up a binomial tree in ⌈log₂ n⌉ steps and broadcasts back down in as many again, with most links idle.

Time model — the α–β model: each step costs a fixed α (launch, synchronisation, first byte) plus bytes ÷ β (one link’s bandwidth, one way). Ring: 2(n − 1)·α + 2(n − 1)/n · S/β; tree: 2⌈log₂ n⌉ · (α + S/β). The plot shows both against message size with the crossover marked; the tree wins small messages, the ring large ones. Tick SHARP to add in-network reduction: each GPU sends its tensor once and the switch returns the sum, 2α + S/β, about half the ring’s bytes. Bus bandwidth is NCCL’s convention (algorithm bandwidth × 2(n − 1)/n).

Scale-up vs scale-out — a hierarchical all-reduce over several NVLink domains: reduce-scatter inside each rack, an all-reduce of each 1/n shard across racks on its rail, then all-gather inside the rack, against a flat ring that runs at the network’s speed. Below it, the per-GPU bandwidth ladder from HBM to a 400 Gb/s NIC shows the cliff.

NVL72 wiring — the GB200 NVL72 rack drawn to its real counts: 18 compute trays (4 Blackwell GPUs and 2 Grace CPUs each) and 9 switch trays (2 NVSwitch chips each). Click a GPU to see its 18 links, one to every switch chip, or a switch chip to see its 72 ports, one to every GPU.

Fat tree & rails — switch counts for a given number of hosts and switch radix k, two tiers while they fit (k²/2 hosts) and three beyond (k³/4), with oversubscription, bisection, and the optical transceivers the switch-to-switch links need. The rail-optimised view wires GPU r of every node to rail r’s leaf.

On-chip networks — a ring, 2D mesh, 2D torus, crossbar or bus of rows × columns tiles: diameter, average hops, bisection in links and GB/s, and a hop-count map from any tile (click one). The plot shows bisection per tile shrinking as √N/N for a mesh as the chip grows.

Related: links, optics and energy per bit (bandwidth and pJ/bit for every link named here), AI hardware calculator, graph algorithms (BFS hop counts), latency numbers.

Companion to Network topologies for multi-GPU clusters, the switch silicon, rail-optimized fat-tree topology, ring vs. tree all-reduce, the collective fabric and on-chip networks of _AI Acceleration: From Transistors to Trillion-Parameter Models.

Every setting can be written into a link by hand. Put the tab after : ring, time, cliff, nvl72, fattree or noc; with example= and no , the example’s own tab opens. Sizes take SI units (25MB = 25 × 10⁶ bytes, 1GiB = 2³⁰); 1e9 works too — in a URL write ` as `%2B` (`1e%2B9`), although a ` typed into the address bar (read back as a space) is understood.

Presets:

Share link copies a URL that opens this tool with your current settings. Opening a link replaces your saved settings for this tool; Restore my settings undoes it.