GPUs: the baseline

HBM, tensor cores and the roofline: why one decode step on a GPU is a memory read, and how batching turns it into a compute problem.

Loading the animation…

Concept

A GPU is the baseline every other family is measured against, so this chapter fixes its numbers and leaves the inside of the GPU to GPU Kernels Explained. An H100 SXM has 132 SMs with tensor cores, 80 GB of HBM read at 3.35 TB/s, a 50 MB L2 cache, and 1.98 PFLOP/s of dense FP8. A B200 has 186 GB at 8 TB/s and 5 PFLOP/s.

Divide the peak rate by the bandwidth and you get the ridge: 591 flops per byte on an H100, 625 on a B200. A step that does fewer flops per byte it reads waits on memory; one that does more waits on the tensor cores.

Decode sits far below the ridge. One step of Llama 3.1 70B in FP8 reads every weight once to do about two flops with it: 2.03 flop per byte at batch 1. On eight H100s that bounds a step at 2.6 ms, or 385 tokens per second for the one user, with the tensor cores almost idle. Batching is how GPUs climb the slope: 64 sequences share each weight read, and the eight GPUs produce 21,400 tokens per second in all, though each user now gets 334. Each sequence also brings its own KV cache to read, so the climb slows; at 1,024-token contexts the step crosses the ridge only near a batch of 1,024, where the bound is 112,000 tokens per second. Memory caps the batch too: eight H100s hold the weights plus 3,394 such caches, and only 424 at 8,192 tokens.

Prefill is the opposite case. A 2,048-token prompt reuses each weight 2,048 times, so it is compute-bound from the start: 18.3 ms on eight H100s.

	NVIDIA H100 SXM	NVIDIA B200 (as in GB200 NVL72)	AMD Instinct MI355X
BF16, dense	990 TFLOP/s (derived)[h100-page]	2.5 PFLOP/s (derived)[gb200-page]	2.5 PFLOP/s[amd-mi355x]
FP8, dense	1.98 PFLOP/s (derived)[h100-page]	5 PFLOP/s[bwu-blog]	5 PFLOP/s[amd-mi355x]
FP4 (NVFP4 or MXFP4), dense	–	10 PFLOP/s[bwu-blog]	10.1 PFLOP/s[amd-mi355x]
Off-chip memory	80 GB[h100-page]	186 GB (derived)[gb200-page]	288 GB[amd-mi355x]
Off-chip bandwidth	3.35 TB/s[h100-page]	8 TB/s (derived)[gb200-page]	8 TB/s[amd-mi355x]
On-chip SRAM	50 MB[hopper-blog]	not disclosed[bwu-blog]	256 MB[amd-mi355x]
Chip-to-chip (both directions)	900 GB/s[h100-page]	1.8 TB/s (derived)[gb200-page]	not disclosed[amd-mi355x]
Power	700 W[h100-page]	1.2 kW[bwu-blog]	1.4 kW[amd-mi355x]

The B200 here is the one in NVIDIA's GB200 NVL72 rack, with the same figures as AI Racks Explained: dense rates are half the GB200 page's sparse figures for each of a superchip's two GPUs, memory is the superchip's 372 GB split two ways (186 GB), and power is Blackwell's maximum board power (TGP), "up to 1,200W" (NVIDIA's Blackwell Ultra blog, Table 2), a GPU figure like the H100's 700 W TDP. The air-cooled HGX B200 is rated lower: half its page's 72 PFLOPS sparse FP8 over eight GPUs is 4.5 PFLOPS dense per GPU (HGX).

AMD's Instinct MI355X is the other GPU line, and on paper it matches the B200 on rate and bandwidth: 2.5 PFLOP/s BF16 and 5 PFLOP/s FP8 dense, 8 TB/s of HBM3E, with a 256 MB last-level cache in front of it (AMD's specifications, launched June 2025). Where it differs is capacity: 288 GB per GPU, against the B200's 186 GB. Llama 3.1 70B in FP8 with a 4,096-token context fits on 1 of either; the difference shows in how much KV cache, so how big a batch, one GPU holds: 324 sequences on an MI355X against 172 on a B200 at that context. Its 1,400 W is a typical board power, so energy per token is not like for like with the B200's maximum. AMD's link figures (seven Infinity Fabric links at 153 GB/s) do not say whether they are per direction, so the model leaves the MI355X's chip-to-chip bandwidth blank.

What every other family is compared with

The GPU's answer to both steps is the same machine: HBM big enough to hold a large model on one to eight chips, and tensor cores fast enough that prefill and big-batch decode run near peak. Its cost is the slope: at small batches, which is what an interactive user gets, a step is a full read of HBM. The chapters that follow are the other families' answers to that slope.

Part	Model, format	Chips	Step	Bound	Tokens/s per user
NVIDIA H100 SXM	llama3-8b, BF16	1	4.64 ms	memory	216
NVIDIA H100 SXM	llama3-70b, FP8	1	20.9 ms	memory	47.7
NVIDIA H100 SXM	llama3-70b, FP8	8	2.62 ms	memory	382
NVIDIA B200 (as in GB200 NVL72)	llama3-70b, FP8	1	8.77 ms	memory	114
NVIDIA B200 (as in GB200 NVL72)	llama3-70b, FP8	8	1.1 ms	memory	912

Maths

With $W$ bytes of weights read per step, $k_{\text{KV}}$ bytes of KV cache per token, $B$ sequences of $T$ tokens, $P_{\text{mm}}$ weights in matrix multiplies, $L$ layers of width $d$ , $n$ GPUs of bandwidth $\beta$ and peak rate $\pi$ :

t_{\text{step}} = \max\!\left( \frac{W + B T k_{\text{KV}}}{n \beta},\; \frac{2 P_{\text{mm}} B + 4 L d\, T B}{n \pi} \right), \qquad I = \frac{2 P_{\text{mm}} B + 4 L d\, T B}{W + B T k_{\text{KV}}}.

The step is compute-bound when the intensity $I$ passes the ridge $\pi / \beta$ . As $B$ grows, $I$ tends to $(2 P_{\text{mm}} + 4 L d T) / (T k_{\text{KV}})$ , which for Llama 3.1 70B at 1,024-token contexts is above the H100's FP8 ridge, and at 8,192 tokens far below it: then decode never becomes compute-bound, however large the batch.

Code

The decode bound, cut from src/lib/accel/model.ts:

const bytesStep = pl.weights_read + pl.kv;
const tMem = bytesStep / (g * pl.bw);
const flops = 2 * s.matmul_params * batch + s.attn_flops_per_pos * ctx * batch;
const pk = peak(pid, fmt);
const tComp = pk.flops === null ? null : flops / (g * pk.flops);