Choosing

The same decoder model on every family, side by side: decode and prefill bounds, energy, chips and utilisation, where each wins and loses, and what they are converging on.

Loading the animation…

Concept

Every chapter ran the same decoder model on one family. This one puts them side by side: Llama 3.1 70B, one user, a 1,024-token context and a 2,048-token prompt, on each family's representative part, using the fewest chips whose memory holds the model. The animation adds the families one at a time; pick the measure to compare.

Family	Part, format	Chips	Per token	Tokens/s per user	Prefill, 2,048	J per token	Decode utilisation
GPUs	NVIDIA H100 SXM, FP8	1	20.8 ms	48.1	147 ms	14.6	0.34%
Systolic designs	Google TPU7x (Ironwood), FP8	1	9.44 ms	106	62.9 ms	not disclosed	0.33%
Wafer-scale	Cerebras WSE-3 (CS-3), BF16	4	6.63 µs	151,000	2.32 ms	0.179	4.27%
Compiler-scheduled	GroqChip (first-generation LPU), INT8	308	1.09 ms	915	387 ms	0.235	0.06%
Reconfigurable dataflow	SambaNova SN40L RDU, BF16	3	25.8 ms	38.8	152 ms	not disclosed	0.29%
Meshes of small cores	Tenstorrent Blackhole (p150a card), FP8	3	45.4 ms	22	146 ms	40.8	0.16%
FPGAs and edge NPUs	Apple M4 (Neural Engine), INT8	3	194 ms	5.17	2.55 s	not disclosed	0.64%
In-memory and analogue	IBM HERMES project chip (PCM, research), INT8	16,570	42.7 µs*	23,400	315 µs	0.0143	0.31%

In the table, * marks crossbar time only: attention, digital partial sums and chip-to-chip transfers are not counted (chapter 9). Utilisation is the share of the chips' peak rate a batch-1 decode step uses at the bound.

Read the table as bounds, not benchmarks. For the designs that read their weights from DRAM the bounds land within a small factor of what systems achieve: the SN40L paper's 8B figure is 58% of its bandwidth bound. The MLPerf rows compare a whole benchmark run with a compute ceiling that now counts everything a sample costs: the prefill of its prompt (222 tokens on average, measured from the benchmark's own dataset) as well as its 294 output tokens, and attention. Prefill is 43% of those flops, so the ceiling falls from 115,204 to 65,319 tokens per second for eight H100s, and the H100 system's measured 21,806 is 33% of it (against the decode-only ceiling it was 19%). A Blackwell system lands in the same place: four GB200 GPUs with FP4 weights in MLPerf Inference v5.1 reached 51,737 tokens per second, 31% of their NVFP4 ceiling. About a third of peak, two generations apart, on an offline benchmark that batches as hard as it likes: that is the share of a GPU's tensor peak a well-tuned LLM inference stack turns into tokens here. The rest goes to what a peak leaves out: memory-bound phases, kernel boundaries, sampling and scheduling.

Once weights are in SRAM: the per-layer floor

Loading the animation…

Concept

The designs that keep their weights in SRAM miss their bounds by far more, and in a telling way. Subtract each one's bound from its published time per token and divide by its layers: Groq's 576-chip Llama 2 70B demonstration pays 38.2 µs per layer beyond the bound, Cerebras 27.7 µs (70B on four wafers) and 17.3 µs (8B on one), SambaNova's SN40L 12.5 µs. Three very different machines (hundreds of chips on a static schedule, a wafer of a million cores, a reconfigurable dataflow chip) land in the same band, 12.5 µs to 38.2 µs a layer. Once the weights sit next to the arithmetic, reading them takes nanoseconds per layer on a wafer; what sets the speed is the per-layer critical path: synchronising the chips or cores, the collectives that combine their partial results, and the trip across the mesh or chain. The model does not split that time into parts; it adds it as one term, a fixed time $\tau$ per layer.

The test of a fitted term is whether it predicts what it was not fitted on. So each system's $\tau$ is the mean of the other three (leave-one-out), and the prediction is compared with the system's own published speed: the bound alone misses by up to 771 times in time per token; the bound plus the floor is within 1.84 times for all four. Pick a system in the animation and watch where its token's time goes: on the wafer the green work is a sliver of every layer, and almost all of the token is floor. The fit is four points, published by the vendors (Groq's through a press report), with the model's guess at each context length, so read $\tau \approx$ 24 µs as an order of magnitude, not a constant of nature.

Where each family wins and loses

GPUs hold the most model per chip (80 GB of HBM per H100: the whole 70B model in FP8 on one), run anything, and decode for one user at the speed of HBM: 48.1 tokens per second at the bound, using 0.34% of the chip's arithmetic. They win by batching: the same step serves hundreds of users (chapter 2). Their energy per token at batch 1, 14.6 J at the TDP, is mostly an idle chip waiting on memory.

Systolic designs make the same trade with a compiler instead of warps: one TPU7x reads its HBM at 7.38 TB/s, so its bound is 106 tokens per second, and like the GPU it needs a large batch to fill its arrays (chapter 3).

Wafer-scale and compiler-scheduled chips put the weights in SRAM and buy per-user speed with silicon: 4 wafers or 308 GroqChips to hold a model one H100 holds. Their published speeds, 450 and over 300 tokens per second for one user, are the fastest in this site's sources. The LPU's static schedule also makes every token take the same time.

Reconfigurable dataflow keeps HBM but fuses a layer into one pipeline and adds a DDR tier for many resident models: 3 SN40L sockets for the 70B model in BF16, and the closest measured speed to its bound of any family here.

Meshes of small cores trade the fastest memory for an open, uniform design and Ethernet built into the chip: 3 Blackhole cards, each bounded by GDDR6 at 512 GB/s.

Edge NPUs and FPGAs run on a few watts and the device's own memory: the 70B model needs 3 M4-class devices even at 8 bits, while the 8B model runs on one at 15.9 tokens per second at the bound. FPGAs win on fixed, predictable latency and exactly-sized numbers, not on large models.

In-memory compute removes weight movement entirely and pays in capacity and precision: 16,570 research chips for the 70B model. Companies have announced in-memory compute products for inference, both digital and analogue, but none of the sources used publishes the per-operation figures this model needs, so the chapter runs on a research chip; read it as a picture of the approach, not of any product.

Check	Kind	Published	Model	Ratio
Llama 3.1 70B, 16-bit: CS-3 systems needed[cerebras-inference]	capacity	4	4	1
a 20B model, 16-bit: CS-3 systems needed (weights only)[cerebras-inference]	capacity	1	1	1
Llama 2 70B, INT8: GroqChips used vs the fewest that hold it[nextplatform-groq]	capacity	576	301	1.91
8x H100, Llama 2 70B, FP8, MLPerf Offline tokens/s vs the compute ceiling (prefill included)[mlperf-h100]	throughput	21,806	65,320	0.334
4x GB200 (Blackwell), Llama 2 70B, FP4, MLPerf Offline tokens/s vs the compute ceiling (prefill included)[mlperf-gb200]	throughput	51,737	165,000	0.314
576 GroqChips, Llama 2 70B: tokens/s per user vs the bound[nextplatform-groq]	latency	300	3,644	0.0823
4x CS-3, Llama 3.1 70B: tokens/s per user vs the bound[cerebras-inference]	latency	450	150,700	0.00299
1x CS-3, Llama 3.1 8B: tokens/s per user vs the bound[cerebras-inference]	latency	1,800	1,387,000	0.0013
16 SN40L sockets, Llama 3.1 8B, BF16: tokens/s per user vs the bound[sn40l]	latency	1,042	1,791	0.582
16 SN40L sockets, Llama 3.1 70B, BF16, speculative decoding: tokens/s per user vs the one-token-per-step bound[sn40l]	speculative	457	203.3	2.25

What the families are converging on

The designs started far apart and are borrowing from each other.

Fast memory beside capacious memory. The SN40L and SN50 keep three tiers (SRAM, HBM, DDR); NVIDIA's announced Groq 3 LPX racks put SRAM-only LPUs beside its GPUs, "jointly computing every layer of the AI model for every output token" (chapter 5); Cerebras's announced CS-4 raises each wafer's SRAM bandwidth to 43.2 PB/s from 21 PB/s.
Narrower numbers. Every current part here quotes an 8-bit rate or narrower: FP8 on the H100, B200, TPU7x and Trainium2, block FP8 on Blackhole, INT4 on the Hailo-10H. Each halving of the weight width halves the bytes a decode step must read (chapter 8).
The compiler owns the data movement. XLA on the TPU, whole-layer fusion on the RDU, the static schedule on the LPU, per-core data-movement programs on Tenstorrent, scheduled DMA on AMD's AI Engines: more of what a cache and a warp scheduler decide at run time is now decided ahead of time.
The interconnect moves onto the chip. NVLink, the TPU's inter-chip links, NeuronLink and Blackhole's Ethernet ports all put scale-up on the accelerator itself (AI Racks Explained, chapter 4).

Maths

For each family the model takes the fewest chips $n = \lceil (W + K)/C \rceil$ whose memory $C$ holds the weights $W$ and the KV cache $K$ , then bounds one decode step by

t_{\text{token}} \ge \max\!\left(\frac{W_{\text{read}} + K}{g\,\beta}, \frac{F}{g\,P}\right) + h\, t_{\text{hop}},

where $g$ chips share each step (all $n$ for a tensor layout, one stage's group for a pipeline), $\beta$ and $P$ are one chip's memory bandwidth and peak rate, and $h$ hops of latency $t_{\text{hop}}$ are added where the vendor gives the latency. Decode utilisation is $F / (t_{\text{token}}\, n\, P)$ . For the crossbar chip the step is instead $L$ layers of four analogue multiplies, $4L\, t_{\text{MVM}}$ .

Code

The row for a family whose part reads its weights, cut from src/lib/accel/model.ts:

const d = decode(p, mid, fmt, batch, ctx);
const pf = prefill(p, mid, fmt, prompt);
const pk = peak(p, fmt);