Wafer-scale

A whole wafer as one chip: the weights in on-chip SRAM, models split across wafers at layer boundaries, and how defects are survived rather than avoided.

Loading the animation…

Concept

A wafer-scale engine skips the step every other chip takes: cutting the wafer into dies. Cerebras's WSE-3 is the largest square that fits on a 300 mm wafer, 46,225 mm² of silicon with 4,000,000,000,000 transistors, 900,000 cores and 44 GB of SRAM, all on one piece. Cerebras builds it from 84 ordinary dies and keeps them joined: its fabric crosses the scribe lines between them, so the 2-D mesh of cores runs edge to edge.

Why nobody else does this is the animation's subject. Defects land on a wafer at random, a density $D_0$ per unit area. A reticle-sized die of 814 mm² at the illustrative 0.1 per cm² expects 0.814 defects and works only 44% of the time; a whole wafer expects 46.2, and would essentially never be perfect. Cerebras's answer is to make the defects small and survivable: each core is about 38,000 µm², so a defect kills one core out of 900,000, and the design has "redundant compute cores, redundant routing, and a fail-in-place architecture that shuts flaws down and routes around them". With even an illustrative 1.5% of cores held back, there are 292 spares for every defect expected.

Weights in SRAM, split across wafers

The prize is memory bandwidth. Each core has 48 kB of SRAM next to its datapath, and together they are read at 21 PB/s, against an H100's 3.35 TB/s of HBM. The cost is capacity: 44 GB per wafer. Llama 3.1 8B in 16 bits fits on one wafer; the 70B model, at 141 GB with its KV cache, needs 4. Cerebras's inference service does exactly that: "we split them at layer boundaries and map them to multiple CS-3 systems", and "70B models fit on as few as four systems". The model's count matches.

For training Cerebras uses the other answer, weight streaming: the weights stay in an external MemoryX unit and stream onto the wafer one layer at a time, each weight applied to a whole batch of activations held on the wafer. That suits training's huge batches, and not one user's decode: streaming 70B 16-bit weights once over a CS-3's 1.2 Tb/s of I/O takes 927 ms, about one token a second.

	Cerebras WSE-3 (CS-3)	Cerebras WSE-3 Turbo (CS-4)(announced)
FP16, sparse (vendor figure)	125 PFLOP/s[cs3-datasheet]	250 PFLOP/s[cs4-datasheet]
BF16, dense	not disclosed[cerebras-wp]	not disclosed[cs4-datasheet]
On-chip SRAM	44 GB[cs3-datasheet]	44 GB[cs4-datasheet]
SRAM bandwidth	21 PB/s[cs3-datasheet]	43.2 PB/s[cs4-datasheet]
Off-chip memory	none[cerebras-wp]	none[cs4-datasheet]
System I/O	1.2 Tb/s[cs3-datasheet]	2.4 Tb/s[cs4-datasheet]
Chip-to-chip hop latency	not disclosed[cs4-page]	2 µs[cs4-datasheet]
Cores / SMs / TensorCores	900,000[cs3-datasheet]	900,000[cs4-datasheet]
Silicon area	46,225 mm²[cs3-datasheet]	46,225 mm²[cs4-datasheet]
Defect tolerance	spare cores and routes; defects shut down and routed round[cerebras-chip]	–
Power	27 kW[cs3-datasheet]	not disclosed[cs4-datasheet]

The CS-4, announced with three WSE-3 Turbo wafers per system and first shipments "this quarter" on the access date, doubles the bandwidth to 43.2 PB/s per wafer and gives a wafer-to-wafer latency of 2 µs. Cerebras gives its compute only as a sparse FP16 figure. Where the model needs a compute rate for a wafer it uses that figure and marks the result "sparse", so a compute term from it is a floor: the WSE-3's dense rate is its 900,000 cores times eight FP16 lanes each times a clock Cerebras does not state.

What the bound says, and what it leaves out

The model's bound for decode on SRAM is tiny: reading 70B 16-bit weights at 21 PB/s takes 6.63 µs, a ceiling of 151,000 tokens per second. Cerebras published 450, which is 0.30% of it. The gap is the point: once the weights are in SRAM, bandwidth stops being the limit. What is left is most plausibly latency: every layer waits for the one before it, and its partial sums travel across a mesh of hundreds of thousands of cores. The published speed implies about 27.7 µs per layer beyond the bandwidth bound (17.3 µs for the 8B model), which the model does not try to explain: the clock, the mesh dimensions and the per-layer schedule are not disclosed. The same tens-of-microseconds band shows up on Groq's chips and SambaNova's, and chapter 10 turns it into a per-layer floor fitted on the other machines, which predicts the wafer's speed to within a small factor where the bound misses by hundreds.

Check	Kind	Published	Model	Ratio
Llama 3.1 70B, 16-bit: CS-3 systems needed[cerebras-inference]	capacity	4	4	1
a 20B model, 16-bit: CS-3 systems needed (weights only)[cerebras-inference]	capacity	1	1	1
Llama 2 70B, INT8: GroqChips used vs the fewest that hold it[nextplatform-groq]	capacity	576	301	1.91
576 GroqChips, Llama 2 70B: tokens/s per user vs the bound[nextplatform-groq]	latency	300	3,644	0.0823
4x CS-3, Llama 3.1 70B: tokens/s per user vs the bound[cerebras-inference]	latency	450	150,700	0.00299
1x CS-3, Llama 3.1 8B: tokens/s per user vs the bound[cerebras-inference]	latency	1,800	1,387,000	0.0013
16 SN40L sockets, Llama 3.1 8B, BF16: tokens/s per user vs the bound[sn40l]	latency	1,042	1,791	0.582

Maths

With defects scattered at random at density $D_0$ , the number on an area $A$ is Poisson with mean $\lambda = D_0 A$ , and the chance of none is $e^{-\lambda}$ :

Y_{\text{die}} = e^{-D_0 A_{\text{die}}}, \qquad Y_{\text{wafer, perfect}} = e^{-D_0 A_{\text{wafer}}}, \qquad \text{cores lost} \approx \lambda .

A wafer with $s$ spare cores survives as long as $\lambda \ll s$ ; at the values above, $s / \lambda$ is 292. The decode bound across $w$ wafers in a pipeline is the weights over one wafer's bandwidth (each wafer reads its share in turn), plus $w - 1$ hops.

Code

The yield comparison, cut from src/lib/accel/model.ts:

const lamWafer = (d0PerCm2 * area) / 100;
const lamDie = (d0PerCm2 * dieMm2) / 100;
const spares = Math.floor(cores * spareFrac);