Compiler-scheduled chips

No caches and no DRAM: a static schedule over SRAM-only chips chained into a pipeline, why one token comes back fast, and what holding a model in SRAM costs.

Loading the animation…

Concept

The fifth family removes the last things a chip decides for itself. Groq's first LPU, the GroqChip, has no caches and no DRAM: 230 MB of SRAM read at up to 80 TB/s, wide functional slices for matrix and vector work at 750,000,000,000,000 INT8 operations per second, and 16 chip-to-chip links. Its compiler writes a static schedule: every instruction, every memory read and every transfer between chips happens on a cycle chosen before the program runs. Groq's ISCA 2022 paper describes "statically scheduled, deterministic execution" across many chips, kept in step by exchanging hardware counters over the links, so that even the network's transfers are scheduled and avoid contention.

That buys two things. A step takes exactly as long every time, so latency has no tail. And nothing is spent guessing: there are no cache misses, because there is no cache, and no arbitration, because the compiler already did it.

The price is capacity. A model must fit in SRAM, so it is cut at layer boundaries into stages spread over many chips, and a token runs down the chain. Llama 2 70B in INT8 needs at least 301 GroqChips just to hold its weights and one KV cache; Groq's 2023 demonstration used 576, in nodes of 8. The animation sends one token down that chain. Each stage's eight chips share its layers, so each reads only its slice of the weights: 108 µs of SRAM reads for the whole token. Sharing costs latency: every layer's all-reduce inside a node and every hand-on between nodes is one hop of 722 ns, and those add up to 167 µs. The bound is 274 µs per token, 3,640 tokens per second for one user. With one chip per stage instead, there are fewer hops but each chip reads its share alone: 1.08 ms.

	GroqChip (first-generation LPU)(retired)	NVIDIA Groq 3 LPU (LPX rack)(announced)
FP16, dense	188 TFLOP/s[groqchip-brief]	–
FP8, dense	not disclosed[groqchip-brief]	1.23 PFLOP/s (derived)[groq-home]
INT8, dense	750 TOP/s[groqchip-brief]	not disclosed[lpx-page]
On-chip SRAM	230 MB[groqchip-brief]	500 MB[lpx-page]
SRAM bandwidth	80 TB/s[groqchip-brief]	150 TB/s[lpx-page]
Off-chip memory	none[groqchip-brief]	none[lpx-page]
Chip-to-chip (both directions)	400 GB/s (derived)[groq-isca22]	2.5 TB/s[lpx-page]
Chip-to-chip hop latency	722 ns[groq-isca22]	not disclosed[lpx-page]
Power	215 W[groqchip-brief]	not disclosed[lpx-page]
Process	14 nm[groqchip-brief]	–

What the demonstration reached, and what it cost

Groq's chain served one user at above 300 tokens per second, as reported by The Next Platform from Groq's demonstration: 8.2% of the model's bound. As with the wafer, the bandwidth bound is not what limits a chain of SRAM chips; the published speed implies about 38.2 µs per layer beyond it, which the sources do not break down. A wafer and a dataflow chip pay the same order of time per layer, so chapter 10 models it as one per-layer floor and tests it on each machine in turn. What the model does capture is the trade: 576 chips, each with a 215 W TDP, for one copy of a 70B model whose INT8 weights fit in one H100's HBM. With the pipeline full, one sequence per stage, that is 0.472 J per token at the bound.

The next generation is announced, not shipping. NVIDIA's Groq 3 LPU has 500 MB of SRAM at 150 TB/s, 256 to a rack, so Llama 3.1 70B in FP8 fits in 142 chips of one rack, with a bound of 2,150 tokens per second for one user before any hop latency, which NVIDIA does not state. NVIDIA positions the LPX rack beside Vera Rubin GPUs, with GPUs and LPUs "jointly computing every layer of the AI model for every output token": the GPU's HBM for capacity and the LPU's SRAM for speed. Groq's own page claims "1,000 tokens/sec/user" for an LPX rack; it is quoted, not used.

Part	Model, format	Chips	Step	Bound	Tokens/s per user
GroqChip (first-generation LPU)	llama2-70b, INT8	576	274 µs	latency	3,640
GroqChip (first-generation LPU)	llama2-70b, INT8	304	250 µs	latency	4,000
GroqChip (first-generation LPU)	llama2-70b, INT8	301	1.08 ms	memory	928
NVIDIA Groq 3 LPU (LPX rack)	llama3-70b, FP8	142	464 µs	memory	2,150
NVIDIA H100 SXM	llama2-70b, INT8	1	20.6 ms	memory	48.6

Maths

With $W$ bytes of weights cut into $S$ stages of $g$ chips each, SRAM bandwidth $\beta$ per chip, $L$ layers and hop latency $t_{\text{hop}}$ , one token takes

t_{\text{token}} = \sum_{s=1}^{S} \frac{W / S}{g\,\beta} + (S - 1)\, t_{\text{hop}} + [g > 1] \cdot 2L \, t_{\text{hop}} = \frac{W}{g\,\beta} + \left( S - 1 + 2L\,[g > 1] \right) t_{\text{hop}}.

More chips per stage divide the reading time by $g$ but add $2L$ all-reduce hops; more stages leave the reading time alone and add hops. Throughput is different: with one sequence in every stage, the chain finishes $S$ tokens per $t_{\text{token}}$ .

Code

One token down the chain, cut from src/lib/accel/model.ts:

for (let k = 0; k < st; k++) {
  const start = t;
  t = t + stageWork + stageReduce;
  const done = t;
  if (k < st - 1) t = t + hop;