Reconfigurable dataflow

SambaNova's RDU and Graphcore's IPU: a layer fused into one on-chip pipeline instead of a kernel per operation, three tiers of memory, and bulk-synchronous tiles.

Loading the animation…

Concept

A GPU runs a model as a sequence of kernels: one for the normalisation, one for the QKV projection, one for the attention scores, and so on. Each kernel reads its input from HBM and writes its result back, because the next kernel may run on any SM at any time. A reconfigurable dataflow chip turns the graph into a circuit instead. Its compiler lays the operations of a whole layer out across the chip, each on its own group of compute and memory units, wired together so that one operation's results stream straight into the next. HBM then sees only what has to come from outside: the weights, the KV cache, and the layer's input and output.

SambaNova's SN40L is the example the model runs. Each socket has 1,040 Pattern Compute Units and as many Pattern Memory Units, 520 MB of on-chip SRAM between them, and 638 TFLOP/s of BF16 compute. The paper describes how the compiler fuses "hundreds of complex operations with arbitrary access patterns into a single kernel call".

The animation runs one decoder layer of Llama 3.1 8B both ways on one socket. In a 2,048-token prefill, the kernel-by-kernel layer moves 1.36 GB to and from HBM: the activations between operations are as big as the weights, and unfused attention writes and re-reads its whole score matrix. Fused, it moves 487 MB, and because the stages overlap, the layer's time is the larger of its total memory time and its total compute time, not the sum of each kernel's: 1.45 ms against 1.75 ms.

Switch the work to a decode step at batch 1 and the difference almost vanishes: 471 MB against 470 MB. One token's activations are tiny next to the weights, which every design must read. What fusion buys in decode is not bytes but overhead: no kernel launches between operations, and weight loads overlapped with compute. The SN40L paper reports that this reaches "over 85% of HBM bandwidth", where GPU implementations "rarely exceed 50%". The model cannot see launch overhead, so it does not credit either claim; the validation below compares the paper's measured speed with the bandwidth bound instead.

Three tiers of memory

The SN40L's second idea is its memory. Besides the SRAM and 64 GB of HBM at 1.8 TB/s, each socket has up to 1.5 TB of DDR at 200 GB/s. HBM holds the models being served; DDR holds many more, ready to swap in. On 16 sockets, HBM holds 63 copies of Llama 3.1 8B in 16 bits and DDR holds 1,494; loading one from DDR into HBM over the sockets' DDR bandwidth takes 5.02 ms. SambaNova built its "composition of experts" serving on this: many small models resident at once, switched per request. A GPU server holds its models in HBM and swaps from host memory over PCIe.

	SambaNova SN40L RDU	SambaNova SN50 RDU(announced)
BF16, dense	638 TFLOP/s[sn40l]	1.6 PFLOP/s[sambanova-rdu]
FP8, dense	not disclosed[sn40l]	3.2 PFLOP/s[sambanova-rdu]
On-chip SRAM	520 MB[sn40l]	432 MB[sambanova-rdu]
Off-chip memory	64 GB[sn40l]	64 GB[sambanova-rdu]
Off-chip bandwidth	1.8 TB/s[sn40l]	not disclosed[sambanova-rdu]
DDR (third tier)	1.5 TB[sn40l]	512 GB[sambanova-rdu]
DDR bandwidth	200 GB/s[sn40l]	–
Cores / SMs / TensorCores	1,040[sn40l]	–
Process	5 nm[sn40l]	–
Scale-up, chips	–	256[sambanova-rdu]

The SN50, SambaNova's fifth generation, keeps the three tiers with 432 MB of SRAM per chip and quotes 1.6 PFLOP/s of BF16; the page does not give its HBM bandwidth or say that it is shipping, so the model does not run it.

What the paper measured

The SN40L paper reports Llama 3.1 8B at 1,042 output tokens per second for one user on 16 sockets, in BF16 at an 8K sequence length: 58% of the model's bandwidth bound, by far the closest any published speed in this site comes to its bound. Its 70B figure, 457 tokens per second, is 2.25 times the one-token-per-step bound: the paper used speculative decoding there, which emits several tokens per read of the weights, so that row is shown but kept out of the validation band.

Check	Kind	Published	Model	Ratio
16 SN40L sockets, Llama 3.1 8B, BF16: tokens/s per user vs the bound[sn40l]	latency	1,042	1,791	0.582
16 SN40L sockets, Llama 3.1 70B, BF16, speculative decoding: tokens/s per user vs the one-token-per-step bound[sn40l]	speculative	457	203.3	2.25

The IPU: bulk-synchronous tiles

Graphcore's IPU took the opposite memory bet. The first-generation GC2 has 1,216 tiles, each a core with 262 kB of its own SRAM, 319 MB in all at 45 TB/s, and nothing else on the device: "the IPU offers no memory other than the distributed memories local to each tile". It runs the bulk-synchronous parallel model: every tile computes on its own memory, then all tiles stop at a barrier, then they exchange data on a schedule the compiler fixed, then compute again. A model must fit in the tiles' SRAM, as on a Groq chain: Llama 3.1 70B in 16 bits needs 444 GC2s. Graphcore's site lists no IPU products on the access date, so the part is marked retired.

	Graphcore GC2 IPU (first generation)(retired)
FP16, dense	125 TFLOP/s[ipu-dissect]
On-chip SRAM	319 MB[ipu-dissect]
SRAM per tile	262 kB[ipu-dissect]
SRAM bandwidth	45 TB/s[ipu-dissect]
Off-chip memory	none[ipu-dissect]
Cores / SMs / TensorCores	1,216[ipu-dissect]
Power	150 W (reported)[ipu-dissect]

Maths

For operation $i$ of a layer, let $B_i$ be the weight and KV-cache bytes it must read, $A_i$ the activation bytes it reads and writes, and $F_i$ its flops. On a chip with memory bandwidth $\beta$ and peak $P$ , kernels run one after another, each bound by its own larger term, so

t_{\text{kernels}} = \sum_i \max\!\left(\frac{B_i + A_i}{\beta}, \frac{F_i}{P}\right) \;\ge\; \max\!\left(\frac{\sum_i B_i}{\beta}, \frac{\sum_i F_i}{P}\right) = t_{\text{fused}},

with $A_i$ gone from the fused pipeline except at the layer's input and output. For a prefill of $T$ tokens with $h$ heads, the unfused attention also writes and reads its score matrix, $2 \cdot T \cdot h \cdot \bar{c}$ activations where $\bar{c}$ is the average number of positions each token attends to ( $T/2$ for a causal prompt).

In the bulk-synchronous model each superstep costs its slowest tile's compute, the barrier, and the exchange:

t_{\text{superstep}} = \max_{\text{tiles}} t_{\text{compute}} + t_{\text{sync}} + t_{\text{exchange}}.

Code

The two ways of counting one operation's traffic, cut from src/lib/accel/model.ts:

let act: number;
if (fused) {
  act = 0.0;
  if (i === 0) act = act + tokens * d * ACT_BYTES;
  if (i === ops.length - 1) act = act + tokens * d * ACT_BYTES;
} else {
  act = (prev + width) * tokens * ACT_BYTES;
}