Meshes of small cores

Tenstorrent's Blackhole: many small cores with their own SRAM on a network-on-chip, GDDR6 beside them, and Ethernet built into the chip to join cards.

Loading the animation…

Concept

The seventh family builds a chip out of many small, ordinary processors. Tenstorrent's Blackhole has 120 Tensix cores on a network-on-chip, each a compute unit with its own SRAM and a few small RISC-V cores whose job is to move data: one fetches, others compute, another sends results on. Tenstorrent's page describes the cores as including "a compute unit, network-on-chip, local cache and 'baby RISC-V' cores, coalescing in powerful data movement through the chip". The chip also carries 16 "big RISC-V" cores that can run Linux. Data movement is programmed explicitly, core by core: a mixed schedule, neither a GPU's run-time warps nor a static whole-graph schedule.

Two things set it apart from the families so far. Its memory is modest: 180 MB of SRAM spread over the cores, and 32 GB of GDDR6 at 512 GB/s, the memory of a graphics card rather than HBM. And its scale-out is built into the chip: the p150a card has four 800G Ethernet ports, 800 GB/s in both directions, so cards connect directly to each other without a separate network card or switch chip.

The animation runs one decoder layer of Llama 3.1 70B in FP8, split over 3 cards by tensor parallelism, the fewest whose GDDR6 holds it. On each matrix step every card streams its third of the weights from GDDR6, spread over its cores: the layer's matrices come to 2.38 MB per core, more than a core's 1.5 MB of SRAM, so a layer cannot stay resident and the cores refill on every token. After the attention output and the MLP down projection, the cards add their partial results with an all-reduce over Ethernet. At batch 1 that moves a few kilobytes, so its bandwidth time is 109 ns of the layer's 559 µs. What it really costs is latency, which Tenstorrent does not state, so the model leaves it out.

	Tenstorrent Blackhole (p150a card)
FP8, dense	664 TFLOP/s[tt-hardware]
On-chip SRAM	180 MB[tt-hardware]
Off-chip memory	32 GB[tt-hardware]
Off-chip bandwidth	512 GB/s[tt-hardware]
Chip-to-chip (both directions)	800 GB/s (derived)[tt-hardware]
Chip-to-chip hop latency	not disclosed[tt-hardware]
Cores / SMs / TensorCores	120[tt-hardware]
'Big RISC-V' cores	16[tt-hardware]
Clock	1,350 MHz[tt-hardware]
Power	300 W[tt-hardware]

Where a mesh wins and loses

Per card the bound is set by GDDR6: one decode step of Llama 3.1 8B in FP8 on one card is 14.8 ms, 67.6 tokens per second for one user, against 442 on an H100, whose HBM is 3.35 TB/s. Splitting over more cards divides that time almost exactly, because each card reads less and the Ethernet carries only activations: with eight cards, the 70B layer takes 210 µs. The board's power is 300 W, so at the bound one token of the 8B model costs 4.44 J on one card at batch 1.

The design's case is not one fast chip but an open, uniform one: the same core everywhere, an open-source software stack ("granting developers full access to the metal", Tenstorrent says), and Ethernet as the only interconnect from one card to a whole cluster. Tenstorrent publishes no Blackhole LLM speeds in the sources used, so this chapter has no validation row: the numbers are bounds.

Part	Model, format	Chips	Step	Bound	Tokens/s per user
Tenstorrent Blackhole (p150a card)	llama3-8b, FP8	1	14.8 ms	memory	67.6
Tenstorrent Blackhole (p150a card)	llama3-70b, FP8	3	45.4 ms	memory	22
Tenstorrent Blackhole (p150a card)	llama3-70b, FP8	8	17 ms	memory	58.8
NVIDIA H100 SXM	llama3-8b, FP8	1	2.26 ms	memory	442

Maths

With $c$ cards, each matrix $W_j$ and its flops $F_j$ divide by $c$ . After the attention output and the MLP down projection, each card holds a partial sum of $b \cdot d$ activations of $a$ bytes; a ring all-reduce sends each card $2(c-1)/c$ of that, so

t_{\text{layer}} = \sum_j \max\!\left(\frac{W_j / c}{\beta_{\text{DRAM}}}, \frac{F_j / c}{P}\right) + t_{\text{attention}} + 2 \cdot \frac{2(c-1)}{c} \cdot \frac{b\,d\,a}{\beta_{\text{eth}}} + 2\,(\text{latency}).

The bandwidth part of the all-reduce grows with the batch $b$ ; the latency part, left out here, grows with the number of hops.

Code

One weight matrix's share on one card, cut from src/lib/accel/model.ts:

const wc = (x.k * x.n * b) / cards;
const fl = (2 * x.k * x.n * batch) / cards;
const tm = wc / bw;
const tc = fl / pk;