In-memory and analogue compute

Weights that never move: a decoder layer on phase-change crossbars, and what it costs in capacity, ADCs and precision.

Loading the animation…

Concept

Systolic Arrays Explained, chapter 8 describes the principle: store each weight as a conductance in a crossbar, apply the inputs as voltages, and the currents summing down each column are the dot products, "the ultimate form of a weight stationary dataflow". This chapter puts a real chip's numbers on it and runs the decoder layer through it.

IBM's HERMES project chip is a research chip, not a product: 64 cores in a 14 nm process, each a 256 × 256 crossbar of phase-change memory, 4 devices per weight, with an ADC on every output line. One analogue matrix-vector multiply takes 133 ns in its fast read mode and 520 ns in its precise one, and the paper measures 63.1 TOP/s at 9.76 TOPS/W with 8-bit inputs and outputs.

The animation programs Llama 3.1 8B's weights into crossbars once, then runs one layer for one token. The layer's four weight matrices cut into 3,328 tiles: 52 chips' worth for a single layer, and 1,789 chips for the whole model. Each matrix stage is one analogue multiply on all its tiles at once, so the crossbars spend just 532 ns on a layer and 17.2 µs on a whole token, and 0.00154 J. No weight moves.

That is the promise. The animation also shows the three problems.

Capacity. A weight occupies its crossbar cells for good, so the model needs one cell group per weight: 16,570 chips for Llama 3.1 70B, where one H100 holds its 8-bit weights. Every family so far reuses a few bytes of fast memory for many weights; this one cannot.
Attention. The KV cache changes every token, so attention cannot be precomputed into conductances. It runs in digital logic beside the crossbars, as on any other chip, and the model does not time it.
Precision. An exact sum of 256 products of 8-bit inputs and 8-bit weights needs 24 bits; the chip's path is 8-bit in and out, and below that sit device noise and drift (a phase-change cell's conductance decays over time as $G(t) = G(t_0)(t/t_0)^{-\nu}$ , which the chip corrects with a global rescaling). The 4-phase read mode buys accuracy with 2.08 µs per layer instead of 532 ns.

	IBM HERMES project chip (PCM, research)(research)
Crossbar cores	64[pcm-imc]
Crossbar rows	256[pcm-imc]
Crossbar columns	256[pcm-imc]
Weights on chip	4,194,304[pcm-imc]
PCM devices per weight	4[pcm-imc]
MVM latency, 1-phase	133 ns[pcm-imc]
MVM latency, 4-phase	520 ns[pcm-imc]
Peak MVM rate, 1-phase	63.1 TOP/s[pcm-imc]
Efficiency, 1-phase	9.76 TOPS/W[pcm-imc]
Peak MVM rate, 4-phase	16.1 TOP/s[pcm-imc]
Efficiency, 4-phase	2.48 TOPS/W[pcm-imc]
Input/output bits	8[pcm-imc]
Die area	144 mm² (derived)[pcm-imc]
Process	14 nm[pcm-imc]

What is left out, and what it means

The crossbar time is a floor on a floor. Each layer's 4,096-row inputs span 16 crossbars, whose partial sums must be added digitally, 808,960 additions per token per layer for the 8B model; activations must cross between 1,789 chips; and the paper demonstrates image and character-level networks, not language models of this size. What the chapter shows is the shape of the trade: in-memory compute removes weight movement, the cost that dominates every other family's decode, and pays for it in capacity and precision.

Computing with light is the other analogue approach, with the same pattern of fast, low-energy linear algebra and costly conversions at the edges. It is covered separately, including where it does and does not pay: the Compute in Transit results site and the Fourier Optics Inference decks.

Maths

A crossbar with conductances $G_{ij}$ and row voltages $V_i$ gives column currents

I_j = \sum_{i=1}^{N} G_{ij}\, V_i ,

a dot product per column in one step. With $b_x$ -bit inputs and $b_w$ -bit weights, each product has $b_x + b_w$ bits and the sum of $N$ of them needs $b_x + b_w + \lceil \log_2 N \rceil$ bits to be exact: $8 + 8 + 8 = 24$ for $N = 256$ . An ADC of $b$ bits keeps the top bits only, and its quantisation noise alone limits the signal-to-noise ratio to about $6.02\,b + 1.76$ dB, before device noise.

A matrix of $k \times n$ weights on $r \times r$ crossbars takes $\lceil k/r \rceil \lceil n/r \rceil$ tiles, and $(\lceil k/r \rceil - 1)\, n$ digital additions per input vector to combine the partial sums.

Code

Cutting one layer's matrices into crossbar tiles, cut from src/lib/accel/model.ts:

const tk = Math.ceil(g.k / rows);
const tn = Math.ceil(g.n / cols);
const tiles = tk * tn;
layerTiles += tiles;
psums += (tk - 1) * g.n;