FPGAs and edge NPUs

Loading the animation…

FPGAs: an HBM card, published LLM designs and the flow

The current part. AMD's Alveo V80 is the HBM FPGA card of this generation: a Versal HBM device (XCV80) with 2,600,000 look-up tables, 10,848 DSP slices and 132 Mb of block RAM plus 541 Mb of UltraRAM (about 84.1 MB on chip), beside 32 GB of HBM2e at 820 GB/s, in a card of up to 190 W (AMD's specifications). Pick it in the animation. Llama 3.1 8B fits in its HBM at any width shown; at 4 bits the decode bound is 215 tokens per second for one user, 0.885 J per token at the TDP, and at 3 bits 285. Two things follow. The on-chip memory, large for an FPGA, holds about a hundredth of the 8B model's 8-bit weights, so an LLM on this card streams its weights from HBM like a GPU does, with a quarter of an H100's bandwidth. And AMD gives no TOPS figure, because there is none to give: the arithmetic is whatever the design builds from those DSPs and LUTs, so the model has no compute term for the FPGA, and the bound is the memory's. The earlier Alveo V70, an AI Engine part with DDR, is discontinued (in the table for comparison: 20.1 tokens per second at 4 bits from 76.8 GB/s of DDR).

What published FPGA LLM designs do with that. FlightLLM (Zeng et al., FPGA 2024) maps whole LLMs onto an Alveo U280 with three FPGA-specific ideas: a configurable chain of DSP48 slices that skips the zeros of several sparsity patterns, an "always-on-chip decode" scheme with mixed-precision support aimed at the decode phase's memory bandwidth, and a "length adaptive" compilation method that cuts the cost of compiling for each sequence length. At batch 1 on LLaMA2-7B the authors report 6.0 times the energy efficiency and 1.8 times the cost efficiency of a V100S, and on a Versal VHK158 1.2 times an A100's throughput (their claims, not this site's runs). Chen et al. go further towards the FPGA's native style: a spatial design, one hardware unit per operator with data streamed between them, built from a library of HLS kernels, with an analytical model of when it beats a GPU; for GPT-2 decode on a U280 they report 1.9 times an A100's speed and 5.7 times its energy efficiency. Both results are for batch 1 and small to mid-size models, which is where the FPGA's fixed pipeline helps most: there is no batch to fill and no kernel launch to wait for.

The flow, and why custom widths pay. An FPGA design for a network is usually written in C++ for high-level synthesis (Vitis HLS) or generated from the trained network by a compiler. hls4ml builds networks into FPGAs for particle-physics triggers, where its example classifier answers in about 100 ns; FINN builds a streaming dataflow accelerator, one hardware stage per layer, with each layer's resources sized to a throughput target. Both rely on the FPGA's freedom to pick any number format per layer. That pays twice. Memory: a weight costs exactly its bits, which is the animation. Logic: a multiplier's size grows about as the square of its width, so a narrow multiply built from LUTs costs a fraction of a wide one, and several narrow multiplies can share one DSP slice. The author's synthesis of a systolic array's PEs (Systolic Arrays Explained, the model page) measures the square law in gates. The cost is the one that has kept FPGAs out of large-model serving: a hardware design flow, a long place-and-route run for every change, and clock speeds well below a fixed-function chip's.

const wb = bits / 8;
const kvb = wb < 1 ? 1.0 : wb;
const weights = s.params * wb;
const nbytes = s.matmul_params * wb + ctx * s.kv_elems_per_token * kvb;