Overview: Breaking the Impossible Quadrangle
Edge deployment of large language models has historically forced engineers to choose three of four constraints: cost, model size, performance, and power. Cloud A100 rentals exceed $1,000 per month; Jetson AGX Orin dev kits exceed $1,500. Running 30B+ models at usable speeds on a low-power, low-cost device has been considered impractical. The Hummingbird+ project from the Institute of Automation, Chinese Academy of Sciences, breaks this constraint by delivering a 30B MoE model at 18 tok/s decoding on a custom $150 board.
Why FPGA
FPGAs sit in an awkward middle ground for AI acceleration: weaker peak compute than GPUs (an H100 delivers 989 TFLOPS FP16 vs. 8–20 TFLOPS for high-end FPGAs like Alveo U280), lower efficiency-per-watt than ASICs at scale, and steep programming complexity.
The key insight is that LLM inference, especially decoding, is memory-bound rather than compute-bound. The dominant operation is generalized matrix-vector multiplication (GEMV): a single token embedding multiplied against a huge weight matrix. The bottleneck is the speed of loading weights from memory, not the number of multipliers. FPGAs are well suited here:
- Flexible memory interfaces tuned for width, frequency, and channel count.
- Low-latency on-chip storage (BRAM/URAM) for activations.
- Dataflow architecture without GPU warp-scheduling overhead.
- W4 mode (4-bit weight, 12-bit activation): Two weights and the activation share a single DSP, producing two weight×activation products per multiply. The +1 correction for signed packing is absorbed into the post-adder via a constant rounding factor on the W multiplexer, eliminating extra LUT logic.
- KV8 mode (8-bit key/value): A port is gated to zero via INMODE[1]; the D port carries the full 8 bits. Five 2-to-1 MUXes handle precision switching with low LUT cost.
- 30B MoE model on a $150 FPGA board: 18 tok/s decode, 50 tok/s prefill, ~10W TDP
- Token Processor couples a 140-DSP GEMV engine with a 7-DSP scalar engine that time-multiplexes all non-linear ops
- W4A12 and KV8A12 packing exploits DSP48E2's four-input structure, absorbing the +1 signed correction into the post-adder
- 24GB DDR4 on PL side delivers 34GB/s bandwidth with near-100% utilization, beating Jetson AGX Orin on tokens-per-dollar (5.2×) and energy efficiency (1.6×)
- First embedded FPGA to deploy a 30B MoE model with both prefill and decode acceleration
- Paper: ACM/SIGDA FPGA 2026, https://dl.acm.org/doi/10.1145/3748173.3779189
- Prior work: Hummingbird (arXiv:2507.03308)
- Qwen3 model: https://modelscope.cn/models/Qwen/Qwen3-30B-A3B
- Conference: https://www.isfpga.org/program/
MoE: A Natural Ally for FPGA
Qwen3-30B-A3B has 30.5B total parameters but activates only 3.3B per token (about 90% sparsity), with 128 experts and Top-8 routing. Memory traffic resembles a small dense model, fitting comfortably on low-cost DDR hardware.
Key specs after GPTQ INT4 quantization: 14.52GB weights plus 0.77GB KV cache for 16K context, totaling roughly 15.29GB.
Hardware Platform: What $150 Buys
The custom PCB centers on a Zynq UltraScale+ MPSoC (XCZU2CG or XCZU3EG), with the programmable logic (PL) connected to 16GB of DDR4 via a SODIMM, plus 8GB of DDR4 on the PS side, totaling 24GB. A 128GB NVMe SSD stores weights, with PCIe 2.0, Gigabit Ethernet, and USB 3.0 also exposed.
Bill of materials:
| Component | Estimated cost | Share | |---|---|---| | XCZU2CG/3EG FPGA | ~$45 | 30% | | 24GB DDR4 (8 + 16) | ~$75 | 50% | | PCB + power + misc | ~$30 | 20% | | Total | ~$150 | 100% |
Memory dominates the cost. PL-side dual-channel DDR4 provides 34GB/s peak bandwidth — only 17% of Jetson AGX Orin's 204GB/s — but Hummingbird+ achieves near-100% bandwidth utilization versus roughly 60–70% on Jetson, partially closing the gap. Total TDP is around 10W versus 60W on Jetson.
Token Processor Architecture
Traditional FPGA LLM accelerators use coarse three-module designs (matrix engine, non-linear unit, memory manager), which suffer from shape mismatch, insufficient decode latency, and wasted resources on over-provisioned non-linear units. Hummingbird+ introduces a compact Token Processor combining a GEMV engine (140 DSPs, 272 GOPs) with a scalar engine (7 DSPs) and a BRAM activation buffer.
Multiple Token Processors run in parallel. In the prefill phase, all processors compute independent tokens simultaneously. In the decode phase, a master processor runs the full pipeline while slaves分担 GQA workload.
GEMV Engine: 140 DSPs of Art
The engine exploits Xilinx DSP48E2's four-input structure (A + D) × B + C to pack multiple low-precision operands:
A chain-tree hybrid reduction delivers 128-way parallelism: 16 DSP chains of 8 produce partial sums, which are reduced through 4 two-level reduction units. Total: 128 MAC DSPs + 12 reduction DSPs = 140 DSPs at 532 MHz, achieving 272 GOP/s using under 1K LUTs — 4× the compute density of the prior generation in W4 mode.
Scalar Engine: 7 DSPs of Magic
The scalar engine handles all non-linear operations: RMSNorm, residual add, MoE-Update, Softmax, UGMul-SiLU, and dynamic FP16-to-INT12 quantization. As these modules do not overlap in time, they share a single resource pool: 3 FP16 multipliers, 2 FP16 adders (with input MUXed via one-hot selection), and 2 lookup tables for SiLU/Sigmoid/Exp. Total DSP cost is just 7, and execution is fully hidden behind GEMV time.
Performance
Two chip configurations are supported: XCZU2CG with 2 Token Processors (1 GEMV, 2 scalar engines) and XCZU3EG with 4 Token Processors (2 GEMV, 4 scalar engines). The XCZU3EG configuration delivers 18 tok/s decoding and 50 tok/s prefill.
Comparison with Jetson AGX Orin:
| Metric | Hummingbird+ (XCZU3EG) | Jetson AGX Orin | Ratio | |---|---|---|---| | Cost | ~$150 | ~$1500 | 0.1× | | Decode speed | 18 tok/s | ~35 tok/s | 0.51× | | Bandwidth utilization | ~100% | ~60% | 1.67× | | Tokens per dollar | 0.12 | 0.023 | 5.2× | | Energy efficiency | 1.8 tok/s/W | 1.1 tok/s/W | 1.6× | | Memory | 24GB | 32GB | 0.75× | | TDP | ~10W | ~60W | 0.17× |
Comparison with prior FPGA LLM accelerators:
| Work | Platform | Model | Precision | Prefill | Decode | Full support | |---|---|---|---|---|---|---| | FlightLLM'24 | VCU128 | LLaMA2-7B | 4-bit | no | 7.7 | decode only | | TeraFly'25 | 4× U280 | OPT-30B | 16-bit | yes | - | multi-node | | LoopLynx'25 | U55C | LLaMA2-7B | 4-bit | yes | 8.8 | - | | Hummingbird+ | XCZU3EG | Qwen3-30B-A3B | 4-bit | yes (50) | 18 | yes |
Limitations and Roadmap
Current limitations include dependence on DDR4 pricing (50% of BOM), the need for manual RTL redesign when model architecture changes, and prefill speed that is still slow for very long inputs. Future plans include scaling to 40GB memory to support Qwen3-Next-80B-A3B (INT4), KU040 with 160-bit DDR4 for 72GB, multi-FPGA systems via SerDes, and CXL memory expansion.
Significance
Hummingbird+ demonstrates that edge AI does not have to choose between GPU and ASIC paths. In memory-bound, cost-sensitive, architecture-stable scenarios, FPGAs can be a viable third option. Notably, since RAM and storage now exceed the FPGA die cost in MoE LLM BOM, even an ASIC cannot dramatically undercut this design on cost unless DRAM prices drop — which may reduce the long-term appeal of edge LLMs ASICs as LLM architectures evolve rapidly.