Hummingbird+ Deep Dive: Running a 30B-Parameter MoE LLM on a $150 FPGA
> Engineers from the Chinese Academy of Sciences achieved what looks impossible: fitting a 30B-parameter mixture-of-experts (MoE) large language model onto a $150 entry-level FPGA chip, delivering fluent conversation at 18 tokens/s. This is not a lab demo — it's a production-grade design.
The "Impossible Four Corners" of Edge AI
Edge LLM deployment has traditionally forced engineers to pick three out of four:
| Dimension | Status quo | |------|------| | 💰 Cost | Cloud A100 rental > $1000/month; Jetson AGX Orin dev board > $1500 | | 🤖 Model scale | 7B dense models feasible; 30B+ nearly impossible | | ⚡ Performance | Decode < 10 tok/s — sluggish | | 📦 Power/size | High-end GPUs draw 300W+ and need active cooling |
The Hummingbird+ team from the Institute of Automation, CAS, broke this trade-off: on a $150 platform, they achieved a 30B MoE model, 18 tok/s decode speed, and a productizable design.
---
1. Why FPGA?
1.1 The Awkward Position of FPGAs
FPGAs have long sat in an awkward spot for AI acceleration:
- Less compute than GPUs: an H100 offers 989 TFLOPS FP16; high-end FPGAs (e.g., Alveo U280) only 8–20 TFLOPS
- Less energy-efficient than ASICs: once taped out, ASIC per-watt performance crushes FPGA
- Very hard to program: steep Verilog/VHDL learning curve; HLS-generated circuits are inefficient
- Transformer training: matrix-matrix multiplication (N³), compute-bound → GPU territory
- LLM decoding (especially): matrix-vector multiplication (GEMV), bottlenecked by memory bandwidth → near the memory wall
- Flexible memory interfaces: optimally configured DDR controllers (width, frequency, channels)
- Low-latency on-chip storage: BRAM/URAM caches activations, reducing external accesses
- Dataflow architecture: no GPU warp-scheduling overhead; every data path is customizable
- Processing System (PS): Quad-core Cortex-A53, dual-core Cortex-R5F, Mali-400 GPU, with 8GB onboard DDR4
- Programmable Logic (PL): XCZU2CG (47K LUTs, 240 DSPs) or XCZU3EG (70K LUTs, 360 DSPs), ~4.5Mb BRAM — this is where the accelerator lives, with 16GB dual-channel DDR4 on SODIMM
- Extra interfaces: M.2 NVMe (128GB SSD) for model weights, PCIe 2.0, Gigabit Ethernet, USB 3.0
- GEMV engine: 140 DSPs, 272 GOPs
- Scalar engine: 7 DSPs (handles all nonlinear operations)
- Activation buffers: BRAM
- Prefill stage: all Token Processors work in parallel on different tokens
- Decode stage: master-slave cooperation — the master runs the full flow (attention + MLP), slaves share the GQA workload
- 3 FP16 multipliers — reused by all modules
- 2 FP16 adders — input channels selected via one-hot
- 2 lookup-table units — storing SiLU/Sigmoid/Exp tables
- First deployment of a 30B MoE model on an embedded FPGA
- First low-cost FPGA accelerator supporting both prefill and decode
- Highest decode performance among embedded FPGAs
- Lowest cost per token
- Next step: expand memory to 8+32 = 40GB, supporting Qwen3-Next-80B-A3B (INT4)
- Longer term: KU040 (160-bit DDR4) → 72GB memory; multi-FPGA systems via SerDes interconnect; CXL memory expansion
- Cloud training: GPUs (high compute, high cost)
- Cloud inference: ASICs (high efficiency, medium cost)
- Edge inference: FPGA/ASIC? (low cost, ultra-low power)
- Paper: ACM/SIGDA FPGA 2026, https://dl.acm.org/doi/10.1145/3748173.3779189
- Predecessor work: Hummingbird (arXiv:2507.03308)
- Qwen3 model: https://modelscope.cn/models/Qwen/Qwen3-30B-A3B
- Conference: https://www.isfpga.org/program/
1.2 LLM Inference Changes the Game
But LLM inference is memory-bound, not compute-bound:
GEMV means the bottleneck is not the number of multipliers but how fast weights can be read from memory. FPGAs fit here because of:
---
2. MoE: A Natural Ally for FPGA
2.1 The Secret of Mixture-of-Experts
MoE is a sparsely activated architecture: a router (gating network) computes softmax scores over all experts, selects Top-K (typically K=8), and combines their outputs with score-weighted summation.
Key insight: total parameters ≠ activated parameters.
| Model | Total params | Activated params | Sparsity | |------|----------|-----------|--------| | Qwen3-30B-A3B | 30.5B | 3.3B | ~90% | | Conventional dense model | 7B | 7B | 0% |
Each token activates only a small subset of experts, so the memory access workload matches a small dense model — right in the comfort zone of low-cost DDR hardware.
2.2 Why Qwen3-30B-A3B
| Property | Value | Significance | |------|------|------| | Total parameters | 30.5B | Large-model capability | | Activated parameters | 3.3B | Manageable compute load | | Experts | 128 | High sparsity | | Experts per layer | 8 | Predictable memory access | | GQA | 32Q/4KV | Reduces KV cache pressure | | GPTQ 4-bit quantized | 14.52GB | Fits in 24GB memory budget |
With a 16K-context KV cache (0.77GB), total memory need is ~15.29GB — a perfect fit for the platform.
---
3. The Hardware Platform: What $150 Buys
3.1 Custom PCB Design
The platform centers on a Zynq UltraScale+ MPSoC:
3.2 Cost Breakdown
| Component | Est. cost | Share | |------|----------|------| | XCZU2CG/3EG FPGA | ~$45 | 30% | | 24GB DDR4 (8+16) | ~$75 | 50% | | PCB + power + misc | ~$30 | 20% | | Total | ~$150 | 100% |
Memory accounts for half the cost — a key insight: in MoE LLM deployment, storage cost already exceeds the compute chip itself. This means even switching to an ASIC offers limited cost reduction unless DRAM prices fall.
3.3 Bandwidth Position
The PL-side dual-channel DDR4 provides 34GB/s peak bandwidth:
| Platform | Memory | Peak BW | Capacity | TDP | |------|----------|----------|----------|-----| | Jetson AGX Orin | LPDDR5 | 204GB/s | 32GB | 60W | | Hummingbird+ | DDR4 | 34GB/s | 24GB | ~10W | | Ratio | — | 0.17x | 0.75x | 0.17x |
Bandwidth is only 17% of Jetson's, but the key is utilization — Hummingbird+ achieves near 100% of theoretical bandwidth through architectural optimization, versus Jetson's typical 60–70%.
---
4. Architectural Innovation: The Token Processor
4.1 Flaws of Conventional Designs
Existing FPGA LLM accelerators typically use a coarse-grained three-module design (matrix engine → nonlinear unit → memory manager), with problems: 1. Shape mismatch → poorly sized matrix engines give low utilization 2. Insufficient pipeline granularity → decode is latency-sensitive; coarse designs waste cycles 3. Loose module coupling → over-provisioned nonlinear units waste resources
4.2 Token Processor: A Compact End-to-End Pipeline
Each Token Processor contains:
Multiple processors (1 master + slaves) share the DDR4 memory bus:
---
5. The GEMV Engine: The Art of 140 DSPs
5.1 Dual-Precision Operand Packing
The core challenge: efficient low-precision compute on limited DSPs (Xilinx DSP48E2: A: 30-bit, B: 18-bit, C: 48-bit, D: 27-bit inputs).
W4 mode (4-bit weights, 12-bit activations): two 4-bit weights are packed into ports A and D with sign extension, so computing (sign_ext(w1) + w2) × act yields two weight×activation products per multiplication.
KV8 mode (8-bit keys/values): port A is gated to zero via INMODE[1], port D carries the full 8-bit data; five 2-to-1 MUXes handle precision switching at low LUT cost.
Eliminating the +1 correction term: signed-integer packing normally needs correction logic for negative multiplications. Hummingbird+ absorbs the correction into the DSP's four-input post-adder via a constant rounding factor on the W mux, eliminating extra LUT logic.
5.2 Chain-Tree Hybrid Reduction
With parallelism K=128 (M=2 via DDR): 16 DSP chains of 8 DSPs each produce partial sums, reduced by two levels of 4-to-1 reduction units (first level SIMD=2, second level SIMD=1). Total: 128 MAC DSPs + 12 reduction DSPs = 140 DSPs.
5.3 Performance
| Metric | Value | |------|------| | Clock frequency | 532 MHz | | MAC throughput | 272 GOP/s | | LUT usage | < 1K | | DSP usage | 140 | | Compute density vs. predecessor | 4x (W4 mode) |
---
6. The Scalar Engine: Magic with 7 DSPs
6.1 Function List
The scalar engine handles all nonlinear operations: RMSNorm (query/key normalization, post-attention normalization), residual addition, MoE-Update (expert weighted sum), Softmax (attention and routing scores), UGMul-SiLU (fused up-projection and gating), and dynamic quantization (FP16 → INT12).
6.2 Resource Sharing via Time Multiplexing
Analysis shows these modules do not overlap in time. So Hummingbird+ shares resources:
Result: total DSP consumption of only 7.
6.3 Execution-Time Hiding
Scalar engine execution is fully hidden within GEMV compute time — nonlinear operations never become the bottleneck.
---
7. System Configuration and Results
7.1 Two Chip Configurations
| Config | XCZU2CG | XCZU3EG | |------|---------|---------| | LUT | 47K | 70K | | DSP | 240 | 360 | | Token Processors | 2 | 4 | | GEMV engines | 1 (time-multiplexed) | 2 | | Scalar engines | 2 | 4 |
7.2 Performance Comparison
vs. Jetson AGX Orin (XCZU3EG configuration):
| Metric | Hummingbird+ | Jetson AGX Orin | Ratio | |------|------------------------|-----------------|------| | Cost | ~$150 | ~$1500 | 0.1x | | Decode speed | 18 tok/s | ~35 tok/s | 0.51x | | Bandwidth utilization | ~100% | ~60% | 1.67x | | Token-per-dollar | 0.12 | 0.023 | 5.2x | | Energy efficiency | 1.8 tok/s/W | 1.1 tok/s/W | 1.6x | | Memory capacity | 24GB | 32GB | 0.75x | | TDP | ~10W | ~60W | 0.17x |
vs. existing FPGA accelerators:
| Work | Platform | Model | Compression | Prefill | Decode | Notes | |------|------|------|------|--------|------|------| | FlightLLM'24 | VCU128 | LLaMA2-7B | 4-bit | ❌ | 7.7 | decode only | | TeraFly'25 | 4×U280 | OPT-30B | 16-bit | ✅ | – | multi-node | | LoopLynx'25 | U55C | LLaMA2-7B | 4-bit | ✅ | 8.8 | – | | Hummingbird+ | XCZU3EG | Qwen3-30B-A3B | 4-bit | ✅ 50 | 18 | ✅ |
7.3 Key Achievements
---
8. Limitations and Future Outlook
8.1 Current Limitations
1. Memory dependency: DDR4 is 50% of BOM cost; price hikes erode the advantage 2. Hand-written RTL: model architecture changes require redesign (though the Transformer architecture has stabilized) 3. Prefill speed: 50 tok/s is still slow for long inputs
8.2 Scalability Roadmap
8.3 Competing with ASICs
Key insight: in MoE LLMs, RAM and storage costs already exceed the FPGA chip itself. Since decode remains memory-bound even with ASIC compute, if FPGAs prove viable for edge LLM deployment and ASICs cannot offer decisive advantages in chip cost or FLOPs, the appeal of ASICs may diminish amid rapidly evolving LLM architectures.
---
9. Engineering Takeaways
9.1 Design Philosophy
1. Problem matching: FPGAs are not a panacea, but have a unique place in memory-bound scenarios 2. Extreme optimization: hand-written RTL is time-consuming but essential under resource constraints 3. Cost first: the primary constraint for edge deployment is cost, not performance 4. Architecture stability: LLM architectures (Transformer + MoE) have stabilized, making dedicated optimization worthwhile
9.2 Implications for AI Hardware
Hummingbird+ shows edge inference can be an FPGA sweet spot. The key distinction: the cloud optimizes TFLOPS/W (compute efficiency); the edge optimizes tokens/$ (cost efficiency).
---
10. Conclusion
| Dimension | Breakthrough | |------|------| | Cost | $150 BOM — first proof that low-cost FPGAs can productively deploy LLMs | | Scale | 30B MoE — largest model among existing FPGA accelerators | | Performance | 18 tok/s decode, 50 tok/s prefill — fluent and usable | | Efficiency | 1.6–1.7x over Jetson AGX Orin; 1.8 tok/s/W |
It tells the industry: edge AI is not limited to GPU and ASIC paths. Under specific constraints (memory-bound, stable architecture, cost-sensitive), the FPGA can be the overlooked "third option."
---
References: