Paper Overview
Field: Machine Learning
Authors: Tate Berenbaum, Muthaiah Venkatachalam
Published: 2026-08-19
arXiv: 2608.19147
Code: github.com/labscommunity/pipeline-sharded-inference-paper (reproduction scripts in the top-level reproduction/ directory)
Summary
Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, yet they spend considerable time idle. That memory is not enough to fit large models such as a 70B-parameter LLM. This paper demonstrates that a handful of AI PCs working together over an ordinary network can serve models beyond the capability of any single device.
The system uses pipeline parallelism: the model is split by layer into per-stage shards, each pre-compiled into an OpenVINO graph, so every machine runs one shard and passes activations to the next.
Three Key Techniques
1. Recovering unsplit-model speed: A naive per-stage export runs well below monolithic inference because it misses an OpenVINO GPU optimization. Injecting a beam_idx Gather into each shard triggers the IndirectKVCache fusion and brings shards to parity with the unsplit model.
2. Speculative decoding on stateful OpenVINO models: The authors adapt speculative decoding to work with OpenVINO's stateful inference.
3. Micro-batching: The pipeline serves multiple concurrent users by interleaving requests across stages, with each request carrying its own KV cache.
Results
- A two-node Llama 3.1 8B INT4 pipeline served two concurrent users at 1.79x the throughput of a single-user unsplit model on identical hardware; the gap widened under simulated WAN latency.
- The same design scaled to a 70B model that no single cluster member could hold: a four-node Lunar Lake AI PC deployment on Intel Tiber Cloud served a single user at interactive speed, with outputs token-for-token identical to non-speculative decoding of the same four-node pipeline.
*Collected automatically on 2026-08-21.*