Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PCs (arXiv 2608.19147)

Forum topic · 小凯 · 2026-08-21

Summary

A new arXiv paper (2608.19147) by Tate Berenbaum and Muthaiah Venkatachalam shows that a few Intel AI PCs, connected over an ordinary network, can jointly serve LLMs larger than any single device's 16+ GB unified memory can hold. The system uses pipeline parallelism: the model is split by layer into per-stage shards, each pre-compiled as an OpenVINO graph, with each machine running one shard and passing activations onward. Three techniques make it practical: injecting a beam_idx Gather into each shard to trigger OpenVINO's IndirectKVCache fusion and recover monolithic-inference speed; enabling speculative decoding on stateful OpenVINO models; and micro-batching concurrent requests, each carrying its own KV cache, across pipeline stages. A two-node Llama 3.1 8B INT4 pipeline serves two users at 1.79x the single-user throughput of an unsplit model on identical hardware, with the gap widening under simulated WAN latency. The design scales to 70B models: a four-node Lunar Lake AI PC deployment on Intel Tiber Cloud serves a single user at interactive speed, with token-identical outputs to non-speculative decoding. Code, benchmark logs, and reproduction scripts are available on GitHub.

This forum post introduces the paper "Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI PCs" (arXiv: 2608.19147) by Tate Berenbaum and Muthaiah Venkatachalam, published 2026-08-19.

Full Translation of the Abstract

Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, and they spend considerable time idle. That is not enough memory to fit a large model such as a 70B-parameter LLM. The authors show that a handful of AI PCs, working together over an ordinary network, can serve models beyond the capability of any single one.

The system uses pipeline parallelism: a model is split by layer into per-stage shards, each pre-compiled into an OpenVINO graph, so that every machine runs one shard and passes activations to the next. Three techniques make this fast enough to be practical:

1. Recovering unsplit-model speed. A naive per-stage export runs well below monolithic inference because it misses an OpenVINO GPU optimization. Injecting a beam_idx Gather into each shard triggers that optimization (IndirectKVCache fusion) and brings shards back on par. 2. Speculative decoding on stateful OpenVINO models. 3. Micro-batching. The pipeline serves multiple users simultaneously by interleaving requests across stages, with each request carrying its own cache.

Results

A two-node Llama 3.1 8B INT4 pipeline serves two concurrent users at 1.79x the single-user, unsplit-model throughput on the same hardware; the gap widens under simulated WAN latency.
The same design scales to a 70B model that no single fleet member could hold: a four-node Lunar Lake AI PC deployment on Intel Tiber Cloud serves a single user at interactive speed, with output tokens token-for-token identical to the same four-node pipeline without speculation.

Reproducibility

Code, raw benchmark logs, and reproduction scripts are available as a self-contained package at https://github.com/labscommunity/pipeline-sharded-inference-paper (see the top-level reproduction/ directory).

*Auto-collected on 2026-08-21.*