[论文] Pre-Compiled Pipeline Shards for Distributed LLM Inference on Intel AI...

论文概要

研究领域: ML 作者: Tate Berenbaum, Muthaiah Venkatachalam 发布时间: 2026-08-19 arXiv: 2608.19147

中文摘要

现代Intel AI PC配备功能强大的集成GPU和NPU，拥有16+ GB统一内存，但它们有大量空闲时间。这不足以容纳大型模型，如70B参数LLM。我们证明，少数AIPC通过普通网络协同工作，可以服务超出任何单一设备能力的模型。我们使用流水线并行：模型按层分割为每阶段分片，每个分片预编译为OpenVINO图，因此每台机器运行一个分片并将激活传递给下一个。三种技术使其足够快以实用。首先，我们恢复未分割模型的速度：朴素的每阶段导出运行远低于单体推理，因为它遗漏了OpenVINO GPU优化，而在每个分片中注入beam_idx Gather触发该优化（IndirectKVCache融合）并使分片达到同等水平。其次，我们在有状态OpenVINO模型上利用推测解码。第三，流水线通过跨阶段交错请求来同时服务多个用户，每个请求携带自己的缓存（微批处理）。综合起来，双节点Llama 3.1 8B INT4流水线以相同硬件上单用户未分割模型吞吐量的1.79倍服务两个并发用户，在模拟广域延迟下差距扩大。同一设计扩展到没有任何单个机群成员能容纳的70B模型：Intel Tiber Cloud上的四节点Lunar Lake AI PC部署以交互速度服务单个用户，输出token与无推测的相同四节点流水线解码token-for-token相同。代码、原始基准日志和复现脚本作为自包含包在https://github.com/labscommunity/pipeline-sharded-inference-paper提供（在顶层reproduction/目录中）。

原文摘要

Modern Intel AI PCs ship capable integrated GPUs and NPUs with 16+ GB of unified memory, and they spend considerable time idle. That is not enough memory to fit a large model such as a 70B-parameter LLM. We show that a handful of AIPCs, working together over an ordinary network, can serve models beyond the capability of any single one. We use pipeline parallelism: a model is split by layer into per-stage shards, each pre-compiled into an OpenVINO graph, so that every machine runs one shard and passes activations to the next. Three techniques make this fast enough to be useful. First, we recover the speed of the unsplit model: a naive per-stage export runs well below monolithic inference because it misses an OpenVINO GPU optimization, and injecting a beam_idx Gather into each shard triggers...

--- *自动采集于 2026-08-21*

#论文 #arXiv #ML #小凯