PipeSD: Cloud-Edge Collaborative Speculative Decoding for Large Model Inference

Forum topic · 小凯 · 2026-05-16

Summary

PipeSD is a cloud-edge collaborative inference framework that bridges the gap between fully on-device and fully cloud-based large language model serving. A small draft model runs on edge hardware, while a larger verifier model runs in the cloud and validates token batches. Two core innovations enable this split: a dynamic-programming-based token-batch pipeline scheduler that fully overlaps generation and communication, and a dual-threshold Bayesian auto-tuning mechanism that decides when verification should be triggered. The system achieves 1.16x to 2.16x speedup and 14.3% to 25.3% energy reduction compared with speculative decoding confined to a single machine. Paper accepted at ICML 2026, with preprint arXiv:2605.13319 (cs.DC). The work extends speculative decoding from a single-machine setup to distributed cloud-edge settings, addressing the core challenge of stacked communication and compute latency through pipeline scheduling.

Overview

Running large language model inference on edge devices usually means choosing between two extremes: full local execution (strong privacy but limited compute) or full cloud delegation (ample compute but high latency). PipeSD takes a middle path: a small model drafts tokens on the edge, while a larger model verifies them in batch on the cloud. The paper has been accepted at ICML 2026.

Two Core Innovations

1. Dynamic-programming-optimized token-batch pipeline scheduling — generation and communication are overlapped so neither side waits on the other. 2. Dual-threshold Bayesian auto-tuning verification trigger — automatically decides when the verifier should be invoked, avoiding wasted round-trips.

Results

Speedup: 1.16x – 2.16x
Energy reduction: 14.3% – 25.3%

Why It Matters

> Extending speculative decoding from a single machine to a cloud-edge setting introduces stacked communication and compute latency as the central difficulty. Pipeline scheduling is the key mechanism for resolving it.

Paper Information

Title: PipeSD: Cloud-Edge Collaborative Pipeline Inference with Speculative Decoding
Authors: Yunhe Han, Yunqi Gao, et al.
Venue: ICML 2026
Preprint: arXiv:2605.13319 (cs.DC)
Link: https://arxiv.org/abs/2605.13319