Paper Overview
Field: Training optimization Authors: Philip Zmushko, Egor Petrov, Nursultan Abdullaev Published: 2026-07-01 arXiv: 2507.00005
Abstract
Modern large-scale LLM pretraining benefits from utilizing Pipeline Parallelism; however, synchronous implementations leave GPUs idle during pipeline bubbles, wasting computational resources. Asynchronous Pipeline Parallelism eliminates these bubbles, maximizing throughput at the cost of gradient staleness. Among asynchronous schedules, PipeDream-2BW is particularly appealing: unlike the original PipeDream schedule, it ensures a constant one-step gradient delay regardless of pipeline depth. However, its adoption remains limited due to the common belief that optimizing under staleness is fundamentally unstable.
In this work, the authors challenge this assumption, demonstrating that degradation under one-step delay depends strongly on optimizer choice rather than being an intrinsic limitation. They provide the first comprehensive empirical analysis showing that although AdamW — dominant when PipeDream-2BW was introduced — suffers severe degradation, newer methods like Muon exhibit strong robustness under one-step delay. They also introduce an optimizer-agnostic, error-feedback-inspired correction to further mitigate delay effects, and provide supporting theoretical analysis proving Muon converges with and without this correction. Extensive evaluations on models up to 10B parameters confirm that their strategies bridge the performance gap with synchronous training, highlighting the practical potential of large-scale asynchronous pipeline parallelism.
--- *Auto-collected on 2026-07-01*