RRFP: A Readiness-Driven Runtime for Pipeline-Parallel Training

RRFP: A Readiness-Driven Runtime for Pipeline-Parallel Training

Source: arXiv 2505.14309

Authors: Ruitao Liu, Xinyang Tian, Shuo Chen

Release Date: 2026-05-19

Motivation

Pipeline parallelism is a key technique for scaling large-model training, but modern workloads exhibit runtime variability in both computation and communication. Existing pipeline systems typically consume static, profiled, or adaptively generated schedules as pre-committed execution orders. When realized task readiness diverges from the pre-committed order, stages may wait for not-yet-ready work even though other executable work is available, leading to stage misalignment, idle bubbles, and reduced utilization.

Approach

This paper presents Runtime-Readiness-First Pipeline (RRFP), a readiness-driven runtime for pipeline-parallel training. RRFP changes how schedules are consumed at runtime: instead of treating a schedule as a sequence that stages must wait to follow, it treats the schedule as a non-binding hint ordering over currently ready work.

To support this mode, RRFP combines:

• Message-driven asynchronous communication for non-blocking coordination between stages.
• Lightweight tensor-parallel coordination to guarantee collective consistency across tensor-parallel groups.
• Ready-set arbitration to enable low-overhead scheduling decisions based on what is actually executable.

Evaluation

RRFP is implemented on top of a Megatron-based training framework and evaluated on up to 128 GPUs across both language-only and multimodal workloads.

Results:

• RRFP outperforms fixed-order pipeline baselines in all tested settings.
• With BFW hints, RRFP achieves up to 1.77x speedup on language-only workloads and up to 2.77x speedup on multimodal workloads.
• In cross-framework comparison, RRFP with the default BF hint is 1.84x faster than the fastest external system, while preserving training correctness.

Key Takeaways

• Pre-committed static schedules are a bottleneck under runtime-variable workloads.
• Treating schedules as non-binding hints over ready work reduces idle bubbles and stage misalignment.
• Decoupling ready-set arbitration from communication enables higher GPU utilization at scale.
• RRFP generalizes across pure language and multimodal training regimes.