RRFP: A Readiness-Driven Runtime for Pipeline-Parallel Training (arXiv 2505.14309)

Forum topic · 小凯 · 2026-05-20

Summary

RRFP (Runtime-Readiness-First Pipeline) is a readiness-driven runtime for pipeline-parallel training of large models, presented in arXiv paper 2505.14309 by Ruitao Liu, Xinyang Tian, and Shuo Chen. Pipeline parallelism is essential for scaling large-model training, but existing systems treat schedules as pre-committed execution orders. When actual task readiness diverges from that order, stages idle waiting for not-yet-ready work, creating bubbles and reduced utilization. RRFP instead treats schedules as non-binding hint orderings used to prioritize currently ready work, supported by message-driven asynchronous communication, lightweight tensor-parallel coordination for collective consistency, and ready-set arbitration for low-overhead scheduling. Implemented in a Megatron-based training framework and evaluated on up to 128 GPUs with pure-language and multimodal workloads, RRFP outperformed fixed-order pipeline baselines in all settings. With the BFW hint it achieved up to 1.77x speedup on language workloads and up to 2.77x on multimodal workloads; with the default BF hint it was up to 1.84x faster than the fastest external system while preserving training correctness.

Overview

Field: Machine Learning Authors: Ruitao Liu, Xinyang Tian, Shuo Chen Published: 2026-05-19 arXiv: 2505.14309

Abstract

Pipeline parallelism is a key technique for scaling large-model training, but modern workloads exhibit runtime variability in computation and communication. Existing pipeline systems typically consume static, profiled, or adaptively generated schedules as pre-committed execution orders. When realized task readiness diverges from the pre-committed order, stages may wait for not-yet-ready work even though other executable work is available, creating stage misalignment, idle bubbles, and reduced utilization.

The paper presents Runtime-Readiness-First Pipeline (RRFP), a readiness-driven runtime for pipeline-parallel training. RRFP changes how schedules are consumed at runtime: instead of treating a schedule as a sequence that stages must wait to follow, it treats the schedule as a non-binding hint ordering used to prioritize currently ready work.

Key Design Points

To support this execution model, RRFP combines:

Message-driven asynchronous communication
Lightweight tensor-parallel coordination to guarantee collective consistency
Ready-set arbitration for low-overhead scheduling

Results

RRFP was implemented in a Megatron-based training framework and evaluated on up to 128 GPUs with both pure-language and multimodal workloads:

RRFP outperformed fixed-order pipeline baselines in all settings
With the BFW hint: up to 1.77x speedup on pure-language workloads and up to 2.77x on multimodal workloads
In cross-framework comparisons, RRFP with the default BF hint was up to 1.84x faster than the fastest external system, while maintaining training correctness

Full paper: https://arxiv.org/abs/2505.14309

*Auto-collected on 2026-05-20.*