Paper Overview
Research area: Machine Learning Authors: Megan Frisella, Shubham Tiwari, Andy Ruan, Yi Pan, Parker Gustafson, Mat Jacob, Gilbert Bernstein, Stephanie Wang Published: 2026-06-09 arXiv: 2606.11169
Abstract (translated from the forum post)
Piper is a user-controllable distributed training system that decouples the strategy from the runtime implementation. Users declare a complete distributed training policy through a small set of model annotations and scheduling directives; each directive applies a transformation to a unified global training DAG, which serves as the intermediate representation (IR). Based on this IR, Piper compiles per-device execution plans and executes them with a policy-agnostic distributed runtime. It maintains performance parity on common strategies such as ZeRO, while achieving additional performance and memory-efficiency improvements on compositional parallelism strategies such as DualPipe (used in DeepSeek-V3) by jointly scheduling computation and communication.
Original Abstract (excerpt)
Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO. Deployed systems for foundation model pretraining often rely on human experts to manually design a high-level parallelism strategy then implement the corresponding low-level execution strategy, making it difficult to adapt the system to new strategies. Meanwhile, many general-purpose frameworks are more flexible but their implementations are still tied to a fixed set of common parallelism strategies, making it challenging to integrate state-of-the-art strategies. We present Piper, a user-controllable distributed training system that decouples the strategy from the runtime implementation...
Key Takeaways
- Strategy/runtime decoupling: training policies are expressed as directives over a global training DAG IR.
- Compiler-based execution: per-device plans are compiled from the IR and run on a policy-agnostic runtime.
- Evaluated results: parity with ZeRO-class strategies; extra performance and memory gains on compositional strategies like DeepSeek-V3's DualPipe via joint compute/communication scheduling.