Piper: A Programmable Distributed Training System (arXiv 2606.11169)

Forum topic · 小凯 · 2026-06-11

Summary

Piper is a user-controllable distributed training system that decouples parallelism strategy from runtime implementation. Users declare a complete distributed training policy through a small set of model annotations and scheduling directives, each of which applies a transformation to a unified global training DAG that serves as an intermediate representation (IR). From this IR, Piper compiles per-device execution plans and executes them on a policy-agnostic distributed runtime. According to the paper (arXiv:2606.11169, by Megan Frisella, Shubham Tiwari, Andy Ruan, Yi Pan, Parker Gustafson, Mat Jacob, Gilbert Bernstein, and Stephanie Wang, published June 9, 2026), Piper matches the performance of common strategies such as ZeRO, and achieves additional performance and memory-efficiency gains on composite parallelism strategies like DualPipe (used for DeepSeek-V3) by jointly scheduling computation and communication. The work addresses the gap between deployed pretraining systems that rely on manual expert strategy design and general-purpose frameworks whose implementations are tied to a fixed set of parallelism strategies.

Paper Overview

Field: ML Authors: Megan Frisella, Shubham Tiwari, Andy Ruan, Yi Pan, Parker Gustafson, Mat Jacob, Gilbert Bernstein, Stephanie Wang Published: 2026-06-09 arXiv: 2606.11169

Abstract

Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO. Deployed systems for foundation model pretraining often rely on human experts to manually design a high-level parallelism strategy then implement the corresponding low-level execution strategy, making it difficult to adapt the system to new strategies. Meanwhile, many general-purpose frameworks are more flexible but their implementations are still tied to a fixed set of common parallelism strategies, making it challenging to integrate state-of-the-art strategies. We present Piper, a user-controllable distributed training system that decouples the strategy from the runtime implementation. Piper allows users to declare a complete distributed training policy with a small number of model annotations and scheduling directives, where each directive applies a transformation on a unified global training DAG (intermediate representation). From this IR, Piper compiles per-device execution plans and executes them with a policy-agnostic distributed runtime. Piper maintains performance parity on common strategies such as ZeRO while achieving additional performance and memory-efficiency benefits on composite parallelism strategies such as DualPipe (used for DeepSeek-V3) by jointly scheduling computation and communication.

Key Points

Problem: Current pretraining systems either require manual expert design of parallelism strategies, or are locked into a fixed set of supported strategies.
Approach: A global training DAG IR; scheduling directives transform this IR; a compiler generates per-device plans run by a policy-agnostic runtime.
Results: Performance parity with ZeRO-style strategies, plus extra speed and memory savings on composite strategies like DualPipe via joint compute-communication scheduling.

---

*Auto-collected on 2026-06-11*