[论文] Piper: A Programmable Distributed Training System

论文概要

研究领域: ML 作者: Megan Frisella, Shubham Tiwari, Andy Ruan, Yi Pan, Parker Gustafson, Mat Jacob, Gilbert Bernstein, Stephanie Wang 发布时间: 2026-06-09 arXiv: 2606.11169

中文摘要

Piper是一个用户可控的分布式训练系统，将策略与运行时实现解耦。用户通过少量模型注释和调度指令声明完整的分布式训练策略，每个指令对统一全局训练DAG（中间表示）应用变换。基于此IR，Piper编译每设备执行计划并用与策略无关的分布式运行时执行。在常见策略如ZeRO上保持性能对等，同时在DeepSeek-V3的DualPipe等组合并行策略中通过联合调度计算和通信实现额外性能和内存效率提升。

原文摘要

Large-scale model training increasingly relies on composing multiple parallelism strategies, such as data, pipeline, and expert parallelism, together with memory-saving optimizations like ZeRO. Deployed systems for foundation model pretraining often rely on human experts to manually design a high-level parallelism strategy then implement the corresponding low-level execution strategy, making it difficult to adapt the system to new strategies. Meanwhile, many general-purpose frameworks are more flexible but their implementations are still tied to a fixed set of common parallelism strategies, making it challenging to integrate state-of-the-art strategies. We present Piper, a user-controllable distributed training系统 that decouples the strategy from the runtime implementation. Piper allows use...

--- *自动采集于 2026-06-11*

#论文 #arXiv #ML #小凯