English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

AstraFlow: Dataflow Architecture Cuts RL Training Costs for Agentic LLMs by 2.7x

Forum topic · 小凯 · 2026-05-19

Summary

Reinforcement learning is increasingly used to improve LLM reasoning, coding, and tool-use capabilities, as demonstrated by DeepSeek-R1. However, scaling RL to agent scenarios is extremely expensive: agents require environment interaction, tool calls, long-context handling, and coordinated training across multiple policies—each demanding additional systems engineering in existing trainer-centric RL frameworks. Researchers from the University of Michigan and UC Berkeley (Zheng, Di, Wang, Jin, Liu, Wu, Mao, Stoica, Zhao, and Chen) propose AstraFlow, which replaces trainer-centric control with a dataflow architecture that decouples rollout services, dataflow management, and training into independent, autonomous components. This design natively supports multi-policy co-training, automatic elastic scaling of rollout workers, and unified scheduling of geographically distributed heterogeneous compute (e.g., cloud GPUs for training, edge GPUs for rollouts). On math, code, search, and AgentBench workloads, AstraFlow matches or exceeds the accuracy of existing systems while achieving a 2.7x training speedup—a pure systems-level gain requiring no changes to the RL algorithm. Open questions remain regarding the hardware configuration behind the 2.7x measurement, whether dataflow network latency bottlenecks high-frequency small-batch updates, and at what scale inter-component communication overhead negates decoupling benefits.

Reinforcement learning is being used to improve LLM reasoning, coding, and tool-use capabilities—the success of DeepSeek-R1 came from RL training of reasoning chains. But scaling RL to agent scenarios is extremely expensive: an agent needs to interact with environments, call tools, handle long contexts, and co-train across multiple policies. Existing RL systems require extra systems engineering for every added capability.

The AstraFlow Approach

Zheng, Di, Wang, Jin, Liu, Wu, Mao, Stoica, Zhao, and Chen (University of Michigan, UC Berkeley, and others) propose AstraFlow, which replaces the traditional trainer-centric control architecture with a dataflow-oriented architecture. The core change is simple: decouple rollout services, dataflow management, and training into independent, autonomous components.

What Decoupling Enables

  • Native multi-policy co-training: different agent behaviors can roll out simultaneously, with the dataflow automatically routing data to the correct training components.
  • Automatic elastic scaling: add more rollout workers when needed, without manually reconfiguring trainers.
  • Unified scheduling of heterogeneous, geographically distributed compute: e.g., cloud GPUs for training and edge GPUs for rollouts—a topology the dataflow model naturally supports.
  • Results

    On math, code, search, and AgentBench workloads, AstraFlow achieves accuracy comparable to or better than existing systems in multi-policy collaborative training, while accelerating training by 2.7x. This is a purely systems-level speedup—no changes to the RL algorithm, only the architecture.

    Open Questions

  • Under what specific hardware configurations and workloads was the 2.7x speedup measured?
  • Does dataflow network latency become a bottleneck for high-frequency, small-sample RL updates?
  • The decoupled architecture introduces new inter-component communication overhead—at what scales does this overhead outweigh the benefits of decoupling?
---

References

1. Zheng, H., Di, Y., Wang, J., et al. (2026). *AstraFlow: Dataflow-Oriented Reinforcement Learning for Agentic LLMs*. arXiv:2605.15565 [cs.LG]. 2. Schulman, J., et al. (2017). *Proximal Policy Optimization Algorithms*. arXiv. 3. DeepSeek-AI. (2025). *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. arXiv.

Tags

#reinforcement-learning#llm#agentic-ai#distributed-systems#dataflow-architecture#astraflow#training-infrastructure#systems-design

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177620364