A May 2026 survey paper by Chenchen Zhang, *Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces* (arXiv:2605.02801), delivers an unsettling statistic: of 84 papers on training LLM-based multi-agent systems with reinforcement learning (2022–May 2026), the number explicitly studying how these systems should learn to stop is zero.
Key points
- Orchestration trace: The paper models all coordination decisions in a multi-agent system as a temporal event graph with 8 atomic event types—spawn, delegate, communicate, tool call, return, aggregate, and stop. It captures system-level interaction history, unlike single-agent chain-of-thought.
- Three technical axes: Zhang maps the 84-paper corpus onto reward design, credit assignment, and orchestration learning.
- Title: Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces
- Author: Chenchen Zhang
- arXiv: 2605.02801 (cs.CL), released May 4, 2026
- Open artifact: github.com/xxzcc/awesome-llm-mas-rl — 84 annotated papers, 32 exclusion logs (116 reviewed), and a JSON Schema for reproducible orchestration traces
- Corpus composition: 42 RL methods, 18 benchmarks, 10 classical MARL foundations, 6 industrial reports, 5 surveys, 3 frameworks
- Six orchestration topologies: centralized orchestrator, planner-executor-critic, debate/committee, parallel swarm, hierarchical, and harness
Axis 1: Reward design
10 reward types were identified. Most common: hybrid (15 papers), shared (10). Critically, only 7 papers (~8.3%) use an orchestration reward covering parallelism, split correctness, and aggregation quality. Most work assumes individual optimality implies system optimality:
The missing piece is an explicit coordination reward:
77 of 84 papers effectively set \(R_{orchestration} = 0\).
Axis 2: Credit assignment
Across 8 granularity levels (team → orchestrator → role → agent → tool → message → turn → token), the counts were: agent 23, role 10, orchestrator 8, message only 2. Message-level credit—understanding which message caused what outcome—requires a differentiable path from message to reward:
This gradient is essentially disconnected in current architectures.
Axis 3: The five orchestration decisions (O1–O5)
| Decision | Question | Papers studying it | |:--|:--|:--| | O1: Spawn | When to create sub-agents? | Present | | O2: Delegate | Which agent gets the task? | Present | | O3: Communicate | What/how to communicate? | Present | | O4: Aggregate | How to combine results? | Present | | O5: Stop | When to terminate? | 0 |
Why stopping matters
Without trained stopping policies, orchestrators fail in three ways: 1. Premature stopping — submitting unverified fixes 2. Late stopping — endless "supplementary" work destroying UX and wasting compute 3. Never stopping — circular dependencies where agents endlessly trigger each other (known in industry as *overthinking* or *token black holes*), currently handled only by hard-coded round caps
The ideal stop condition is a conjunction:
All these terms currently rely on hand-coded heuristics, never RL-trained.
The academia–industry scale gap
The survey links public evidence from three industrial systems—Kimi Agent Swarm (Moonshot AI), OpenAI Codex, and Anthropic Claude Code—and finds their training details, reward designs, and stopping mechanisms nearly entirely undisclosed. As the paper states: *"The resulting scale gap is a gap between publicly reported deployment envelopes and open academic evaluation regimes, not independent verification of industrial training traces."* Academia evaluates 2–5 agent toy environments with fixed stopping heuristics; industry deploys hundreds of coordinated agents with dynamic, unpublished stop logic.
Proposed directions
1. Explicit RL objective functions for O5, e.g. a value function \(V_{stop}(s_t)\) comparing continue-vs-stop expected returns 2. Recording stop signals (why the system stopped) inside orchestration traces for future learning 3. Cross academia–industry evaluation standards for orchestration quality 4. Message-level credit assignment methods requiring new gradient estimation over discrete message→reward paths