English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

84 Papers on RL for LLM Multi-Agent Systems, Zero on How to Stop: The Orchestration Trace Gap

Forum topic · 小凯 · 2026-05-05

Summary

A 2026 survey by Chenchen Zhang (arXiv:2605.02801) systematically reviewed 84 academic papers on training LLM-based multi-agent systems with reinforcement learning, tagging each for reward type, credit assignment granularity, orchestration topology, and application domain. Its most striking finding: none of the 84 papers explicitly studies the stopping decision—when an orchestrator should terminate a multi-agent workflow. Using the novel orchestration trace framework, a temporal event graph capturing spawn, delegate, communicate, aggregate, and stop decisions, the survey reveals sparse coverage elsewhere too: only 7 papers (~8%) address orchestration-level rewards, and only 2 (~2.4%) tackle message-level credit assignment. The author also documents a scale gap between academic evaluations (small simulated environments, fixed-round stopping heuristics) and industrial deployments like Kimi Agent Swarm, OpenAI Codex, and Anthropic Claude Code, whose training and stopping mechanisms remain opaque. Without learned stopping policies, production systems risk overthinking, circular dependencies, runaway resource consumption, and alignment drift. The paper proposes explicit RL objectives for stopping, stop-signal annotation in traces, cross academia-industry evaluation standards, and message-level credit methods, with an open artifact of 84 annotated papers released on GitHub.

A May 2026 survey paper by Chenchen Zhang, *Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces* (arXiv:2605.02801), delivers an unsettling statistic: of 84 papers on training LLM-based multi-agent systems with reinforcement learning (2022–May 2026), the number explicitly studying how these systems should learn to stop is zero.

Key points

  • Orchestration trace: The paper models all coordination decisions in a multi-agent system as a temporal event graph with 8 atomic event types—spawn, delegate, communicate, tool call, return, aggregate, and stop. It captures system-level interaction history, unlike single-agent chain-of-thought.
  • Three technical axes: Zhang maps the 84-paper corpus onto reward design, credit assignment, and orchestration learning.
  • Axis 1: Reward design

    10 reward types were identified. Most common: hybrid (15 papers), shared (10). Critically, only 7 papers (~8.3%) use an orchestration reward covering parallelism, split correctness, and aggregation quality. Most work assumes individual optimality implies system optimality:

    \[R_{total} = \sum_{i=1}^{n} R_{individual}^{(i)}\]

    The missing piece is an explicit coordination reward:

    \[R_{orchestration} = \alpha \cdot R_{parallelism} + \beta \cdot R_{split} + \gamma \cdot R_{aggregate}\]

    77 of 84 papers effectively set \(R_{orchestration} = 0\).

    Axis 2: Credit assignment

    Across 8 granularity levels (team → orchestrator → role → agent → tool → message → turn → token), the counts were: agent 23, role 10, orchestrator 8, message only 2. Message-level credit—understanding which message caused what outcome—requires a differentiable path from message to reward:

    \[Credit(msg_i) = \frac{\partial R}{\partial msg_i}\]

    This gradient is essentially disconnected in current architectures.

    Axis 3: The five orchestration decisions (O1–O5)

    | Decision | Question | Papers studying it | |:--|:--|:--| | O1: Spawn | When to create sub-agents? | Present | | O2: Delegate | Which agent gets the task? | Present | | O3: Communicate | What/how to communicate? | Present | | O4: Aggregate | How to combine results? | Present | | O5: Stop | When to terminate? | 0 |

    Why stopping matters

    Without trained stopping policies, orchestrators fail in three ways: 1. Premature stopping — submitting unverified fixes 2. Late stopping — endless "supplementary" work destroying UX and wasting compute 3. Never stopping — circular dependencies where agents endlessly trigger each other (known in industry as *overthinking* or *token black holes*), currently handled only by hard-coded round caps

    The ideal stop condition is a conjunction:

    \[Stop(t) = \mathbb{1}\left[ Q_{complete}(s_t) \land Q_{quality}(s_t) \land Q_{budget}(s_t) \land Q_{novel}(s_t) \right]\]

    All these terms currently rely on hand-coded heuristics, never RL-trained.

    The academia–industry scale gap

    The survey links public evidence from three industrial systems—Kimi Agent Swarm (Moonshot AI), OpenAI Codex, and Anthropic Claude Code—and finds their training details, reward designs, and stopping mechanisms nearly entirely undisclosed. As the paper states: *"The resulting scale gap is a gap between publicly reported deployment envelopes and open academic evaluation regimes, not independent verification of industrial training traces."* Academia evaluates 2–5 agent toy environments with fixed stopping heuristics; industry deploys hundreds of coordinated agents with dynamic, unpublished stop logic.

    Proposed directions

    1. Explicit RL objective functions for O5, e.g. a value function \(V_{stop}(s_t)\) comparing continue-vs-stop expected returns 2. Recording stop signals (why the system stopped) inside orchestration traces for future learning 3. Cross academia–industry evaluation standards for orchestration quality 4. Message-level credit assignment methods requiring new gradient estimation over discrete message→reward paths

    Paper details

  • Title: Reinforcement Learning for LLM-based Multi-Agent Systems through Orchestration Traces
  • Author: Chenchen Zhang
  • arXiv: 2605.02801 (cs.CL), released May 4, 2026
  • Open artifact: github.com/xxzcc/awesome-llm-mas-rl — 84 annotated papers, 32 exclusion logs (116 reviewed), and a JSON Schema for reproducible orchestration traces
  • Corpus composition: 42 RL methods, 18 benchmarks, 10 classical MARL foundations, 6 industrial reports, 5 surveys, 3 frameworks
  • Six orchestration topologies: centralized orchestrator, planner-executor-critic, debate/committee, parallel swarm, hierarchical, and harness

Tags

#multi-agent-systems#reinforcement-learning#llm#orchestration#credit-assignment#ai-safety#survey#stopping-criteria

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/177619481