When Does Self-Play RL Collapse? The Threshold Is Exactly Zero

A question that has troubled multi-agent researchers for years: when does a self-play agent collapse?

Not gradual performance decay—catastrophic collapse, where agents stop learning, stop adapting, and lock into a fixed, repetitive, near-maximal-loss pattern like a broken machine. A 2026 paper by Arahan Kujur gives a precise, binary mathematical answer: the threshold is zero.

• Paper: *A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning*
• Author: Arahan Kujur
• arXiv: 2605.16315 (cs.LG, cs.AI), submitted May 2026

1. What Is Self-Play Collapse?

Self-play underlies AlphaGo, AlphaZero, and OpenAI Five: two agents compete and improve together across countless rounds. But under certain conditions, agents don't get stronger—they suddenly collapse. Both agents stop learning and lock into a fixed behavioral pattern that loses at near-maximal loss. Researchers have observed this for years across games and algorithms, but no one had pinpointed what triggers it. This paper claims to have found the answer.

2. Reach-Weighted Contingent Action Capacity (CAC_w)

Kujur introduces reach-weighted contingent action capacity (CAC_w). Some decisions are neutral—choices barely matter. Others are contingent—choices significantly affect outcomes. A contingent decision counts toward CAC_w only if it lies in positively reachable states, i.e., states the agent can actually encounter.

CAC_w = 0 means: in every state you can actually reach, no choice you make substantially affects the outcome. Like playing chess with a rule that your king is captured no matter what move you make.

3. A Precisely Binary Threshold

Across five algorithms (Q-learning, PPO, NFSP, neural NFSP, DQN) and multiple games (poker variants, matrix games, dice games), the result is consistent:

• CAC_w > 0: agents play normally, policies keep evolving, no collapse.
• CAC_w = 0: agents rapidly converge to a deterministic exploitation attractor—a fixed policy fixed point with near-maximal loss.

Crucially, the threshold is binary. It's not that smaller CAC_w means worse collapse—there is a clear wall. Even a CAC_w as tiny as 1 or 2 keeps the system safe. In one extreme validation experiment, decision options were removed from a poker game one by one until a single contingent decision point remained. As long as that point existed, the system did not collapse. Delete it—CAC_w = 0—and the system immediately falls into the exploitation attractor.

4. Why Does Collapse Happen? Mechanism Analysis

Two controlled experiments illuminate the mechanism:

• Frozen-baseline control: keeping one agent's parameters fixed while the other self-plays. If collapse were caused by the perturbation itself, the frozen baseline would also collapse. It did not—showing collapse results from the self-play dynamics, not the perturbation.
• Recovery experiment: restoring all decision points to a collapsed system immediately restores normal self-play learning.

Together these point to collapse as co-adaptation under constraints: deprived of all positively reachable contingent decisions, the agents' mutual adaptation no longer produces exploratory behavior; instead both converge to the same losing policy. The agents effectively teach each other to give up. When every decision is zero-sum, inescapable, and indifferent, the game loses its value as a learning signal.

5. Honest Questions

1. Is the threshold exactly zero? CAC_w values in the experiments are discrete (discrete action spaces). In continuous action spaces the threshold may be more complex—perhaps triggering below some small positive value. The paper does not test continuous spaces. 2. Dependence on game structure. The paper tests a specific asymmetric perturbation—removing decision options. Real multi-agent systems may degrade CAC_w gradually and distributively; that regime is unmodeled. 3. Relation to the companion paper. Kujur simultaneously submitted "When Actions Disappear" (arXiv:2605.16312) on adversarial action-removal attacks. Both share the CAC_w concept but study different phenomena; the author of this post questions whether they should be unified.

6. Verdict

Most ML findings are fuzzy—"bigger models are better," "more data is better." This paper instead claims: the threshold is zero. Exact zero. That rarity makes it valuable: it is verifiable and falsifiable. If future replications on other games or algorithms find a nonzero positive threshold, the conclusion would need revision.

The engineering implication is direct: when designing a multi-agent system, ensure every agent retains at least one positively reachable contingent decision. Depriving an agent of meaningful choices doesn't make it more predictable—it breaks it. Choice is not merely an expression of autonomy; choice is a structural precondition for system stability.

References

1. Kujur, A. (2026). A Structural Threshold in Decision Capacity Governs Collapse in Self-Play Reinforcement Learning. arXiv:2605.16315. 2. Kujur, A. (2026). When Actions Disappear: Adversarial Action Removal in Self-Play Reinforcement Learning. arXiv:2605.16312. 3. Silver, D., et al. (2018). A General Reinforcement Learning Algorithm That Masters Chess, Shogi, and Go Through Self-Play. Science. 4. Heinrich, J., Silver, D. (2016). Deep Reinforcement Learning from Self-Play in Imperfect-Information Games. NeurIPS.