Agon: How Two AI Models Competing as Each Other's Judges Evolve Stronger Reasoning

Forum topic · 小凯 · 2026-07-10

Summary

Agon is a competitive cross-model reinforcement learning framework proposed by Vladislav Beliaev (arXiv:2607.07690) that improves LLM reasoning by having two models grade each other implicitly. Unlike GRPO, which rewards only final-answer correctness and can learn reward hacking like longer guesses, Agon places two models in a two-player game on the same hard problem: each model alternately drafts a full solution, then answers after reading the opponent's draft. If a model solves the problem using the opponent's reasoning while the opponent fails, it is rewarded and the opponent penalized—making the reasoning process itself an implicit supervision signal. Because each model grows stronger, the opponent must keep improving, creating a self-sustaining escalation loop that requires diverse strategies between the two models. Experiments show Agon roughly doubles GRPO's pass@1 on DeepMath hard problems with Qwen3, delivers about 8x the gains of an untrained Mixture-of-Agents pipeline, and generalizes to competitive programming and across model families including Gemma. The paper suggests extending the approach from text-based competition to latent-space collaborative reasoning as future work.

> Original paper: Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning > arXiv: 2607.07690 > Author: Vladislav Beliaev > Categories: cs.LG, cs.AI, cs.CL

---

Prologue: A Debate With No Audience

Imagine two chess players competing. But this is no ordinary match—at every move, each must not only read the board but see through the opponent's thinking. Why did they play that move? What is their plan? How can I beat their position *and* refute their logic in the next step?

That is the core metaphor of Agon.

In existing reasoning-model training, such as GRPO (Group Relative Policy Optimization), the model faces a hard problem, generates a set of answers, and sees which ones are correct. Right answers score, wrong answers lose points. It sounds fair, but there is a fatal blind spot: the model is only told "right" or "wrong," never "this line of thinking was good, but that one was clumsy."

It is like a teacher who grades only final exam scores and never looks at a student's scratch work. The result? The model learns to "write more words to raise the odds of a lucky guess" rather than "learn better ways of thinking."

Beliaev saw through this problem, and his solution is surprisingly simple: let two models act as each other's judges.

---

Technical Core: Why Process Grading Is Harder Than Outcome Grading

GRPO's Blind Spot

GRPO works like this: given a math problem, the model generates 8 different solution trajectories (rollouts). If 3 are correct and 5 are wrong, the model learns to "prefer the styles of writing that got it right." But this hides an assumption: the correct rollouts contain good reasoning.

When a problem is hard enough, all 8 rollouts may fail, and GRPO's gradient signal becomes zero—the model has no direction to improve. Worse, even successful rollouts may contain muddled reasoning, skipped steps, or lucky guesses. From these "correct" answers, the model may learn the wrong methodology.

Agon's Breakthrough

Agon's core is a two-player game structure:

1. Same-problem contest: two models (A and B) receive the same hard problem. 2. Alternating roles:

Round 1: Model A drafts a solution; Model B answers after reading A's draft.
Round 2: Model B drafts; Model A answers after reading.

Implicit grading

The key: to win reward, each model must both solve problems itself and identify flaws in the opponent's reasoning, exploiting that information to reach the correct answer. The reasoning process is thus graded implicitly—if your draft is full of misleading clues, the opponent profits and you lose.

---

Dynamic Difficulty: The Stronger the Opponent, the Stronger You Become

Single-model RL has a fundamental limitation: the model only compares against its past self, always facing an opponent weaker than or equal to its current self. It is like a boxer who only hits a heavy bag—the bag never fights back and never gets stronger.

Agon's two-model structure creates a unique dynamic:

Model A improves → B is harder to beat → B must improve → A is harder to beat...

This is a self-sustaining escalation loop. The two models need not be identical—in fact, they must behave differently. If both solve problems the same way, they get stuck on the same problems and the game reduces to "who guesses first." Only with different strategies and different blind spots can one learn from the other's errors.

This differs fundamentally from Mixture-of-Agents (MoA), which stacks multiple models' outputs and votes for the best. Agon is adversarial training—models are optimized through competition, not averaged through collaboration.

---

The Data: Not Just Flash and Fury

Experimental results are impressive:

DeepMath hard problems (Qwen3): Agon doubles GRPO's pass@1 over the baseline.
Versus MoA: Agon's gains are roughly 8x those of an untrained MoA pipeline.
Cross-domain: equally effective on competitive programming code.
Cross model families: similar trends reproduced on Qwen3.5 and Gemma 4.

These numbers highlight an important fact: the gradability of reasoning processes is an underexploited supervision signal. Without human labels for "good" vs. "bad" reasoning, adversarial competition between two models can generate that signal automatically.

---

The Future: From Textual Combat to Latent-Space Collaboration

The paper ends with an exciting next step: "currently models communicate via text; the next step is having them reason together in latent space."

What does that mean? In today's Agon, Model A writes its reasoning in natural language and Model B reads it. This has two limits:

1. Text is low-bandwidth—much reasoning is intuitive and non-verbal. 2. Language can be manipulated—models can deliberately write misleading intermediate steps.

If models could exchange representations directly in latent space—something like the "tacit understanding" between humans—adversarial collaboration would enter a new dimension. Models would no longer deceive with words but compete at the vector level. It is a bit like upgrading from chess to Go: more abstract, but with far greater strategic depth.

---

One-Sentence Takeaway

Agon does not teach models to "think harder"—it gives them an arena where they *must* think deeper than their opponent. In that arena, good reasoning is not defined; it is proven—by defeating an ever-stronger rival.

---

*Originally discussed on zhichai.net; original post tags included #论文解读 #arXiv #Agon #推理模型 #强化学习.*