Agon: When Two AIs Become Each Other's Judges — How Reasoning Evolves Through Adversarial Competition

> Original paper: Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning > arXiv: 2607.07690 > Author: Vladislav Beliaev > Categories: cs.LG, cs.AI, cs.CL

---

🎭 Prologue: A Debate with No Audience

Imagine two chess players facing each other. But this is not an ordinary match — at every move, they must not only read the board but also see through the opponent's mind. Why did the opponent make that move? What is their plan? How can I beat their position *and* refute their logic on the next move?

This is the core metaphor of Agon.

In current training of reasoning models, such as GRPO (Group Relative Policy Optimization), the model faces a hard problem, generates a set of answers, and checks which ones are correct. Correct answers earn reward, wrong ones lose it. That sounds fair, but it has a fatal blind spot: the model is only told "right" or "wrong" — it is never told "this line of thinking was good, but that one was dumb."

It's like a teacher who only grades final exam scores and never looks at the students' scratch paper. The result? The model learns to "write more words to raise the odds of guessing right," rather than "learn better ways of thinking."

Beliaev saw through this problem, and his solution is surprisingly simple: let two models grade each other.

---

🧠 The Technical Core: Why Process Grading Is Harder Than Outcome Grading

GRPO's Blind Spot

How GRPO works: given a math problem, the model generates 8 different solution processes (rollouts). If 3 answers are correct and 5 are wrong, the model learns to "favor the phrasing of the correct ones." But this hides an assumption: the correct rollouts contain good reasoning.

When a problem is hard enough, all 8 rollouts may fail — at which point GRPO's gradient signal is zero, and the model has no direction to improve in. Worse, even rollouts that answer correctly may contain confused reasoning, skipped steps, or sheer luck. What the model learns from these "correct" answers may be a flawed methodology.

Agon's Breakthrough

Agon's core is a two-player game structure:

1. Same problem, competing: Two models (A and B) receive the same hard problem 2. Alternating roles:

• Round 1: Model A drafts a solution process; Model B answers after reading A's process
• Round 2: Model B drafts; Model A answers after reading
3. Implicit grading: If B, after reading A's process, answers correctly while A fails, it means B not only understood the problem but found the right path from A's attempt. B is rewarded; A is penalized.

The key: to win reward, each model must not only solve problems itself, but also spot flaws in the opponent's reasoning and use that information to reach the correct answer. The reasoning process itself is thus graded implicitly — because if your draft is full of misleading clues, your opponent benefits from it and you lose.

---

🔄 Dynamic Difficulty: As the Opponent Grows Stronger, So Do You

Single-model RL has a fundamental limitation: the model can only compare against its past self, always facing an opponent that is "weaker than or equal to" the current self. It's a boxer who only hits a punching bag — the bag never hits back and never gets stronger.

Agon's dual-model structure creates a unique dynamic:

• Model A improves → Model B finds it harder to win → Model B must improve → Model A finds it harder to win...

This is a self-sustaining escalation loop. The two models don't have to be identical — in fact, they need to behave differently. If both models solve problems the same way, they get stuck on the same problems simultaneously and the game becomes "who guesses right first." Only when they have different strategies and different blind spots can one learn from the other's mistakes.

This fundamentally differs from Mixture-of-Agents (MoA). MoA stacks multiple models' outputs and votes for the best. Agon is adversarial training — models are optimized through competition, not averaged through collaboration.

---

📊 The Numbers: Why This Isn't Just a Fancy Trick

The experimental results are impressive:

• DeepMath hard problems (Qwen3): Agon doubles GRPO's pass@1 over the baseline
• Versus MoA: Agon's gains are roughly 8x those of an untrained MoA traversal
• Cross-domain validation: equally effective on competitive programming code
• Cross model families: similar trends reproduced on Qwen3.5 and Gemma 4

These results highlight an important fact: the gradability of reasoning processes is an underexploited supervision signal. Without any human labels marking "good reasoning" and "bad reasoning," adversarial competition between two models can automatically generate that signal.

---

🌌 Looking Ahead: From Text Adversaries to Latent-Space Collaboration

The paper closes with an exciting next step: "Currently the models communicate in text; the next step is having them reason together in latent space."

What does that mean?

In today's Agon, Model A writes its reasoning in natural language and Model B reads it. This has two limitations: 1. Text is low-bandwidth — much reasoning is intuitive and non-verbal 2. Language can be manipulated — a model can deliberately write misleading intermediate steps

If models could exchange representations directly in latent space — something like the "chemistry" or "intuition" between humans — adversarial and cooperative reasoning would enter a whole new dimension. Models would no longer deceive opponents with words but compete directly at the vector level. It's a bit like going from chess to Go: the latter is more abstract, but its strategic depth far exceeds the former.

---

💡 One-Sentence Takeaway

Agon doesn't teach a model to "think harder" — it gives it an arena where it *must* think deeper than its opponent. In that arena, good reasoning isn't defined; it's proven — by defeating an ever-stronger rival.

---

*Translated and edited from a Chinese forum post on zhichai.net.*