2026 FIFA World Cup: 39 days, 104 matches, 4,494 predictions. Six of the world's strongest LLMs—Claude, GPT, Gemini, Kimi, GLM, and Seed—received a detailed pre-match briefing before every game and filled out a seven-item prediction card: winner, score, goal count, group advancement, and more. No answer existed on the internet at prediction time.
This was a carefully designed "leak-free" experiment. Not hindsight analysis—the answers were locked into a frozen archive before kickoff.
The result? The six AIs averaged a 63.9% hit rate.
Sounds good? The problem: simply betting the bookmaker's favorite also yields 63.9%.
An Honest Exam Hall
Forecasting benchmarks have an awkward flaw: most events have already happened, answers sit in some corner of the internet, and models may be "reciting" rather than "predicting." Researchers call this "data leakage."
WorldCup Arena sidesteps this trap by construction. Before each match, each of the six models received a briefing—team form, head-to-head history, injuries, tournament context—and filled out a prediction card. The answer did not exist when the question was asked. This isn't post-hoc filtering of leaked questions; it guarantees nothing to leak.
Over 39 days, the frozen archive accumulated 4,494 timestamped predictions—genuinely forward-looking evidence.
Six Models, One Mold
All six models ran their strongest configurations:
- Claude (claude-opus-4-8 thinking, 24k budget)
- GPT (gpt-5.5 reasoning effort high)
- Gemini (gemini-3.1-pro-preview, thinking budget 24,576)
- Kimi (kimi-k2.6 built-in thinking)
- GLM (glm-5.2 thinking)
- Seed (doubao-seed-2-0-pro reasoning effort high)
- Lopsided matches (strong vs. weak): accurate predictions—but anyone can guess those
- The most evenly matched matches—precisely where briefings were richest—prediction collapse
- Tournament-level predictions (who advances, who wins) were actually answered decently
All ran at maximum reasoning mode, all with native web search—the strongest lineup money could buy in August 2026.
Yet their behavior was strikingly uniform:
1. They all did the same thing—back the favorite. Average accuracy of 63.9% on match outcomes, identical to betting the odds table. Six top AIs combined don't beat one odds sheet.
2. They are too alike. Inter-model agreement far exceeded agreement with the truth. The majority vote of the six offered zero gain—because they voted the same answer anyway. Not six independent judgments forming an ensemble; six repeaters.
3. They collectively avoid draws and low scores. Draws and low-scoring games are football's most common upsets. But the models systematically under-predicted draw probability and total goals, favoring "plausible-looking" scorelines like 2:1 or 3:0 over the boring-but-common 1:1.
4. Score predictions cluster on "prototype outcomes." All six models concentrated their score forecasts on a few combinations, as if cast from the same mold. Real football's score distribution is far more dispersed.
The More Judgment Required, The Harder They Fall
The most counterintuitive finding: accuracy was unrelated to match information content, and related to mismatch size.
Among the Six: Small Gaps, Messy Middle
The leaderboard pattern: stable top and bottom, churning middle.
Across all 39 days, the strongest and weakest models held their positions while mid-table rankings kept shifting. And throughout, score gaps between models were narrow.
This means: on "predicting real-world events," the current generation of frontier models has not genuinely differentiated. They share the same blind spots, the same preferences, the same failure mode. It's not "some models are smarter"—it's "this generation hasn't learned yet."
What This Means
WorldCup Arena delivers an uncomfortable conclusion: the strongest current LLMs are, at their core, sophisticated "odds repeaters" for real-world prediction.
They can read briefings, generate plausible-sounding analysis, and produce self-consistent explanations after the fact. But when the answer doesn't yet exist—when judgment, not recall, is required—they tie the most naive strategy of all: back the favorite.
More alarming is the collective-agreement finding. Six independently-run models produced highly similar predictions—ensembling them yields no diversity benefit. If you hoped "multi-model voting" would improve forecast quality, this experiment says: no, they cast the same ballot.
An Honest Exam, An Honest Answer
WorldCup Arena's value isn't "AI is bad"—it's that it provides a genuinely honest exam. 4,494 predictions, 39 days, zero leakage—currently the cleanest evidence on LLM forecasting ability.
Its message is not "models can't predict" but "the current generation has not yet surpassed the most naive baseline on real-world event prediction." That's a baseline problem, not a capability-ceiling problem. Will the next generation break through? Unknown. But at least we now have an honest ruler.
The researchers have released all briefings, schedules, official results, and scoring code. At the next big tournament, anyone can measure their own model with the same ruler.
---
Paper: https://arxiv.org/abs/2608.04008
Code & data: The paper states the frozen archive and scoring code will be released.