English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

WorldCup Arena: Six Top AI Models Tied a Whole World Cup Forecast Against Betting Odds

Forum topic · ✨步子哥 · 2026-08-05

Summary

WorldCup Arena is a leak-free benchmark that evaluated six frontier LLMs—Claude, GPT, Gemini, Kimi, GLM, and Seed—across the 2026 FIFA World Cup: 39 days, 104 matches, and 4,494 pre-match predictions locked in a frozen archive before kickoff. Each model received a detailed pre-match briefing and submitted a seven-item prediction card covering match outcome, score, goals, and group advancement. The average accuracy was 63.9%—exactly matching the naive baseline of always betting the bookmaker's favorite. Key findings: models showed high mutual agreement with no ensemble benefit; they systematically under-predicted draws and low-scoring games; score forecasts clustered on a few prototypical results; and accuracy was unrelated to information richness, collapsing precisely on the most evenly matched matches. The study concludes that current frontier LLMs function as sophisticated 'odds repeaters' on real-world event prediction, and the researchers have released briefings, schedules, official results, and scoring code for future evaluation. Paper: https://arxiv.org/abs/2608.04008

2026 FIFA World Cup: 39 days, 104 matches, 4,494 predictions. Six of the world's strongest LLMs—Claude, GPT, Gemini, Kimi, GLM, and Seed—received a detailed pre-match briefing before every game and filled out a seven-item prediction card: winner, score, goal count, group advancement, and more. No answer existed on the internet at prediction time.

This was a carefully designed "leak-free" experiment. Not hindsight analysis—the answers were locked into a frozen archive before kickoff.

The result? The six AIs averaged a 63.9% hit rate.

Sounds good? The problem: simply betting the bookmaker's favorite also yields 63.9%.

An Honest Exam Hall

Forecasting benchmarks have an awkward flaw: most events have already happened, answers sit in some corner of the internet, and models may be "reciting" rather than "predicting." Researchers call this "data leakage."

WorldCup Arena sidesteps this trap by construction. Before each match, each of the six models received a briefing—team form, head-to-head history, injuries, tournament context—and filled out a prediction card. The answer did not exist when the question was asked. This isn't post-hoc filtering of leaked questions; it guarantees nothing to leak.

Over 39 days, the frozen archive accumulated 4,494 timestamped predictions—genuinely forward-looking evidence.

Six Models, One Mold

All six models ran their strongest configurations:

  • Claude (claude-opus-4-8 thinking, 24k budget)
  • GPT (gpt-5.5 reasoning effort high)
  • Gemini (gemini-3.1-pro-preview, thinking budget 24,576)
  • Kimi (kimi-k2.6 built-in thinking)
  • GLM (glm-5.2 thinking)
  • Seed (doubao-seed-2-0-pro reasoning effort high)
  • All ran at maximum reasoning mode, all with native web search—the strongest lineup money could buy in August 2026.

    Yet their behavior was strikingly uniform:

    1. They all did the same thing—back the favorite. Average accuracy of 63.9% on match outcomes, identical to betting the odds table. Six top AIs combined don't beat one odds sheet.

    2. They are too alike. Inter-model agreement far exceeded agreement with the truth. The majority vote of the six offered zero gain—because they voted the same answer anyway. Not six independent judgments forming an ensemble; six repeaters.

    3. They collectively avoid draws and low scores. Draws and low-scoring games are football's most common upsets. But the models systematically under-predicted draw probability and total goals, favoring "plausible-looking" scorelines like 2:1 or 3:0 over the boring-but-common 1:1.

    4. Score predictions cluster on "prototype outcomes." All six models concentrated their score forecasts on a few combinations, as if cast from the same mold. Real football's score distribution is far more dispersed.

    The More Judgment Required, The Harder They Fall

    The most counterintuitive finding: accuracy was unrelated to match information content, and related to mismatch size.

  • Lopsided matches (strong vs. weak): accurate predictions—but anyone can guess those
  • The most evenly matched matches—precisely where briefings were richest—prediction collapse
  • Tournament-level predictions (who advances, who wins) were actually answered decently
In other words, the models completely failed at "more information → better prediction." The matches with the thickest dossiers were where they fell hardest—directly challenging the intuition that more information means better decisions.

Among the Six: Small Gaps, Messy Middle

The leaderboard pattern: stable top and bottom, churning middle.

Across all 39 days, the strongest and weakest models held their positions while mid-table rankings kept shifting. And throughout, score gaps between models were narrow.

This means: on "predicting real-world events," the current generation of frontier models has not genuinely differentiated. They share the same blind spots, the same preferences, the same failure mode. It's not "some models are smarter"—it's "this generation hasn't learned yet."

What This Means

WorldCup Arena delivers an uncomfortable conclusion: the strongest current LLMs are, at their core, sophisticated "odds repeaters" for real-world prediction.

They can read briefings, generate plausible-sounding analysis, and produce self-consistent explanations after the fact. But when the answer doesn't yet exist—when judgment, not recall, is required—they tie the most naive strategy of all: back the favorite.

More alarming is the collective-agreement finding. Six independently-run models produced highly similar predictions—ensembling them yields no diversity benefit. If you hoped "multi-model voting" would improve forecast quality, this experiment says: no, they cast the same ballot.

An Honest Exam, An Honest Answer

WorldCup Arena's value isn't "AI is bad"—it's that it provides a genuinely honest exam. 4,494 predictions, 39 days, zero leakage—currently the cleanest evidence on LLM forecasting ability.

Its message is not "models can't predict" but "the current generation has not yet surpassed the most naive baseline on real-world event prediction." That's a baseline problem, not a capability-ceiling problem. Will the next generation break through? Unknown. But at least we now have an honest ruler.

The researchers have released all briefings, schedules, official results, and scoring code. At the next big tournament, anyone can measure their own model with the same ruler.

---

Paper: https://arxiv.org/abs/2608.04008

Code & data: The paper states the frozen archive and scoring code will be released.

Tags

#llm-benchmark#world-cup-2026#ai-prediction#data-leakage#forecasting#claude#gpt#gemini

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178595031