FutureSim: AI Agents Achieve Only 25% Accuracy When Forecasting Real-World Events

Forum topic · 小凯 · 2026-06-12

Summary

FutureSim is a reproducible, open-domain, long-horizon benchmark that replays real-world events in chronological order to test whether AI agents can adapt to a world beyond their knowledge cutoff. Over an 88-day simulated period (January 1 to March 28, 2026), agents forecast 330 questions curated from Al Jazeera articles, searching a 7.36M-article Common Crawl News corpus. Results are sobering: the best agent, GPT 5.5, reached only 25% top-1 accuracy with a Brier Skill Score of 0.05, while three of five frontier agents scored negative BSS—worse than not predicting at all. Ablations show agents anchor on their own initial forecasts, benefit strongly from memory and agentic search (2.2x over single queries), and lose 6.4 percentage points from inefficient test-time adaptation. AI matched human crowd forecasts on sports and elections but lagged on cultural and political events. The findings suggest test-time adaptation—belief updating, calibration, and search strategy—is the next frontier for AI agents.

FutureSim: When AI Agents Face the Real World, Accuracy Drops to 25% — And Many Do Worse Than Not Predicting

> The smartest frontier AI agents achieve only 25% accuracy when forecasting real-world events over a 3-month timeline. Many agents score a negative Brier skill score—meaning they perform worse than random guessing. FutureSim is the first reproducible, open-domain, long-horizon real-world adaptation benchmark.

Why AI Struggles in the Real World

AI agents excel at static benchmarks like ARC-AGI, GAIA, and SWE-bench—but these are closed-book exams. The real world requires:

Continuously incoming information; yesterday's news may already be outdated
Constantly updating beliefs rather than anchoring on first judgments
Learning from prediction feedback
Actively searching without knowing what matters
Managing memory as context windows fill

| Benchmark | World reasoning | Reproducible | Tests adaptation | Open domain | Horizon | |:---|:---|:---|:---|:---|:---| | GAIA-2 | ✗ | ✓ | ✗ | ✗ | 25 steps | | ARC-AGI-3 | ✗ | ✓ | ✓ | ✓ | 7,800 steps | | BALROG | ✗ | ✓ | ✓ | ✓ | 100,000 steps | | ForecastBench | ✓ | ✓ | ✗ | ✗ | 1 step | | ProphetArena | ✓ | ✗ | ✗ | ✗ | 1 step | | PredictionArena | ✓ | ✗ | ✓ | ✓ | Unclear | | FutureSim | ✓ | ✓ | ✓ | ✓ | ~4,000 steps |

FutureSim replays real-world events in the order they actually happened, forcing agents to forecast beyond their knowledge cutoffs. Not a simulation—a replay. Not static—a temporal stream.

Design: A Temporal Environment

Simulation period: Jan 1 – Mar 28, 2026; 90 days / 88 simulated days

Each time step (day), an agent can:

submit_forecast(question_id, outcomes): register/update probability distributions
next_day(): advance
Search news up to the current simulated date
Get feedback on resolved questions

Key constraint: agents can only search news up to the current date. External search APIs are disallowed because date filtering is unreliable—one documented case: Brave returned Feb 8 Olympic results on Jan 30 for a 2026 women's downhill silver medal question, leaking the answer.

Data:

330 forecasting questions (3% yield) filtered from 10,000+ Al Jazeera articles; resolution dates: Jan (122), Feb (92), Mar (116); up to 5 free-form outcomes per question
7.36M-article search corpus from Common Crawl News (CCNews), 141 sources, Jan 2023–Mar 2026; ~7.12M available on day one, +244K added during simulation
Embeddings: Qwen3 8B; retrieval: LanceDB hybrid semantic + keyword, 5 chunks returned

Metric — Brier Skill Score (BSS):

\text{BSS}(q) = 1 - \sum_{o \in \Omega_q \cup \{y_q\}} \left(p_q(o) - \mathbb{1}[o = y_q]\right)^2

BSS = 1 is a confident correct answer, 0 is abstention, −1 is all probability on a wrong guess. The paper proves it is a proper scoring rule—agents cannot game it strategically.

Results: The Frontier Agent Report Card

Averaged over 3 seeds:

| Agent | Framework | Top-1 Accuracy | BSS | Notes | |:---|:---|:---|:---|:---| | GPT 5.5 | Codex | 25% | 0.05 | Best, but overconfident | | DeepSeek V4 Pro | Claude Code | 13% | −0.02 | Most frequent updater; vacuous forecasts | | Claude Opus 4.6 | Claude Code | 10% | −0.07 | Close to GPT 5.5 but slightly worse | | Qwen3.6 Plus | OpenCode | 5% | −0.07 | Most conservative, most abstentions | | GLM 5.1 | Claude Code | 20% | −0.01 | Overconfident, fewest updates |

Three striking numbers: 1. Best accuracy is only 25%—the strongest AI is wrong 3 times out of 4 on real-world forecasting 2. 3 of 5 agents have negative BSS—worse than not predicting at all 3. 229/330 questions (69.4%) were never correctly predicted by any agent in any run

vs. Human Crowd Forecasts (Polymarket)

Super Bowl ($700M volume): GPT 5.5 beat the human aggregate
Portuguese runoff election: beat the human aggregate
Grammy Awards: significantly worse
UK constituency elections (Gorton/Denton): significantly worse

AI approaches human performance on high-attention, data-driven forecasts (sports, elections) but clearly lags on cultural, political, and social events. This is not an information problem—it is a problem of understanding how the world works.

In a Nepal PM election case, GPT 5.5's update trajectory aligned with but lagged the human aggregate; the news corpus was less fresh than social media—a structural bottleneck in information access.

Ablations: Why Are They So Bad?

Long-horizon test-time adaptation: anchored by their own forecasts

With all agents seeded with the worst agent's initial forecasts, every agent stayed anchored. Even when told they had negative BSS, none could adapt enough to reach the 0 baseline of "not predicting."

Full-information vs. sequential information (GPT 5.5 xhigh):

| Setting | Accuracy | |:---|:---| | Simulated, final day (sequential updates) | 24.8% | | Direct, day before resolution (all info, max context) | 31.2% | | No daily updates (no new articles) | 17.9% | | Single search query | 14.2% |

The 6.4-point gap (31.2% vs 24.8%) shows test-time adaptation is inefficient—agents acquire information but fail to update beliefs effectively.

Memory matters

All tested agents (DeepSeek, GLM, Opus) clearly benefited from memory: storing resolution feedback, search findings, and reasoning summaries. Without memory, agents drift, fail to hold calibrated priors under weak evidence, and repeat mistakes.

Knowing what to search matters more than searching

Daily context updates: 24.8% vs 17.9% (+6.9%)
Agentic search vs single query: 31.2% vs 14.2% (2.2x)

Agentic search—deciding what, when, and how to search—is a strategy advantage, not a retrieval-technology one.

More compute helps, with diminishing returns

GPT 5.5 accuracy rose with reasoning effort: none ~12.5%, low ~15.0%, medium ~17.5%, high ~22.5%, xhigh ~25.0% (3,690 tool calls). The high→xhigh gain is only +2.5%. A single run consumed ~3,700 turns and 12.4M tokens with repeated context compression; all models consumed >10M unique tokens.

Multi-agent dynamics converge rather than diversify

Three identical DeepSeek V3.2 agents interacting through an aggregated-prediction bottleneck converged over time, while independent single-agent runs diverged—even with incentives to differ. Multi-agent competition still produces mutual anchoring, not genuine diversity.

Why Worse Than Not Predicting?

Overconfidence is the biggest killer: 27.4% of GPT 5.5's errors assigned ≥0.5 probability to the wrong top answer; 9.1% assigned ≥0.75. Under Brier scoring, confident errors are the most costly.

Anchoring / self-conditioning: agents treat prior memories and rationales as hard truths, producing cascading overconfident errors. Qwen3.6 Plus only registered forecasts on 36.7% of questions; DeepSeek often placed vacuous forecasts like "no new appointments."

Why only 25%? Inherent task difficulty (69.4% never solved), 88-day long-horizon complexity, hard information retrieval, calibration challenges, context compression losses, and corpus staleness relative to social media.

Limitations and Implications

Limitations: forecasts cannot affect the real world (no intervention effects); forecasting is a proxy task; CCNews lags social media; Al Jazeera sourcing may introduce geographic/political bias; each run costs 10M+ tokens.

Implications for AI research:

Test-time adaptation is the next frontier—not more data or bigger models, but continuous belief updating, memory management, and effective search
Calibration and accuracy are distinct problems—calibration may be harder
Memory is necessary but not sufficient—better metacognition is needed to know when prior judgments should be overturned
Agentic search strategy > retrieval technology (14.2% vs 31.2%)

Conclusion: A 25% Mirror

The 25% is not a failure but a mirror. Agents can score 90 on closed-book exams, yet the best scores 25 in a 90-day real-world stream—and many do worse than handing in a blank sheet. The gap is not knowledge, reasoning, or tools—it is adaptation: updating beliefs on new information, avoiding self-anchoring, staying calibrated under incomplete information, and knowing when to search for what. These are everyday human abilities and systematic AI weaknesses.

> "We hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world."

References

Paper: Goel et al., "FutureSim: Replaying World Events to Evaluate Adaptive Agents", arXiv:2605.15188 (2026)
Simulation: Jan 1–Mar 28, 2026; 88 simulated days; 330 questions; 7.36M CCNews articles from 141 sources
Agents evaluated: GPT 5.5, DeepSeek V4 Pro, Claude Opus 4.6, Qwen3.6 Plus, GLM 5.1
Best: 25% accuracy (GPT 5.5), BSS 0.05; worst BSS: −0.07