FutureSim: AI Agents Only Reach 25% Accuracy When Forecasting Real-World Events Over 88 Days

> The smartest frontier AI agents achieve only 25% accuracy when forecasting real-world events across a 3-month timeline. Many agents score a negative Brier Skill Score—meaning they are worse than not predicting at all. FutureSim is the first reproducible, open-domain, long-horizon real-world adaptation benchmark.

1. The Problem: Why Do AI Agents Falter in the Real World?

AI agents excel at static benchmarks—ARC-AGI, GAIA, SWE-bench—essentially closed-book exams where agents can search, reason, and call tools before giving one answer.

But the real world is not a closed-book exam. It requires:

• Continuously updating beliefs as new information streams in
• Learning from prediction feedback
• Actively deciding what information matters and searching for it
• Managing memory as context windows fill

Existing benchmarks cannot measure these capabilities.

| Benchmark | World reasoning | Reproducible | Tests adaptation | Open-domain | Horizon | |:---|:---|:---|:---|:---|:---| | GAIA-2 | ✗ | ✓ | ✗ | ✗ | 25 steps | | ARC-AGI-3 | ✗ | ✓ | ✓ | ✓ | 7,800 steps | | BALROG | ✗ | ✓ | ✓ | ✓ | 100,000 steps | | ForecastBench | ✓ | ✓ | ✗ | ✗ | 1 step | | ProphetArena | ✓ | ✗ | ✗ | ✗ | 1 step | | PredictionArena | ✓ | ✗ | ✓ | ✓ | Uncertain | | FutureSim | ✓ | ✓ | ✓ | ✓ | ~4,000 steps |

FutureSim's uniqueness: it replays real-world events in the exact chronological order they occurred, forcing agents to predict events beyond their knowledge cutoff. Not a simulation—a replay. Not static—a temporal stream.

2. FutureSim Design: A Temporal Environment

2.1 Core Mechanism

Simulation period: January 1 to March 28, 2026—90 days / 88 simulated days.

Actions available each time step (one day):

• submit_forecast(question_id, outcomes): register/update probability distributions on active questions
• next_day(): advance to the next day
• Search news up to the current simulated date
• Retrieve feedback on resolved questions

Key constraint: Agents can only search news up to the current date. Future information access is strictly blocked. The environment is minimal—you supply your own model, tools, and timestamped event data.

2.2 Data Sources

Forecasting questions (330):

• Curated from 10,000+ Al Jazeera articles down to 330 high-quality questions (3%)
• Resolution dates: January (122), February (92), March (116)
• Up to 5 free-form outcomes; agents propose their own probability distributions

Search corpus (7.36M articles):

• Common Crawl News (CCNews), 141 news sources
• Range: January 2023 to March 2026; ~7.12M articles available on day one, 244K added during the simulation
• Embedding model: Qwen3 8B (trained mid-2025, no recent-world knowledge)
• Retrieval: LanceDB hybrid semantic + keyword search, 5 chunks returned

No external search APIs: Brave-style APIs have unreliable date filtering and leak future information. The paper reports one case where, on January 30, Brave returned the February 8 results of a 2026 Winter Olympics women's downhill event—a direct leak.

2.3 Evaluation Metric: Brier Skill Score

\[\text{BSS}(q) = 1 - \sum_{o \in \Omega_q \cup \{y_q\}} \left(p_q(o) - \mathbb{1}[o = y_q]\right)^2\]

| BSS | Meaning | |:---|:---| | 1 | Fully confident correct answer | | 0 | Abstention (no probability assigned) | | -1 | All probability on a wrong guess |

The paper proves this is a proper scoring rule—agents cannot game it through strategic betting.

3. Results: The World-Forecasting Report Card

3.1 Main Results (averaged over 3 seeds)

| Agent | Framework | Top-1 Accuracy | BSS | Notes | |:---|:---|:---|:---|:---| | GPT 5.5 | Codex | 25% | 0.05 | Best, but overconfident | | DeepSeek V4 Pro | Claude Code | 13% | -0.02 | Most frequent updater; empty forecasts | | Claude Opus 4.6 | Claude Code | 10% | -0.07 | Close to GPT 5.5 but slightly worse | | Qwen3.6 Plus | OpenCode | 5% | -0.07 | Most conservative, most abstentions | | GLM 5.1 | Claude Code | 20% | -0.01 | Overconfident, fewest updates |

Three striking numbers: 1. Best accuracy is only 25%—even the strongest AI gets 3 out of 4 real-world predictions wrong 2. 3 of 5 agents have negative BSS—worse than not forecasting 3. 229 of 330 questions (69.4%) were never correctly predicted by any agent in any run

3.2 Comparison with Human Crowd Forecasts

Against Polymarket human aggregates:

| Market | Volume | GPT 5.5 | |:---|:---|:---| | Super Bowl | $700M | Ahead of humans | | Portugal runoff election | High | Ahead of humans | | Grammy Awards | Medium | Significantly worse | | UK constituency elections (Gorton/Denton) | Medium | Significantly worse |

Takeaway: AI approaches human performance on high-attention, data-driven forecasts (sports, elections) but clearly lags on cultural, political, and social events. This is not an information problem—it's a problem of understanding how the world works.

Nepal PM election case: GPT 5.5's update trajectory aligned with human aggregates but lagged. It cited reasonable evidence, but the search corpus is less fresh than social media—information access channels are a structural bottleneck.

4. Ablations: Why Are They So Bad?

4.1 Long-Horizon Test-Time Adaptation: Anchored by Their Own Forecasts

Experiment: Fixing all agents' initial forecasts to the worst agent's (Qwen3.6 Plus) forecast set to maximize improvement room.

Result: All agents anchored on their initial forecasts. Even when told they had a negative BSS, none adapted enough to reach the 0 baseline of not predicting.

| Agent | Initial BSS (fixed worst) | Final BSS | Reached 0 baseline? | |:---|:---|:---|:---| | GPT 5.5 | ~-0.07 | ~-0.02 | No | | Claude Opus 4.6 | ~-0.07 | ~-0.02 | No | | DeepSeek V4 Pro | ~-0.07 | ~-0.02 | No | | GLM 5.1 | ~-0.07 | ~-0.04 | No | | Qwen3.6 Plus | ~-0.07 | ~-0.06 | No |

Full-information vs. sequential-information:

| Setting | GPT 5.5 xhigh Accuracy | Notes | |:---|:---|:---| | Simulation-final day (sequential updates) | 24.8% | Standard FutureSim | | Direct-day-before-resolution (full info) | 31.2% | Single question, max context, all available info | | No daily updates (no new articles) | 17.9% | No continuous information inflow | | Single search query | 14.2% | Search using only question title |

The gap: 31.2% vs. 24.8% = a 6.4-point efficiency loss. Agents' test-time adaptation is inefficient—even when information is accessible, they don't know how to update beliefs effectively.

4.2 Memory: Forgetting Is Fatal

All ablated agents (DeepSeek, GLM, Opus) clearly benefited from memory, which stored:

• Post-resolution feedback ("I was wrong last time—why?")
• Search findings
• Summaries of past reasoning

Without memory: agents drift more, cannot maintain calibrated priors under weak evidence, and repeat mistakes.

4.3 Search: Knowing What to Search Matters More Than Searching

| Comparison | Settings | Accuracy | Finding | |:---|:---|:---|:---| | Daily context updates | With vs. without | 24.8% vs. 17.9% | +6.9%—continuous fresh evidence is crucial | | Agentic vs. single query | Full agentic vs. single semantic search | 31.2% vs. 14.2% | +17% (2.2x)—sequential information-seeking is crucial |

Agentic search—deciding autonomously what to search, when, and how to combine—is 2.2x stronger than single queries. This is an information strategy problem, not a retrieval technology problem.

4.4 Reasoning Scale: More Compute Helps, With Diminishing Returns

GPT 5.5 across reasoning-effort levels:

| Effort | Tool calls | Accuracy | |:---|:---|:---| | none | 2,603 | ~12.5% | | low | 2,419 | ~15.0% | | medium | 2,568 | ~17.5% | | high | 3,572 | ~22.5% | | xhigh | 3,690 | ~25.0% |

More reasoning compute improves accuracy, but gains shrink from high to xhigh (+2.5 points).

Resource consumption: GPT 5.5 used 3,700 turns, 12.4M tokens, with multiple sequential context-window compressions per run. All models consumed over 10M unique tokens and made 500–4,000+ tool calls.

4.5 Multi-Agent Dynamics: Convergence, Not Diversity

Experiment: Three identical DeepSeek V3.2 agents competing simultaneously, influencing each other through a current-aggregate-forecast information bottleneck.

Findings:

• Multi-agent runs: forecasts converged to similar values over time
• Independent single-agent runs: forecasts diverged
• Despite being prompted that they'd be scored on a "peer score" incentivizing informative forecasts distinct from the aggregate

Even when designed as competitors, agents anchor on each other and fail to produce genuine diversity. Absolute performance was similar between single- and multi-agent setups.

5. Why Worse Than Not Predicting?

5.1 Overconfidence: The Biggest Killer

Of GPT 5.5's wrong predictions:

• 27.4% assigned ≥0.5 probability to the wrong top answer
• 9.1% assigned ≥0.75 probability to the wrong top answer

Agents are not just wrong—they are highly confident in their errors. Under Brier scoring, high-confidence mistakes are the most damaging, penalized more than random uniform guessing.

5.2 Anchoring: Held Hostage by Their Own Forecasts

• Self-conditioning: agents treat prior memories and rationales as "hard truths," causing downstream overconfident errors (Sinha et al., 2026)
• Conservatism/abstention: Qwen3.6 Plus registered forecasts for only 36.7% of questions; GLM 5.1 updated least
• Ineffective forecasts: DeepSeek V4 Pro often placed empty forecasts like "no new appointments," lowering scores

5.3 Why Only 25%?

| Factor | Explanation | |:---|:---| | Intrinsic task difficulty | 69.4% of questions were never predicted correctly by any agent | | Long-horizon complexity | 88 days of continuous updating, memory management, context compression | | Information retrieval challenge | Requires creative reasoning about "what to search"; evidence is scattered | | Probability calibration difficulty | Accurate forecasts vs. well-calibrated confidence are distinct challenges | | Context window limits | Multiple sequential compressions, information loss | | Corpus freshness | CCNews lags social media; real-time developments are missed |

This is not about agents being "not smart enough." It's the inherent difficulty of real-world adaptation—the world doesn't run according to your benchmark.

6. Limitations and Implications

6.1 FutureSim's Limitations

1. Simulation, not live deployment: agent forecasts cannot change the real world (avoiding intervention effects), but the benchmark can't test agents' impact on the world 2. Forecasting is a proxy task: not the real goal of agent deployment, but an ideal testbed for open-domain adaptation 3. Corpus freshness: CCNews lags social media, possibly underestimating agent performance in live information environments 4. Question generation quality: 330 questions filtered from 10,000+ Al Jazeera articles may introduce geographic/political bias 5. Resource cost: a single run consumes 10M+ tokens, making replication expensive

6.2 Implications for AI Research

Test-time adaptation is the next frontier:

• Not more training data, not bigger models—but how agents continuously update beliefs, manage memory, and search effectively during inference
• The 6.4-point efficiency loss (full information vs. sequential updates) reveals a huge gap between accessing information and using it well

Calibration and accuracy are different problems:

• Agents can be decent on accuracy but poor on Brier score due to overconfidence; calibration may be harder than accuracy

Memory is necessary but not sufficient:

• Memory helps, but doesn't solve anchoring and self-conditioning; better "metacognition"—knowing when prior judgments should be overturned—is needed

Agentic search is a core competency:

• Knowing what to search, when, and how to synthesize matters more than retrieval technology itself; the 2.2x gap (14.2% vs. 31.2%) shows strategy > technique

7. Conclusion: 25% as a Mirror

FutureSim's 25% is not failure—it's a mirror, reflecting the true capability boundary of today's frontier AI agents in real-world adaptation.

On closed-book exams, agents score 90. In a 90-day real-world temporal stream, the best score 25—and many do worse than handing in a blank sheet.

Where is the gap?

• Not knowledge (LLM knowledge cutoffs are fixed, but search can supplement)
• Not reasoning (GPT 5.5 is strong on reasoning benchmarks)
• Not tools (all agents had full shell and search tools)

The gap is adaptation: how to update beliefs when new information arrives, how to avoid anchoring on prior judgments, how to stay calibrated under incomplete information, and how to know when and what to search.

These are everyday human capabilities—and systematic AI weaknesses.

> "We hope our benchmark design paves the way to measure AI progress on open-ended adaptation spanning long time-horizons in the real world."

FutureSim is not an endpoint. It is the first map—showing how far we are from AI that can adapt in the real world over long horizons.

---

References

• Paper: Goel et al., "FutureSim: Replaying World Events to Evaluate Adaptive Agents", arXiv:2605.15188 (2026)
• Simulation period: January 1 – March 28, 2026, 88 simulated days
• 330 questions curated from 10,000+ Al Jazeera articles
• Corpus: 7.36M CCNews articles from 141 news sources
• Agents evaluated: GPT 5.5, DeepSeek V4 Pro, Claude Opus 4.6, Qwen3.6 Plus, GLM 5.1
• Best accuracy: 25% (GPT 5.5), BSS 0.05
• Worst BSS: -0.07 (Opus 4.6, Qwen3.6 Plus)