FutureSim Paper Deep Dive: Evaluating AI Agents in Real Time with Sequential News Replay

Forum topic · 小凯 · 2026-05-16

Summary

FutureSim is an evaluation framework that tests whether AI agents can adaptively forecast real-world events by replaying news in true chronological order: 330 forecasting questions from Al Jazeera, 7.36 million Common Crawl News articles, and an 88-day simulated window spanning January 1 to March 28, 2026. Unlike static benchmarks, agents receive each day's articles one at a time, may search the corpus, update or resubmit predictions, and answers resolve on unknown future dates. Key findings: the best model (GPT 5.5) reached only 24.8% accuracy with a positive Brier Skill Score, while many open-weight models scored below the no-prediction baseline, revealing a systematic calibration failure—an inability to know when they don't know. Tool design mattered more than model choice: the same Qwen model jumped from 5% to 15% accuracy when given custom baseline tools with structured memory and prediction guidance. Ablations showed agent-driven iterative search added 10.6 points, with daily context updates accounting for most of the gain, while disabling memory write access degraded all models. Agents also converged toward Polymarket crowd aggregates in multi-agent runs. Limitations include the short window, single-source questions, LLM-based answer judging, cost barriers (~$50+ per run), and the absence of individual human superforecaster baselines.

FutureSim Paper Deep Dive: When AI Is Thrown Into Real Time

Key points

Framework: FutureSim replays world events in true chronological order — 330 forecasting questions from Al Jazeera, 7.36M Common Crawl News articles, and an 88-day simulated window (Jan 1 – Mar 28, 2026). Agents receive each day's articles sequentially, can search, resubmit predictions, and outcomes resolve on 84 different dates without being told when.
Calibration failure: Using the Brier Skill Score (BSS; 1 = perfect, 0 = abstain, negative = worse than not predicting), GPT 5.5 was the only model with a positive score under native tools. Open-weight models like Qwen 3.6 Plus scored BSS ≈ -0.07 — actively harmful. When all models started from a deliberately bad initial prediction, none recovered to even the 0 (abstain) line.
Tools matter more than models: Qwen 3.6 Plus dropped to 5% accuracy with native OpenCode tools (spamming the "advance to next day" action), but reached 15% with custom baseline tools featuring structured memory, programmatic forecasting guidance, and forced memory-update phases.
Sequential replay is the real innovation: Because predictions don't affect the world, FutureSim sidesteps the counterfactual problem of interactive environments. The paradigm is transferable to code review (GitHub commit replay), medical diagnosis (patient-record replay), and finance (trading-day replay).
Search and memory ablations: Agent-driven iterative search lifted accuracy from 14.2% to 24.8% (+10.6 pts); removing daily context updates dropped it to 17.9%. Disabling memory writes degraded every model — over 88 days, external memory is infrastructure, not a bonus.
Reasoning scaling flatlined: Raising GPT 5.5 from high to very high effort (3,572 → 3,690 calls) changed accuracy by nothing.
Crowd convergence: Multi-agent DeepSeek runs converged toward Polymarket crowd aggregates (TV distance shrinking over time), while independent single-agent runs diverged. The paper observes but does not explain the mechanism (ensemble averaging vs. social learning).
Cost inequality: A full GPT 5.5 evaluation consumed ~12.4M tokens and ~3,700 tool calls; analyst estimates exceed $50 per run (not officially reported). Open-weight models are partly handicapped by lacking expensive custom tooling.

Limitations raised in the post

The 88-day window is too short to show long-horizon effects like compounding "belief traps."
All 330 questions come from a single source (Al Jazeera), risking source bias.
Answer matching is judged by DeepSeek v3.2 — AI judging AI introduces invisible evaluation bias.
No individual human superforecaster baseline exists, making the 24.8% figure impossible to contextualize (Polymarket crowd aggregation is the only reference).

Paper reference

Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping. *FutureSim: Replaying World Events to Evaluate Adaptive Agents*. arXiv:2605.15188, 2026. Institutions: ELLIS Institute Tübingen; Max Planck Institute for Intelligent Systems; University of Stuttgart; Tübingen AI Center; University of Tübingen; University of Southampton.

FutureSim complements existing benchmarks such as ARC-AGI 3, BALROG, ProphetArena, ForecastBench, and GAIA-2 on horizon length and dynamism dimensions (paper Table 1, Section 2).