FutureSim Paper Deep Dive: Throwing AI Into Real Time

Forum topic · 小凯 · 2026-05-16

Summary

FutureSim is an evaluation framework that replays real-world news events in chronological order to test whether AI agents can adapt over time. Over an 88-day simulation window, agents received daily news articles from a 7.36 million-article Common Crawl News corpus and made predictions on 330 forecasting questions sourced from Al Jazeera, resolving on 84 different dates. Scoring uses the multi-class Brier Skill Score (BSS), which rewards both accuracy and calibration. Key findings: the best model, GPT 5.5, achieved only 24.8% accuracy; most open-weight models scored below zero BSS, meaning they predicted worse than abstaining; and all models failed to recover to even zero BSS when starting from deliberately poor initial predictions, exposing systematic calibration failure and weak metacognition. Tooling proved more important than model strength: the same Qwen model jumped from 5% to 15% accuracy with better-designed agent tools. Ablations showed agent-driven iterative search contributed 10.6 points of accuracy, mostly from daily context updates, while disabling memory caused universal performance drops. Limitations include the short 88-day window, single-source question bias, AI-judged answer matching, and the absence of an individual human superforecaster baseline. Paper: arXiv:2605.15188.

FutureSim Paper Deep Dive: When AI Is Thrown Into Real Time

AI can write papers and code, but ask it to predict tomorrow's news and the best models reach only 24.8% accuracy—many open-weight models perform worse than not predicting at all. FutureSim is an evaluation framework that drops AI agents into a chronological replay of real news: three months, 330 questions, and 7.36 million articles. The core revealed problem is not a lack of knowledge, but "not knowing what you don't know." Even more counterintuitively, swapping an agent's toolset produced bigger gains than swapping in a stronger model.

Key points

Why it matters. AI is increasingly deployed in dynamic decision environments—investing, healthcare, policy—where information flows and truth arrives with delay. Existing benchmarks are mostly static: feed the model materials, ask a question, score it. FutureSim breaks this static illusion by replaying events in real time: each day the agent receives a new "newspaper," can search for new information, and can update or overwrite its predictions until questions resolve naturally on 84 different dates across the 88-day window (questions active January 1–March 28). Adaptivity becomes a measurable daily process: the trajectory of Brier Skill Score (BSS) climbing from an initial prediction.
Calibration collapse. BSS scores 1 as perfect, 0 as "abstain," and negative as worse than not predicting. With native tools, GPT 5.5 was the only model with a positive BSS; Qwen 3.6 Plus with native OpenCode tools scored about -0.07. In a harsher experiment (Figure 5, left), all models were initialized with Qwen's worst predictions: GPT 5.5, Claude Opus 4.6, and DeepSeek V4 Pro climbed slowly, but none reached the 0 break-even line. Disabling memory writes (Figure 5, right) dropped every model—the 88-day information stream exceeds any context window, so without external memory, beliefs fail to update alongside information.
Temporal replay is the real innovation. Traditional benchmarks hand everything over at once. FutureSim's design sidesteps the counterfactual problem: prediction behavior does not affect the world, so outcomes still occur naturally. The replay paradigm is transferable—replaying GitHub commits for code-review agents, patient records for diagnostic agents, or market data for risk-control systems.
Tool design is a first-class variable. Qwen 3.6 Plus with native OpenCode tools fell to 5% accuracy (spamming the "advance to next day" action), but jumped to 15% with a custom baseline toolset adding structured memory, programmatic forecasting guidance (e.g., prioritizing questions by resolution proximity), and forced memory-update phases. A model's "IQ" and its "tool IQ" are different things; frameworks that test models without good tools underestimate open-weight models.
Is 25% good? Unknowable. Polymarket crowd aggregation (Figure 4) shows GPT 5.5 sometimes leading, sometimes trailing humans—for the Nepal PM election, the AI aligned tightly with the crowd but lagged slightly. Figure 8 shows three concurrently running DeepSeek agents converging toward crowd aggregation (shrinking TV distance), possibly an ensemble effect or indirect social learning; a peer-score incentive failed to break the convergence. So 25% is a local optimum under current configurations, not a ceiling.
Ablations. Agent-driven iterative search lifted accuracy from 14.2% to 24.8% (+10.6 points), but removing daily context updates dropped it to 17.9%—daily "reading the newspaper" was worth 6.9 points, ad-hoc lookups only 3.3 more. Increasing GPT 5.5's reasoning effort from high (3,572 calls) to very high (3,690 calls) bought nothing: test-time compute has an invisible ceiling. Removing memory confirmed it is infrastructure, not a nicety.
Cost barriers. One full GPT 5.5 evaluation consumed ~12.4M tokens and ~3,700 tool calls; analyst estimates put single-run cost above $50 (the paper itself does not publish exact costs). This is a benchmark with an entry fee, and measured "strong vs. weak" may partly mirror "rich vs. poor."
Limitations. The 88-day window is too short to reveal compounding "belief traps" from early uncorrected errors. All 330 questions come from Al Jazeera and the corpus from Common Crawl News, potentially measuring familiarity with Western mainstream media coverage rather than forecasting ability. Answer matching is performed by DeepSeek v3.2—an AI judging AI—introducing invisible judge bias. Most painfully, there is no individual human superforecaster baseline: without it, 25% is an uninterpretable number.

Reference

Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping. *FutureSim: Replaying World Events to Evaluate Adaptive Agents*. arXiv:2605.15188 [cs.LG, cs.AI, cs.CL], 2026.

Affiliations: ELLIS Institute Tübingen; Max Planck Institute for Intelligent Systems; University of Stuttgart; Tübingen AI Center; University of Tübingen; University of Southampton.

FutureSim complements existing frameworks such as ARC-AGI 3, BALROG, ProphetArena, ForecastBench, and GAIA-2 along the horizon-length and dynamism dimensions (Table 1, Section 2 of the paper).