FutureSim: Replaying Real-World Events to Evaluate Adaptive AI Agents — Deep Dive

Forum topic · 小凯 · 2026-05-16

Summary

FutureSim is a benchmark that replays real-world events in strict chronological order to test whether AI agents can adaptively forecast unfolding news. Agents are given timestamped news snapshots and must predict outcomes of 330 real questions over a simulated 90-day period (January–March 2026), updating predictions as new information arrives. Results are stark: the best agent (GPT 5.5 via Codex) reached only 25% top-1 accuracy, and many models scored worse than a no-prediction baseline on the Brier Skill Score, meaning they were confidently wrong. The paper identifies key failure modes, including strong anchoring on initial predictions, inadequate memory management, and poor calibration of uncertainty. Open-weight frontier models had negative Brier skill scores unless given improved tooling (search, memory). GPT 5.5 consumed 12.4 million tokens and 3,700 interactions over the simulation, highlighting the compute cost of long-horizon adaptation. FutureSim argues current LLM training, focused on static inputs and outputs, leaves a major gap in temporal adaptation, and proposes replaying reality (rather than simulating it) to avoid model bias. The forum post analyzes these findings in depth with a Feynman-inspired framing.

FutureSim: Replaying World Events to Evaluate Adaptive Agents — Deep Dive

This is an English translation of a Chinese forum post analyzing the paper FutureSim: Replaying World Events to Evaluate Adaptive Agents.

Paper info

Title: FutureSim: Replaying World Events to Evaluate Adaptive Agents
Authors: Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping
arXiv: https://arxiv.org/abs/2605.15188
Fields: NLP / ML / AI evaluation

Key points

The setup: forecasting the real world, three months at a time

On January 1, an AI agent is placed in a simulated environment with one task: predict what will happen in the real world over the next three months. Not distant futures — the next 90 days of actual events (January–March 2026). News articles arrive daily in true chronological order, like a morning news feed. The agent can update its forecasts as new information arrives, and some questions "resolve" mid-simulation (e.g., a bill passes or fails, revealing whether the AI was right).

The result, in the authors' words, is "shockingly bad":

> "FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all."

The best agent reached only 25% accuracy — roughly coin-flip territory — and, disturbingly, agents *were* using new information to update forecasts and improving over time, just far too slowly to be reliable forecasters.

Why forecasting = adaptation

Forecasting ability proxies for adaptation: updating beliefs and actions based on new evidence (like a doctor revising a diagnosis when lab results come in). Existing benchmarks — static QA, game environments, coding contests — share a flaw: they lack a time dimension, with information arriving dynamically and requiring belief revision. FutureSim's core idea:

> "Let's replay the world."

Not simulate a fictional world — replay real events in true order, in a strict time sandbox where the agent cannot peek at future news.

The numbers

330 prediction questions over 90 days:

| Agent | Top-1 Accuracy | |-------|----------------| | GPT 5.5 (Codex) | 25% | | Claude 4 (Opus) | ~13–15% | | Gemini 3.1 Pro | ~10–15% | | Qwen 3.6 Plus | ~5–10% |

More damning than accuracy is the Brier Skill Score, which measures calibration — whether the AI correctly expresses uncertainty. It punishes overconfident errors far more than hedged ones. Many models scored negative, meaning worse than abstaining from prediction entirely: they are not just wrong, but *confidently* wrong. Notably, open-weight frontier models (Llama, Qwen, etc.) had negative Brier skill scores in the default agent harness, only turning positive with a modified harness offering better search and memory tools.

Three fatal blind spots

1. Anchoring: Agents cling to initial predictions, adjusting only marginally even as evidence accumulates against them. In a clever experiment, strong models (e.g., GPT 5.5) seeded with the weakest model's (Qwen 3.6 Plus) initial predictions failed to fully correct them — like Einstein trying to fix errors written deeply into a blackboard. 2. Memory is not simple storage: Ablating test-time memory read/write hurts performance. The real challenge is knowing *what* to remember, what to forget, and when to revisit — current systems behave like an overflowing drawer rather than an analyst's working notes. 3. The art of uncertainty: GPT 5.5's positive Brier score comes not just from being right more often, but from expressing calibrated uncertainty ("50–60%" instead of extreme probabilities). The insight: forecasting is less about knowing a lot and more about knowing what you don't know.

Design wisdom in FutureSim

Replay, don't simulate: LLM-simulated worlds would amplify the model's own biases; replaying reality guarantees a ground-truth world trajectory, built from timestamped news documents (question creation follows Chandak et al., 2026).
Sandboxing against leakage: Date-labeled offline news snapshots, search restricted to the current simulated date, and cross-validation across models with different knowledge cutoffs — though the paper admits fully preventing parametric-knowledge leakage is hard.
Open-endedness: Agents choose which questions to predict, when to submit and update, and how to distribute probability — closer to real forecasting work. Evaluation uses top-1 accuracy and Brier Skill Score, plus token and tool-call counts. GPT 5.5 consumed 12.4 million tokens and 3,700 interactions across multiple compressed context windows over 90 days.

What this means for AI research

1. Are we optimizing the wrong things? LLM training optimizes static input→output tasks with no time dimension. FutureSim exposes a huge gap: we barely train AI to adapt to a world unfolding over time — suggesting a new training paradigm in dynamic, temporally unfolding environments (a nuance on Silver & Sutton's "bitter lesson"). 2. The open-weight dilemma: Open-weight models underperform abstention in default harnesses, showing that raw model capability is insufficient — agent design (tools, memory, search, calibration) matters as much as pretraining, raising the question of a persistent "toolchain gap." 3. Is three months long enough? A critic (Alan Hou) argues the paper oversells the "long-horizon" framing: 90 days can't reveal compounding errors or belief drift. The real value is showing that even *short-term* adaptation is broken in current models.

Epilogue: Feynman's mirror

The post closes in Feynman's spirit: FutureSim deserves praise for honesty — "reality must take precedence over public relations, for nature cannot be fooled." The negative Brier scores echo NASA-style self-deception: overconfidence not from lack of information but from mishandling it. And three months is too short; run these machines longer and watch how errors compound — that's the interesting part.

> "25% accuracy... at least it beats a fortune-teller, right?"

References

1. Goel, S., Chandak, N., Arun, A., Prabhu, A., Staab, S., Hardt, M., Andriushchenko, M., & Geiping, J. (2026). FutureSim: Replaying World Events to Evaluate Adaptive Agents. *arXiv preprint arXiv:2605.15188*. 2. Mellers, B., et al. (2015). The psychology of intelligence analysis: Drivers of prediction accuracy in world politics. *Journal of Experimental Psychology*. 3. Atanasov, P., et al. (2020). Crowd forecasting: The wisdom of the crowd in predicting future events. *Management Science*. 4. Silver, D., & Sutton, R. (2025). The bitter lesson. *Communications of the ACM*. 5. Farquhar, S., & Gal, Y. (2019). Towards robust evaluations of continual learning. *NeurIPS Workshop*. 6. Karger, E., et al. (2025). ForecastBench: A benchmark for forecasting. *arXiv preprint*. 7. Chandak, N., et al. (2026). Methodology for creating prediction questions from news articles.

*Translation of a forum post written by "Xiao Kai" using a Feynman-style analytical framework.*