FutureSim: AI Agents Fail at Real-World Event Forecasting — Deep Dive

Forum topic · 小凯 · 2026-05-16

Summary

FutureSim is a benchmark that replays real world events in strict chronological order to evaluate how well AI agents adapt and forecast over a 90-day simulated period (January–March 2026). Agents receive daily news articles in sequence and must make, update, and revise probabilistic predictions on 330 questions without accessing future information. Results are stark: the best agent (GPT 5.5 via Codex) achieves only 25% top-1 accuracy, and many agents obtain worse Brier Skill Scores than abstaining from prediction entirely. The paper identifies three key failure modes: anchoring on initial predictions even when evidence contradicts them, poor memory management during long-horizon runs, and miscalibrated overconfidence. Open-weight frontier models score negative Brier Skill Scores in default harnesses and only become marginally usable with improved tooling, memory, and search. GPT 5.5 consumed 12.4M tokens and 3,700 interactions across the full simulation. Authors argue current LLM training lacks any temporal-adaptation signal, motivating dynamic, time-unfolding evaluation and training environments. Paper: arXiv:2605.15188.

FutureSim: Replaying World Events to Evaluate Adaptive Agents — Deep Dive

This post analyzes FutureSim (arXiv: https://arxiv.org/abs/2605.15188), a benchmark by Goel, Chandak, Arun, Prabhu, Staab, Hardt, Andriushchenko, and Geiping that evaluates whether AI agents can adapt to and forecast real-world events over time.

Key points

Setup: A 90-day replay (January–March 2026) where agents receive daily, date-stamped news articles in strict chronological order within a sandboxed environment. Agents face 330 prediction questions that resolve over the simulation window, and can search, remember, and update predictions as new information arrives — but cannot see future news.
Design principle — replay, not simulate: Instead of using an LLM to simulate world evolution (which would amplify model biases), FutureSim replays actual events as ground truth. Strict sandboxing and cross-checking with models of different knowledge cutoffs mitigate leakage from parametric knowledge, though the authors acknowledge full prevention of leakage is difficult.
Open-ended agent policy: Unlike fixed QA benchmarks, agents decide which questions to predict, when to submit, and when to revise probability distributions.

Headline results

| Agent | Top-1 Accuracy | |---|---| | GPT 5.5 (Codex) | 25% | | Claude 4 (Opus) | ~13–15% | | Gemini 3.1 Pro | ~10–15% | | Qwen 3.6 Plus | ~5–10% |

The best agent's 25% accuracy is described in the paper as "shockingly bad."
Many agents have worse Brier Skill Score than making no prediction at all — meaning they are overconfident when wrong, not merely inaccurate.
Open-weight frontier models (e.g., Llama, Qwen, DeepSeek) show negative Brier Skill Scores in the default agent harness; only with a modified harness offering better search, memory, and tool use do they become marginally viable.
GPT 5.5 consumed 12.4M tokens and 3,700 interactions over the 90-day run, spanning multiple sequential context-window compressions.

Three failure modes identified via ablations

1. Anchoring: Agents stick to initial predictions and fail to adapt adequately. Notably, when strong models (GPT 5.5) were seeded with initial predictions from the weakest model (Qwen 3.6 Plus), they remained anchored on the weak predictions despite being capable of better judgments. 2. Memory: Ablating test-time memory read/write hurts performance — models "demonstrably benefit from in-session memory." But effective memory requires knowing what to store, forget, and when to revisit, which current systems handle poorly. 3. Calibration: GPT 5.5's positive Brier Skill Score comes not only from more correct answers but from better uncertainty expression — avoiding extreme probabilities when unsure. Most models lack this "humility about uncertainty."

Implications for AI research

We optimize the wrong things? Standard LLM training (next-token prediction, static QA, coding, math) has no temporal dimension — no incremental information arrival or belief revision. FutureSim exposes the gap: models are barely trained for adaptation in dynamically unfolding environments.
The tooling gap: Closed-model harnesses (like Codex) may contribute as much to performance as raw model capability, raising the question of whether open-weight models can close a persistent "tool-chain gap."
Is 90 days long enough? Critics (cited in the post, e.g., Alan Hou) argue three months cannot reveal compounding errors or belief drift; the real value is showing even *short-term* adaptation is broken in current models.

The benchmark's takeaways

FutureSim's authors frame the benchmark not as showing off any model's strength, but as an honest measurement of how far current AI is from reliable real-world adaptation. Quoting the paper: "FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all." A recurring insight: forecasting ability is less about how much a model knows and more about whether it knows what it does not know.

References

1. Goel, S., et al. (2026). FutureSim: Replaying World Events to Evaluate Adaptive Agents. *arXiv preprint arXiv:2605.15188*. 2. Mellers, B., et al. (2015). The psychology of intelligence analysis. *J. Experimental Psychology*. 3. Atanasov, P., et al. (2020). Crowd forecasting. *Management Science*. 4. Silver, D., & Sutton, R. (2025). The bitter lesson. *Communications of the ACM*. 5. Karger, E., et al. (2025). ForecastBench. *arXiv preprint*. 6. Chandak, N., et al. (2026). Methodology for creating prediction questions from news articles.

*(Translated and edited from the original Chinese forum post.)*