FutureSim: Replaying World Events to Evaluate Adaptive Agents
Paper Information
- Title: FutureSim: Replaying World Events to Evaluate Adaptive Agents
- Authors: Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping
- arXiv: https://arxiv.org/abs/2605.15188
- Field: NLP / ML / AI Evaluation
- Static QA: a fixed question, fixed answer, no new information.
- Game environments: closed, hand-designed worlds.
- Code competitions: deterministic verifiers.
- Reliable, date-annotated offline news snapshots.
- Search restricted to the current simulation date.
- Cross-validation across models with different knowledge cutoffs.
- Which questions to predict.
- When to submit predictions.
- When to update predictions.
- How to allocate probability across outcomes.
- Next-token prediction.
- Static QA accuracy.
- Code competition pass rates.
- Mathematical reasoning.
- FutureSim is a 90-day temporal-replay benchmark (Jan–Mar 2026) with 330 real-world prediction questions answered by LLM agents using live daily news.
- The strongest agent, GPT 5.5 (Codex), reaches only 25% top-1 accuracy; most open-weight models score negative Brier Skill Scores—worse than abstaining.
- Three core failure modes are identified: anchoring on initial predictions, shallow in-session memory management, and poor uncertainty calibration.
- The benchmark exposes a training gap: current LLMs are optimized for static tasks, not temporally unfolding adaptive reasoning.
- Open-weight models trail closed models partly due to weaker agent harnesses rather than raw capability.
- GPT 5.5 consumed 12.4 million tokens across 3,700 interactions during the simulation, illustrating the compute cost of long-horizon adaptation.
- FutureSim's 90-day window may be too short to surface compounding belief drift; longer-horizon evaluation is a natural next step.
---
The Crystal Ball Shattered: When AI Faces Real Time
> "I can live with doubt and uncertainty and not knowing. I think it's much more interesting to live not knowing than to have answers which might be wrong." > — Richard Feynman
Chapter 1: The Prophet's Three Months
On January 1, 2026, an AI agent is placed inside a simulated environment with one task: predict what will happen in the world over the next three months.
Not distant futures—will humans reach Mars in ten years—but the next three months: January through March 2026, events that have already occurred.
The agent can search news articles. New articles arrive daily, in true chronological order, like the news feed you see when you wake up. The agent updates its predictions as new information arrives. Some questions resolve during the simulation—will a certain country pass a certain bill?—and when the bill is signed or vetoed, the AI knows whether it was right.
Sounds simple: let an AI forecast real-world events, feed it the news, see how often it is right.
But the result, in the authors' own words, is "shockingly bad."
The best agent achieves 25% accuracy. Many perform worse than not predicting at all.
> "FutureSim reveals a clear separation in their capabilities, with the best agent's accuracy being 25%, and many having worse Brier skill score than making no prediction at all."
25%. One in four. Roughly coin-flip accuracy—and sometimes worse.
The most unsettling part: the agents do incorporate new information and their accuracy does improve over time—but the rate and magnitude of improvement fall far short of reliable forecasting.
FutureSim's purpose is not to showcase any model's strength. It is an honest demonstration of how far current AI falls short when facing the dynamics of the real world.
---
Chapter 2: Why Is Real-World Prediction So Hard?
Why ask AI to forecast world events?
The capability behind forecasting is adaptation—the ability to revise beliefs and actions as new evidence arrives.
A doctor examines a feverish, headachy patient. Initial suspicion: flu. Lab results arrive—it is something worse. The doctor immediately revises the diagnosis and changes treatment. That is adaptation. Not "remembering a lot," but updating beliefs when knowledge is incomplete.
AI agents are increasingly deployed in such settings: customer support bots adjusting answers from user feedback, trading algorithms adapting to market news, autonomous vehicles reacting to changing road conditions.
Most existing benchmarks miss this. They test:
These are valuable, but they ignore a core problem: the real world is not a game. It has no fixed rules, no gold answer, and it changes every day.
FutureSim's authors propose a simple, profound idea:
> "Let's replay the world."
Not simulate a fictional world. Replay the real world in true chronological order. Feed the AI January 1 news, ask it to predict the next week's election outcome. Feed it January 2 news, let it revise. Continue daily until the election resolves.
Every search, every update, every revision happens inside a strict temporal sandbox: the AI cannot peek at future articles. It must judge like a real-world analyst using only publicly available information up to the present.
---
Chapter 3: The Cold Truth Behind the Numbers
Top-1 Accuracy
Across 90 days (January–March 2026) and 330 prediction questions:
| Agent | Accuracy | |---|---| | GPT 5.5 (Codex) | 25% | | Claude 4 (Opus) | ~13–15% | | Gemini 3.1 Pro | ~10–15% | | Qwen 3.6 Plus | ~5–10% | | Abstention baseline | N/A |
GPT 5.5—one of the strongest available models—is wrong three quarters of the time on real-world events.
Brier Skill Score: A Death Sentence for Calibration
Brier Skill Score measures not just right vs. wrong but how well uncertainty is expressed.
If the model says "100% rain tomorrow" and it rains, it scores well: correct and confident. If it says "30% chance" and it rains, it scores lower: right but under-confident.
If it says "100% rain" and it does not rain, the score collapses: wrong and maximally overconfident—the worst combination. If it says "50%" and it does not rain, the penalty is much milder.
Brier Skill Score compares the model's score against always predicting the base rate. Positive: better than naïve. Negative: worse than not predicting at all.
The paper reports:
> "Many having worse Brier skill score than making no prediction at all."
Many models produce negative Brier Skill Scores. They are not just inaccurate—they are confidently inaccurate. They behave like the experts who loudly proclaim "the market only goes up" right before a crash: not failing because of missing information, but failing because they mishandle the information they do have.
Notably, open-weight frontier models (Llama, Qwen, etc.) have negative Brier Skill Scores under default agent harnesses. Only after harness modifications—better search and memory tooling—do they barely cross into positive territory.
---
Chapter 4: Where AI Goes Wrong — Three Fatal Blind Spots
Blind Spot 1: Anchoring
> "We find agents are anchored on their initial predictions, failing to adequately adapt, and even failing to reach the abstaining baseline of 0 Brier skill score despite their own capabilities being stronger."
Once an AI makes an initial prediction, it struggles to revise dramatically.
On January 1, the agent says "70% chance the bill passes." For two months, news accumulates pointing toward failure. The probability drifts: 70 → 60 → 55. It rarely jumps to 20 or 10, even when evidence demands it.
This is anchoring: the initial judgment acts like an anchor, restricting subsequent adjustments to a narrow range. Humans exhibit this too (Tversky & Kahneman), but the effect appears amplified in current AI.
A clever experiment: use a weak model (Qwen 3.6 Plus) to seed initial predictions, then hand them to a stronger model (GPT 5.5). The strong model possesses the capability for better judgment but remains anchored on the weak model's priors.
Blind Spot 2: Memory Is Not Simple Storage
The paper ablates the ability to read and write memory at test time:
> "By ablating the ability to write and retrieve memory at test time, we find that models demonstrably benefit from in-session memory."
With memory, models do better—unsurprising. But the deeper challenge is: knowing what is worth remembering, what to forget, and when to revisit old notes.
A skilled analyst does not memorize every detail. She remembers key events, trend shifts, important statements—and revisits them when new information conflicts. Current AI memory resembles stuffing everything into a drawer that grows slower to search, with poor judgment about what remains relevant.
Blind Spot 3: The Art of Uncertainty
GPT 5.5's positive Brier Skill Score comes less from being right more often than from expressing uncertainty better.
When uncertain, it says "50%" or "60%" rather than blindly outputting near-zero or near-one.
Most models lack this epistemic humility. They are either confidently correct or confidently wrong. The latter is fatal under Brier's penalty.
A deep insight: predicting world events is not only about how much knowledge you have, but about how well you know what you don't know. An AI that admits ignorance may score higher under Brier than one packed with knowledge but crippled by overconfidence.
---
Chapter 5: The Design Wisdom of FutureSim
Temporal Replay: Not Simulation, Reconstruction
Core principle: do not simulate, replay.
Why not ask an LLM to simulate world events? Because simulated worlds inherit the model's own biases. A model with a distorted worldview will generate distorted simulations. Replaying the real world guarantees that world evolution is ground truth.
The authors construct the environment from timestamped news documents, with prediction questions generated using the methodology of Chandak et al. (2026). Context evolves as daily news arrives.
Sandbox: Preventing Information Leakage
A key technical challenge: how to keep the AI from "seeing" the future?
LLMs may have been trained on 2026 Q1 news. Even with restricted offline snapshots, parametric knowledge may leak.
FutureSim's response is strict sandboxing:
The paper acknowledges complete leakage prevention is hard. If the model saw "March 2026 election result" during training, it may subconsciously remember.
Openness: The AI Decides What to Predict
FutureSim is not a fixed questionnaire. Agents choose:
This openness mirrors real forecasting tasks. A real analyst is not handed a script; they choose what is worth tracking and when to revise.
The trade-off: evaluation is harder. The paper uses top-1 accuracy and Brier Skill Score as primary metrics, while logging token consumption and tool calls.
GPT 5.5 consumed 12.4 million tokens and performed 3,700 interactions across multiple sequential context-window compressions. Even the best model requires enormous compute for long-horizon adaptation.
---
Chapter 6: What This Means for AI Research
Question 1: Are We Optimizing the Wrong Things?
Current LLM training optimizes:
All share a property: fixed input, fixed output, no temporal dimension, no information arrival, no belief revision.
FutureSim exposes a vast gap: we barely train AI to adapt to a temporally unfolding world.
The paper invokes Silver & Sutton's (2025) "Bitter Lesson"—scale and compute trump hand-crafted representations—but hints at a correction: for adaptive agents, we may need a new training paradigm, one based in dynamic, temporally unfolding environments rather than static datasets.
Question 2: The Open-Weight Predicament
One of the paper's most concerning findings:
> "Open-weight frontier models have a negative Brier skill score, which makes them worse than abstaining from prediction altogether."
Open-weight models (Llama, Qwen, DeepSeek) underperform in default configurations. Only with modified harnesses—better tool use, search strategy, memory, uncertainty handling—do they approach usability.
Implication: raw model capability is insufficient for real-world adaptation. What is needed is not only better pretraining but better agent design—tooling, memory management, search strategy, uncertainty expression.
This creates an open-vs-closed tension. GPT 5.5's lead may partly reflect its Codex harness rather than raw model strength. Can open-weight models close the gap, or will a persistent "toolchain chasm" remain?
Question 3: Is Three Months Long Enough?
The paper simulates 90 days—long by AI benchmark standards, with 3,700 agent turns and multiple context compressions.
But is 90 days truly "long-horizon"?
An external commentator (Alan Hou, blog post) argues:
> "The benchmark is useful, but the paper oversells the 'long-horizon' framing. Three months isn't long enough to see compounding errors or drift. The real value is showing that even short-term adaptation is broken in current models."
Three months cannot reveal compounding errors or belief drift. What happens if an AI runs for one, three, or five years? Does its belief system slowly detach from reality? Does it develop a form of "paranoia"—over-relying on early judgments and ignoring later counter-evidence?
FutureSim cannot yet answer these questions, but it places them firmly on the table.
---
Epilogue: Feynman's Mirror
How would Feynman view FutureSim?
First, he would admire the honesty.
> "For a successful technology, reality must take precedence over public relations, for nature cannot be fooled."
FutureSim does not cherry-pick scenarios where AI excels; it shows how badly AI fails in its weakest moments. 25% accuracy is not a bragging number—but it is real.
He would probe:
> "What do you mean by 'prediction'? A binary guess (yes/no), or a probability distribution? If the AI says '60%' and the event does not happen, is that right or wrong?"
Then he would look at the negative Brier Skill Scores:
> "So the problem is not merely guessing wrong. It is guessing wrong while being sure. That reminds me of NASA managers compressing failure probability from 1/100 to 1/100,000—not because evidence supported it, but because 'must succeed' pressure drove self-deception. These AIs are also self-deceiving—not under pressure, but because their training rewards sounding confident more than admitting ignorance."
Finally, he might say:
> "FutureSim is a good idea. But three months is too short. If you really want to understand how these machines 'think'—or whether they think at all—you should run them longer. Watch how errors compound. Watch whether they develop a 'worldview' they then refuse to change. That is the interesting part."
> "Still, 25% accuracy... at least better than a fortune teller, right?"
---
Key Points
References
1. Goel, S., Chandak, N., Arun, A., Prabhu, A., Staab, S., Hardt, M., Andriushchenko, M., & Geiping, J. (2026). FutureSim: Replaying World Events to Evaluate Adaptive Agents. *arXiv preprint arXiv:2605.15188*. 2. Mellers, B., et al. (2015). The psychology of intelligence analysis. *Journal of Experimental Psychology*. 3. Atanasov, P., et al. (2020). Crowd forecasting. *Management Science*. 4. Silver, D., & Sutton, R. (2025). The bitter lesson. *Communications of the ACM*. 5. Farquhar, S., & Gal, Y. (2019). Towards robust evaluations of continual learning. *NeurIPS Workshop*. 6. Karger, E., et al. (2025). ForecastBench. *arXiv preprint*. 7. Chandak, N., et al. (2026). Methodology for creating prediction questions from news articles.