FutureSim: Replaying World Events to Evaluate Adaptive AI Agents

Forum topic · 小凯 · 2026-05-17

Summary

FutureSim is a benchmark that evaluates AI agents by replaying real-world events in chronological order, requiring them to predict events beyond their knowledge cutoff. Built by researchers including Shashwat Goel, Ameya Prabhu, and Jonas Geiping (arXiv:2605.15188), the benchmark delivers real news articles incrementally during a simulation covering January to March 2026, with questions progressively resolved as events unfold. Results reveal stark capability differences among frontier agents: the best agent achieved only 25% accuracy in forecasting world events, and many agents scored worse than chance on the Brier skill score. Ablation studies show FutureSim provides a realistic testbed for emerging research directions such as long-horizon test-time adaptation, search, memory, and reasoning under uncertainty. The authors position FutureSim as a step toward measuring open-ended adaptation of AI systems over long time horizons in real-world settings.

Paper Overview

Field: NLP Authors: Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping Published: 2026-05-14 arXiv: 2605.15188

Abstract

AI agents are increasingly deployed in dynamic, open-ended environments and must adapt as new information arrives. To measure this capability efficiently in realistic use cases, the authors propose building reality-grounded simulations that replay real-world events in the order they occurred.

FutureSim works by having agents interact with a chronological replay of the world and predict events beyond their knowledge cutoff: real news articles arrive during the simulation, and questions are progressively resolved over time.

Key Findings

Setup: Frontier agents are evaluated in their native settings, predicting world events over a three-month window from January to March 2026.
Results: FutureSim reveals a clear stratification of agent capabilities. The best agent achieved only 25% accuracy, and many agents earned a Brier skill score worse than never forecasting at all.
Ablations: Careful ablation studies demonstrate how FutureSim provides a realistic environment for studying emerging research directions such as long-horizon test-time adaptation, search, memory, and reasoning under uncertainty.

Significance

The authors hope this benchmark design paves the way for measuring AI progress on open-ended adaptation over long time horizons in the real world.

--- *Auto-collected on 2026-05-17*