FutureSim: Replaying World Events to Evaluate Adaptive Agents

Forum topic · 小凯 · 2026-05-17

Summary

FutureSim is a benchmark introduced on arXiv (2605.15188) that evaluates adaptive AI agents by replaying real-world events in chronological order. Agents interact with a simulation where real news articles arrive over time and they must predict world events beyond their knowledge cutoff. The authors—Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, and Jonas Geiping—evaluate frontier agents on forecasting world events during January–March 2026. Results reveal clear capability gaps: the best agent achieved only 25% accuracy, and many agents produced Brier skill scores worse than naive non-prediction baselines. Ablation studies show FutureSim provides a realistic testbed for emerging research areas including long-horizon test-time adaptation, search, memory, and uncertainty reasoning. The benchmark aims to measure open-ended, real-world adaptation of AI systems over long time horizons.

Paper Overview

Field: NLP Authors: Shashwat Goel, Nikhil Chandak, Arvindh Arun, Ameya Prabhu, Steffen Staab, Moritz Hardt, Maksym Andriushchenko, Jonas Geiping Published: 2026-05-14 arXiv: 2605.15188

Abstract

AI agents are increasingly deployed in dynamic, open-ended environments where they need to adapt as new information arrives. To measure this capability efficiently in realistic use cases, the authors propose building simulations grounded in reality by replaying real-world events in the order they occurred.

They construct FutureSim, in which agents predict world events beyond their knowledge cutoff while interacting with a chronological replay of the world: real news articles arrive during the simulation and questions are resolved incrementally. The authors evaluate frontier agents in their native settings, testing their ability to forecast world events during a three-month period from January to March 2026.

Key Findings

FutureSim reveals a clear stratification of agent capabilities.
The best-performing agent achieved only 25% accuracy.
Many agents obtained Brier skill scores worse than not predicting at all.
Careful ablation studies show FutureSim provides a realistic environment for studying emerging research directions such as long-horizon test-time adaptation, search, memory, and uncertainty reasoning.

Overall, the authors hope this benchmark design paves the way for measuring open-ended AI adaptation over long time horizons in the real world.

--- *Auto-collected on 2026-05-17*