Overview
Field: NLP Authors: Shashwat Goel, Nikhil Chandak, Arvindh Arun Posted: 2026-05-16 arXiv: 2505.08630
Abstract
AI agents are being increasingly deployed in dynamic, open-ended environments that require adapting to new information as it arrives. To efficiently measure this capability for realistic use-cases, the authors propose building grounded simulations that replay real-world events in the order they occurred.
They build FutureSim, where agents forecast world events beyond their knowledge cutoff while interacting with a chronological replay of the world: real news articles arriving and questions resolving over the simulated period. Frontier agents are evaluated in their native harness, testing their ability to predict world events over a three-month period from January to March 2026.
Key Findings
- FutureSim reveals a clear separation in agent capabilities: the best agent's accuracy is only 25%.
- Many agents have worse Brier skill scores than not making predictions at all.
- Careful ablations show FutureSim provides realistic scenarios for studying emerging directions such as long-horizon test-time adaptation, search, memory, and uncertainty reasoning.
- The benchmark design aims to pave the way for measuring open-ended adaptation of AI in the real world across long time spans.