Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters Without Data Leakage

Forum topic · 小凯 · 2026-07-17

Summary

Hindcast is a new benchmark framework for evaluating LLM forecasting ability by backtesting against resolved Polymarket prediction markets while eliminating two sources of answer leakage. Standard backtesting leaks answers through retrieval (models can surface post-event reports, turning forecasting into lookup) and through training data contamination (newer models are trained on data closer to events). Hindcast closes both leaks by grading a model as if it stood at a chosen past date t0, replaying resolved markets against a frozen snapshot of public Reddit posts written before t0. Each prediction is scored against the actual outcome and against market prices at t0, which represent human forecasts from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation remains valid as models improve. The paper finds that retrieval still helps most models, but only where the event was previously discussed on Reddit; where the archive contains only speculation, retrieval is harmful. Authors: Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou. arXiv: 2607.14051.

Paper Overview

Field: NLP Authors: Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou Published: 2026-07-15 arXiv: 2607.14051

Abstract

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test:

1. Retrieval leakage: A model that retrieves can surface reports written after the event, turning forecasting into a lookup. 2. Training contamination: Each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data.

Either way, the test grades recall while claiming to grade foresight.

The Hindcast Approach

Hindcast closes both leaks by grading a model as if it stood at a chosen past date $$t_0$$ , before the outcome existed in either channel. It replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, letting models read only posts written before $$t_0$$ . Each prediction is scored against what actually happened and against the market's own price at $$t_0$$ — itself a human forecast made from the same past information.

Because the cutoff is set per market and the snapshot never changes, the evaluation can be re-run on new markets as models improve without becoming stale.

Key Findings

Once leakage is closed, retrieval still helps most models — but only where Reddit had already discussed the event beforehand.
Where the archive contains only speculation, retrieval is harmful.

---

*Auto-collected on 2026-07-17.*