Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters Without Data Leakage

Forum topic · 小凯 · 2026-07-17

Summary

A new benchmark called Hindcast addresses a core flaw in evaluating LLM forecasting ability: backtesting leaks answers through two channels—retrieval of post-event reports and training data that already contains resolved outcomes. Hindcast grades models as if they stood at a chosen past date t0, before the outcome existed in either channel. It replays resolved Polymarket prediction markets against a frozen public Reddit snapshot, allowing models to read only posts written before t0, and scores each prediction against both the actual outcome and the market's own price at t0 (a human forecast made from the same past information). Because cutoffs are set per market and snapshots never change, the evaluation remains valid as new models improve and can be rerun on new markets. Experiments show that once leakage is closed, retrieval still helps most models, but only where the outcome was discussed on Reddit before t0; where the archive contains only speculation, retrieval actively hurts performance.

Paper Overview

Field: NLP
Authors: Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou
Published: 2026-07-15
arXiv: 2607.14051

Summary

Forecasters are typically evaluated by backtesting, which replays resolved questions and grades the probability a system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test:

1. Retrieval leakage: A model that retrieves can surface reports written after the event, turning forecasting into a lookup. 2. Training-data leakage: Each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data.

Either way, the test grades recall while claiming to grade foresight.

The Hindcast Benchmark

Hindcast closes both leaks by grading a model as if it stood at a chosen past date $$t_0$$ , before the outcome existed in either channel. The design works as follows:

Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit data.
Models can read only posts written before $$t_0$$ .
Each prediction is scored against both what actually happened and the market's own price at $$t_0$$ , which is itself a human forecast made from the same past information.

Because the cutoff is set per market and the snapshot never changes, the evaluation stays valid as models improve and can be rerun on new markets without becoming stale.

Findings

Once leakage is closed, retrieval still helps most models, but only in cases where Reddit discussed the event before the cutoff. Where the archive contains only speculation, retrieval is harmful.

--- *Auto-collected on 2026-07-17*

Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters Without Data Leakage

Summary

Paper Overview

Summary

The Hindcast Benchmark

Findings

Tags