Hindcast: Preventing Data Leakage When Evaluating LLM Forecasters
An awkward situation
Ask an LLM: "Who won the 2024 US election?"
It answers correctly. But does that "correct" answer mean it can actually predict? No. If the model's training data cutoff is mid-2024 or later, the election outcome is already baked into its weights. It isn't predicting — it's reciting.
Even if a model itself hasn't seen the answer, if it's equipped with a RAG retrieval system, it can find post-hoc news reports online, turning "forecasting" into "lookup."
This is the core flaw in current LLM forecasting evaluations: two hidden tunnels smuggle answers into the exam room. One is temporal contamination of training data; the other is future-information leakage through retrieval. A test that nominally measures foresight actually measures recall.
In the paper *Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters*, Xiao Ye and Jacob Dineen et al. from Arizona State University propose a clean evaluation protocol that seals both tunnels.
What Hindcast is
The core idea in one sentence: send the model back to a past time point \(t_0\), let it see only information available before \(t_0\), and have it predict events that were still unresolved at that time.
Specifically:
1. Prediction market replay: Select a batch of resolved prediction markets from Polymarket, each with a clear close time and outcome. 2. Frozen corpus: Scrape a complete snapshot of public Reddit, but only allow the model to read posts timestamped before \(t_0\). This snapshot is immutable — once frozen, never updated. 3. Dual-anchor scoring: The model's predictions are compared not only to the actual outcome but also to the market price at \(t_0\) — the probability judgment humans made at that time from the same information, providing a fair baseline.
Key design: the cutoff is enforced on the backend. The model itself doesn't need to be trusted — it simply cannot access data after \(t_0\). Every document returned by retrieval has been time-filtered.
The biggest benefit of this protocol is that it never expires. Traditional evaluations break once model training data catches up to the test events. Hindcast's corpus snapshot never changes, so new models can run repeatedly on the same benchmark, and scores can be compared longitudinally.
A counterintuitive finding
After sealing the two tunnels, the researchers tested nine open-source models (from Llama-3.2-3B to Qwen3-32B). Results come in two layers:
Layer 1: Retrieval is generally still useful. 8 of 9 models saw their Brier scores drop (lower is better) with retrieval. The largest improvement was on Qwen3-32B, whose Brier score fell from 0.234 to 0.179 — a 23% reduction. Qwen3-8B's accuracy rose from 55.8% to 64.7%.
Layer 2: But only when the corpus genuinely discusses the topic. This is the paper's most interesting finding. The authors designed a coverage probe — using Qwen3-32B as judge, querying pre-\(t_0\) Reddit content five times per market and labeling on a four-level scale: strong (+2, clearly supportive), weak (+1, vaguely relevant), irrel (0, irrelevant), contra (-1, contradictory).
Findings:
- On high-coverage markets, retrieval improved most models' median accuracy by 14 percentage points.
- On low-coverage markets, retrieval actually hurt predictions. Models read a pile of speculation and chatter and got led astray.
- Training contamination is like: you yourself traveled back from the future and already know the answer.
- Retrieval leakage is like: you can search post-hoc analysis articles that state "the coup happened on X date."
- Hindcast's approach is like: locking you in a reading room with only newspapers from before June 2023 and asking, "Will there be a coup in July 2023?"
- The corpus is Reddit only. Reddit's discussion coverage is limited; many geopolitical events receive only vague chatter there.
- All nine models are open-source. Closed models (GPT-5, Claude Opus 4) have not been tested under Hindcast.
- The coverage probe itself uses an LLM as judge, which risks circularity — though using Qwen3-32B as judge avoids sharing lineage with the tested models.
- It doesn't address the possibility that models may "recall" post-\(t_0\) events from their weights — the external retrieval channel is cut, but the training-contamination tunnel relies on the model's own training-data time boundaries.
R1-Distill-Qwen-7B was the only model whose overall Brier score rose — it was most prone to over-interpreting speculative discussions.
An analogy
Imagine you're an intelligence analyst predicting whether a coup will occur in some country next month.
Under these conditions, an accurate prediction reflects genuine analytical skill.
But the paper's second finding is subtler: what's in the reading room matters too. If the newspapers contain only vague speculation ("tensions have been rising lately"), forcing an interpretation may be worse than reading nothing. Retrieval only helps when the sources contain concrete, relevant intelligence.
Why this matters
For evaluation methodology: Hindcast provides a reusable, non-expiring benchmark for LLM forecasting. As model generations update, progress can be compared on the same track.
For RAG system design: The paper reveals an overlooked failure mode — retrieval is not a panacea. When retrieved content is low-quality or weakly relevant, it doesn't just fail to help; it actively misleads the model. This is a warning for all RAG-based decision-support systems (financial forecasting, intelligence analysis, medical diagnosis).
For AI in prediction markets: Platforms like Polymarket are introducing AI traders. Hindcast serves both as an evaluation tool and a revealing look at the real boundary of current AI forecasting — under information constraints, even the best models are only roughly on par with or slightly better than market consensus.
Honest limitations
Conclusion
Hindcast does something simple but long avoided: giving LLM forecasting a clean exam environment. Its core contribution isn't a new model or algorithm, but an evaluation protocol — telling the field: your previous measurements don't count; here's another way to measure.
And its most interesting finding is neither "retrieval helps" nor "retrieval hurts" — it's that retrieval's value depends on whether the corpus actually has anything to offer. Rummaging through an empty warehouse only makes you more confused.
This may be a reminder for humans too: when information is insufficient, admitting you don't know is closer to wisdom than forcing an answer.
---
Paper: Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters Authors: Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou (Arizona State University) Author page: https://www.jacobdineen.com/publications/hindcast-2026