Hindcast: A Cheat-Proof Evaluation Protocol for LLM Forecasting via Prediction Market Replay

Forum topic · ✨步子哥 · 2026-07-16

Summary

This post explains Hindcast, an evaluation protocol from Arizona State University researchers (Xiao Ye, Jacob Dineen, et al.) designed to measure true LLM forecasting ability by blocking two contamination channels: training-data temporal leakage and retrieval of post-event information. Hindcast sends models back to a past time t0, letting them read only an immutable Reddit corpus snapshot filtered to pre-t0 content, and asks them to predict unresolved Polymarket markets. Predictions are scored against both real outcomes and market prices at t0, creating a reusable benchmark that never expires. Testing nine open-source models (Llama-3.2-3B to Qwen3-32B), the authors found retrieval generally helps (Qwen3-32B's Brier score dropped 23%, from 0.234 to 0.179), but only when the corpus genuinely discusses the event: with strong coverage, median accuracy rose 14 points; with weak coverage, retrieval actively hurt predictions. The post discusses implications for benchmark methodology, RAG system design, and AI trading on prediction markets, plus honest limitations including the Reddit-only corpus and untested closed-source models. Paper: arxiv.org/abs/2607.14051.

Hindcast: When LLMs Predict the Future, How Do You Stop Them from Peeking at the Answers?

An Awkward Situation

Suppose you ask an LLM: "Who won the 2024 US election?"

It answers correctly. But does that "correct answer" mean it can actually forecast? No. Because the model's training data extends to mid-2024 or later, the election result is already written into its weights. It isn't predicting — it's reciting.

Furthermore, even if the model itself hasn't seen the answer, if it's equipped with a RAG retrieval system, it might find post-event reports online, turning "forecasting" into "lookup."

This is the core flaw in current evaluations of LLM forecasting ability: two hidden channels smuggle the answers into the exam hall — temporal contamination of training data, and future-information leakage through retrieval systems. The test is nominally measuring foresight, but what it actually measures is recall.

Xiao Ye and Jacob Dineen et al. from Arizona State University, in the paper *Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters*, propose a clean evaluation protocol that seals both channels at once.

What Hindcast Is

The core idea in one sentence: send the model back to a past time point $$t_0$$ , let it see only information from before $$t_0$$ , and have it predict events that were unresolved at that time.

The specific approach:

1. Prediction market replay: Select a batch of resolved prediction markets from Polymarket. Each market has a clear closing time and outcome. 2. Frozen corpus: Capture a complete snapshot of public Reddit, but the model may only read posts published before $$t_0$$ . The snapshot is immutable — once frozen, it is never updated. 3. Dual-anchor scoring: The model's forecast is compared not only with the real outcome, but also with the market's own price at $$t_0$$ . Market prices are the probabilistic judgments humans made at that time based on the same information — a fair reference point.

Key design: the cutoff is enforced on the backend. The model itself doesn't need to be trusted — it simply cannot access data after $$t_0$$ . Every document returned by the retrieval system has been time-filtered.

The protocol's biggest benefit: it doesn't expire. Traditional evaluations become invalid once model training data catches up to the test events. Hindcast's corpus snapshot never changes, so new models can run repeatedly on the same testbed, and results can be compared longitudinally.

A Counterintuitive Finding

After sealing both channels, the researchers tested nine open-source models (from Llama-3.2-3B to Qwen3-32B). The results come in two layers:

Layer 1: Retrieval is still useful overall. Eight of the nine models saw their Brier scores drop (lower is better) with retrieval enabled. The biggest improvement was on Qwen3-32B, whose Brier score fell 23%, from 0.234 to 0.179. Qwen3-8B's accuracy rose from 55.8% to 64.7%.

Layer 2: But it only helps when the corpus "actually discusses the topic." This is the paper's most interesting finding. The authors designed a coverage probe — using Qwen3-32B as a judge to run five queries against pre- $$t_0$$ Reddit content for each market, labeling them on a four-level scale: strong (+2, clearly supportive), weak (+1, vaguely related), irrel (0, irrelevant), contra (−1, opposing).

Findings:

On markets with strong coverage, retrieval lifted the median accuracy of most models by 14 percentage points.
On markets with weak coverage, retrieval actually hurt forecasts. Reading piles of speculation and chatter led the models astray.

R1-Distill-Qwen-7B was the only model whose overall Brier score worsened — it was the most prone to over-interpreting speculative discussion.

An Analogy

Imagine you're an intelligence analyst asked to predict whether a coup will happen in a certain country next month.

Training contamination is: you yourself are a time traveler from the future who already knows the answer.
Retrieval leakage is: you can find after-the-fact analysis articles that state "the coup happened on X date."
Hindcast's approach is: locking you in a reading room with only newspapers published before June 2023, then asking "will there be a coup in July 2023?"

Under these conditions, if your forecast is accurate, that's genuine analytical ability.

But the paper's second finding is subtler: what's in the reading room matters too. If the newspapers contain only vague speculation ("tensions have been rising lately"), forcing yourself to interpret them may be worse than reading nothing. Retrieval only helps when the newspapers actually contain concrete, relevant intelligence.

Why This Matters

For evaluation methodology: Hindcast provides a reusable, never-expiring testbed for LLM forecasting. As model generations update, progress can be compared on the same track.

For RAG system design: The paper reveals an overlooked failure mode — retrieval is not a panacea. When retrieved content is low-quality or weakly relevant, it doesn't just fail to help; it actively misleads the model. This is a warning for all RAG-based decision-support systems (financial forecasting, intelligence analysis, medical diagnosis).

For AI applications on prediction markets: Platforms like Polymarket are introducing AI traders. Hindcast serves both as an evaluation tool and a revelation of the real boundaries of current AI forecasting — under information constraints, even the best models are only slightly better than, or on par with, market consensus.

Honest Limitations

The corpus is Reddit only. Reddit's discussion coverage is limited; many geopolitical events receive only vague chatter there.
All nine models are open-source. The performance of closed-source models (GPT-5, Claude Opus 4) under Hindcast has not been tested.
The coverage probe itself uses an LLM as judge, risking circularity — though using Qwen3-32B as judge avoids sharing provenance with the tested models.
The protocol doesn't account for models possibly "recalling" post- $$t_0$$ events from their weights — it only cuts off external retrieval; the training-contamination channel is addressed by the temporal boundaries of the models' training data itself.

Closing Thoughts

Hindcast does something simple but long avoided: giving LLM forecasting a clean exam environment. Its core contribution is not a new model or algorithm, but an evaluation protocol — telling the field: your previous tests don't count; here's another way to measure.

And its most interesting finding is not "retrieval helps" or "retrieval doesn't," but that retrieval's value depends on whether the corpus actually has substance. Rummaging through an empty warehouse only leaves you more confused.

Perhaps this is a reminder for humans too: when information is insufficient, admitting you don't know is closer to wisdom than forcing an answer.

---

Paper: Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters Authors: Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou (Arizona State University) Author page: https://www.jacobdineen.com/publications/hindcast-2026