[论文] Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

论文概要

研究领域: NLP
作者: Xiao Ye, Jacob Dineen, Evan Zhu, Shijie Lu, Kevin Song, Ben Zhou
发布时间: 2026-07-15
arXiv: 2607.14051

中文摘要

预测者通过回测评估，重放已解决问题并评分系统在结果已知前会分配的概率。对于LLM，两个渠道将答案泄漏到此测试中。可检索的模型可显示事件后撰写的报告，将预测转化为查找，且每个新模型在更接近事件的数据上训练，因此去年模型面临未来问题位于今年训练数据内。无论哪种方式，测试评分回忆却声称评分预见。我们引入Hindcast，通过将模型评分置于选定过去日期\(t_0\)之前来关闭两个泄漏，在结果存在于任一渠道之前。Hindcast重放已解决的Polymarket预测市场对抗冻结的公共Reddit快照，让模型只读取\(t_0\)前撰写的帖子，并将每个预测与发生的事情和\(t_0\)时市场自身价格（本身是从相同过去信息做出的人类预测）评分。由于截止按市场设置且快照永不改变，评估随着模型改进在新市场上重新运行，不会过时。一旦泄漏关闭，检索仍帮助大多数模型，但仅在Reddit事先讨论事件的地方。在档案仅携带猜测的地方，检索有害。

原文摘要

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date \(t_0\), before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Red...

自动采集于 2026-07-17

#论文 #arXiv #NLP #小凯