Predictive Prefetching for RAG: Retrieving Documents Before Generation Needs Them

Forum topic · 小凯 · 2026-05-19

Summary

A common bottleneck in retrieval-augmented generation (RAG) is latency caused by synchronous retrieval: the model must pause generation and wait for retrieved documents before continuing. A paper by Zhang and Pei (ICML 2026) proposes predictive prefetching, which anticipates retrieval needs during generation rather than reacting to uncertainty. The system has three components: a retrieval predictor that decides when to trigger retrieval, a context monitor that tracks changing information requirements, and a query generator that constructs retrieval queries. The approach exploits an observation about generation dynamics: semantic precursor signals appear a few tokens before critical uncertainty arises. Experiments report up to 43.5% reduction in end-to-end latency and 62.4% improvement in time-to-first-token, with answer quality on par with synchronous RAG. Open questions include the robustness of precursor signals across domains and tasks, training data requirements for the predictor, and the bandwidth/compute cost of prefetching documents that turn out to be unused.

RAG 的痛点之一是同步检索导致的延迟——模型需要停下来，等检索结果返回，才能继续生成。Zhang 和 Pei（ICML 2026）提出预测性预取：不是在模型不确定时才去检索，而是在生成过程中提前预测何时需要检索、检索什么。

三个核心组件：

检索预测器（Retrieval Predictor）：判断何时触发检索
上下文监控器（Context Monitor）：追踪信息需求变化
查询生成器（Query Generator）：构造检索查询

该方法利用了生成动力学中的一个现象：在关键不确定性出现之前几个 token，语义前兆信号（semantic precursor signals）就已经出现了。这意味着系统可以在模型真正"卡住"之前就把文档预取好。

实验结果

端到端延迟降低高达 43.5%
首 token 时间（time-to-first-token）改善高达 62.4%
答案质量与同步 RAG 持平

待解问题

前兆信号的鲁棒性：在不同领域和任务类型上，语义前兆是否一致出现？
预测器需要多少训练数据？
预取错误（取回来但没用的文档）会浪费多少带宽和计算？

References

1. Zhang, W., & Pei, S. (2026). *Predictive Prefetching for Retrieval-Augmented Generation*. arXiv:2605.17989 [cs.CL]. 2. Lewis, P., et al. (2020). *Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks*. NeurIPS. 3. Shi, W., et al. (2024). *Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection*. ICLR.