73.9% of Queries Can Hide Latency: When Streaming RAG Actually Helps

You are having a voice conversation with an AI assistant: "Will it rain tomorrow in Beijing—"

Before you finish the sentence, the AI has already started querying the weather. By the time your "—?" lands, the answer is ready. This is the promise of Streaming RAG: fire off tool queries in parallel while the user is still speaking, "hiding" retrieval latency inside the time the user spends typing or talking.

Sounds great. But there is a fundamental question almost no one has asked: how many queries can actually benefit from this kind of speculation?

Elroy Galbraith of SMG Labs published a paper that does something plain but important: on the CRAG benchmark (1,371 validation questions), it precisely measures the distribution of "tool-intent stabilization" points, then derives a model-agnostic theoretical upper bound H that tells you how much streaming RAG can actually save you.

What is "tool-intent stabilization"?

The key concept is the tool-intent stabilization point: the moment in the input stream when the retrieval results of a speculative query have converged to include the answer.

Example. Suppose the user asks: "Who won the 2024 Nobel Prize in Physics?" The information is distributed unevenly across the sentence:

• "2024" → don't yet know what's being asked
• "Nobel" → could be Peace, Literature, Physics...
• "Physics" → the correct query can now be determined: retrieve "2024 Nobel Prize Physics"
• "who won" → redundant, doesn't change the query

Here the stabilization point comes after "Physics." If the user speaks at 3 words per second and the tool query takes 600 ms, there's roughly 1–2 seconds between stabilization and the end of the utterance — enough to hide the entire 600 ms query.

But not every question cooperates. Consider: "Why is the sky blue?" — the query intent can't be fixed until "blue," which is nearly the last informative word. The stabilization point is late, leaving almost no time to hide latency.

Two kinds of stabilization, two fates

The paper distinguishes two stabilization points:

1. Self-consistency stabilization (t_sc): the top-1 retrieval result of the speculative query stops changing. This is "lexical-level" stability — the BM25-retrieved documents no longer change. 2. Sufficiency stabilization (t_suf): the speculative query retrieved a document containing the gold answer. This is "semantic-level" stability — you actually have the answer.

The gap between them is huge. On CRAG:

• Only 21.3% of questions have gold evidence appearing verbatim and retrievable by BM25 — these stabilize early on t_suf, and 95.2% can be streamed
• The remaining 78.7% can only rely on t_sc (lexical stability), and streaming gains are much less certain

Where does 73.9% come from?

At the "realistic operating point" (L = 600ms tool latency, δ = 3 words/sec input speed, θ = 0.8 coverage threshold):

73.9% of queries can hide at least 80% of the tool latency.

That number looks optimistic, but the paper honestly decomposes it:

• 21.3% of questions (gold evidence appears verbatim): 95.2% streamable → near-perfect benefit
• 78.7% of questions (no gold guarantee): can only rely on lexical stability → streaming benefit uncertain

In other words, 73.9% is a "blended number." If you only look at the signal "top-1 retrieval has stabilized," you get this optimistic figure. But "top-1 stable" does not mean "correct answer in hand" — the retrieved document may have been wrong from the start, just consistently wrong.

A model-agnostic upper bound

The paper's most elegant contribution is deriving a theoretical upper bound H:

H = fraction of tool latency hideable = f(L, δ, φ)

where L is tool latency, δ is input speed, and φ is the stabilization score (stabilization point position / total input length).

This upper bound is model-agnostic — no matter which LLM, retriever, or speech recognizer you use, streaming RAG's benefit cannot exceed H. The constraint is informational: if the query intent doesn't stabilize until word N, the input time for the first N words is your entire latency-hiding budget.

The bound also has a practical use: pre-deployment feasibility assessment. Given your tool latency L and user input speed δ, you can compute H directly and judge whether streaming RAG is worth the investment for your scenario. If H < 0.3, probably not.

What predicts early stabilization?

The paper also runs regression analysis to find which query properties predict early vs. late stabilization:

• Queries with high entity density stabilize early — once entity words appear, query intent is clear
• Questions with temporal/spatial qualifiers up front stabilize early — "2024 Beijing" stabilizes earlier than "in Beijing in 2024"
• Open-ended questions ("why," "how") stabilize late — information is spread across the sentence
• Multi-hop questions stabilize latest — they need multiple queries, each with a different stabilization point

Practical implication: if your system mainly serves entity-dense queries (e.g., knowledge Q&A), streaming RAG pays off; for open-ended reasoning queries ("analyze..."), it doesn't.

No training needed, runs on CPU

One charming aspect of this paper: zero training, pure analysis. The entire study is reproducible on an ordinary CPU — no GPU, no model training. It just measures a distribution precisely, derives an upper bound, and validates with a working pipeline.

In 2026, when everything is a large model, this kind of plain measure-model-validate work has a classical scientific beauty. It doesn't invent a new method or chase SOTA; it answers the question everyone should ask first: under what conditions does this method work, and under what conditions does it fail?

Takeaways for practitioners

If you're considering deploying streaming RAG:

1. Measure the stabilization distribution first: measure t_sc and t_suf on your actual query distribution; don't rely on aggregate numbers 2. Distinguish the two stabilizations: t_sc gives an optimistic estimate, t_suf a conservative one — the truth lies in between 3. Query type determines benefit: entity-dense queries benefit most; open-ended reasoning queries least 4. The upper bound H is your friend: use it for pre-deployment feasibility assessment, so you don't invest and then discover the gains are insufficient

73.9% is a number that makes you want to ship immediately. But taken apart, only 21.3% of questions get a "near-perfect" streaming experience. For the remaining 78.7%, streaming RAG's benefit depends on how much you trust the "top-1 stable" signal.

The golden rule of latency optimization: measure, don't assume. This paper hands you the measuring tool.

---

Paper: When Does Streaming Tool Use Help? Characterizing Tool-Intent Stabilization in Streaming Retrieval-Augmented Generation Author: Elroy Galbraith (SMG Labs) arXiv: https://arxiv.org/abs/2606.20113 Code: https://github.com/elroy-galbraith/stablize_CRAG