Self-Speculation: How LLM Agents Hide Tool-Call Latency by Predicting Themselves
The Problem: GPUs Idle While Waiting for Tool Results
When an LLM search agent answers a query, it thinks for a few seconds, decides to call web_search("Nature 2026 physics counterintuitive"), and then waits. Search APIs return in 300 ms to 2 s; the model does nothing during that window.
This is systemic. The UCSB and LinkedIn paper *Speculate While You Reason* (July 2026) notes that a large fraction of LLM agent reasoning time is spent waiting on remote calls: search, DB queries, code execution, sub-LLM calls. Token generation runs at millisecond scale; tool calls run at hundreds-of-milliseconds-to-seconds scale—one order of magnitude slower.
The inspiration is CPU branch prediction. In the 1990s, CPUs solved branch-stall by guessing the next direction, fetching instructions down that path, and discarding them on a miss. Modern branch predictors exceed 95% accuracy and sustain the entire pipeline.
The paper's core insight: port branch prediction into LLM tool calls. While the agent is still reasoning, predict the next tool name and arguments and dispatch the call early. On a hit, the result is already buffered. On a miss, discard and wait for the real call.
The key question is who predicts.
The External Speculator Problem: speculator–agent gap
The naive approach pairs a small speculator with the main agent. With a 4B agent and a 0.6B speculator, the small model sees the trajectory prefix and predicts the next call—mirroring how simple and complex predictors cooperate in CPUs.
The numbers are disappointing:
| Speculator | MuSiQue Hit@1 | τ-bench Hit@1 | |---|---|---| | Qwen3-0.6B (external) | 4.3% | 16.8% | | Qwen3-1.7B (external) | 14.7% | 25.4% | | Qwen3-4B self | 25.3% | 37.2% |
The 4B model predicting itself beats the 1.7B external speculator by more than 2×. The authors call this the speculator–agent gap: an external small model is trained as a generic tool-use assistant and predicts what *a typical* agent would do, but the deployed 4B agent has its own habits—search-then-filter preferences, parameter shapes like splitting dates into year/month, quirks visible only at deployment.
Overhead makes it worse. The external speculator needs its own weights, its own KV cache, and parameter swapping on the GPU. In the paper's measurements, adding a 1.7B external speculator pushed inference time from 8.3 s to 33.1 s and memory from 8.7 GB to 12.8 GB. To hide latency, the system introduced more latency—a textbook engineering irony.
Key Design: One Model, Two Modes
The proposed solution is counter-intuitive: let the agent be its own speculator.
The same 4B model, the same parameters, plus a "speculation suffix"—a fixed prompt instructing the model to enter prediction mode and emit the next tool call without reasoning. The trajectory prefix stays identical; only the suffix differs. KV cache is fully reused. No extra model, no extra memory.
The elegance: the speculator sees exactly what the agent sees. No speculator–agent gap, because the speculator *is* the agent. It knows its own preferences and parameter habits.
The concern: can one model simultaneously solve problems *and* predict its own next tool call without the two objectives fighting?
Joint Agent-Speculator RL: Taking Turns
The technical contribution is joint agent-speculator reinforcement learning with shared data but separated updates.
Each round:
1. Sample a batch of queries, run the agent mode to produce G=8 trajectories. 2. These trajectories are agent training data (reward from final-answer correctness) *and* speculator training data—each tool-call prefix plus the speculation suffix is speculator input; the actual call is the target. 3. Alternate updates: train the agent for one step, the speculator for the next, and repeat.
A subtle but important detail: the speculator target is on-policy. It predicts not a fixed ground truth but what the current agent, under its current policy, will actually call. Whenever the agent changes, the target shifts with it, preventing the speculator from fitting a stale snapshot of agent behavior.
Reward design treats the structured call a = (name, args) as two parts:
- Name match:
R_name = 1[name_hat = name] - Arguments: token-F1 per key, macro-averaged, conditioned on the name being correct
- Combined:
R_sp = R_name × R_args - Only tested at 4B scale; optimization dynamics may differ for larger models.
- Only search QA and conversational tool calling covered; code execution, web navigation, and multi-agent setups untested.
- Only read-only tools supported.
- Multi-step speculation: predicting more than one tool call ahead may help in chains of calls, though accuracy likely decays exponentially.
- Learning to abstain: when the agent's next move is highly uncertain, speculating is worse than declining. Requires an explicit "abstain" action and reward shaping.
- Multi-agent speculation: one agent predicting another's next call—what form does the speculator–agent gap take there?
- arXiv: 2607.25816
- HTML: https://arxiv.org/html/2607.25816v1
- Authors: Jiabao Ji, Yujian Liu, Li An, Rohit Jain, Gungor Polatkan, Siyu Zhu, Shiyu Chang
- Affiliations: UC Santa Barbara, LinkedIn Inc.
- Code: not yet released (no GitHub link in the paper)
A wrong name zeroes out the score; a correct name with partial arguments still gives signal. This aligns with real reuse requirements (exact-match reuse) while providing gradient signal for "close" predictions.
Three Stabilization Tricks
The biggest joint-training risk is mode interference—agent updates destroying speculator progress and vice versa. Three techniques matter:
SFT warmup: Before RL, run supervised fine-tuning on a small set of successful trajectories and speculation examples so the model learns that the speculation suffix triggers structured tool-call output. Without this, RL early steps waste capacity on format learning and speculation reward is unstable.
Optimizer reset: Reset optimizer state when switching modes. Adam's momentum and adaptive learning rate are estimated from historical gradients; momentum accumulated in agent mode pushed onto speculator updates steers them wrong. Resetting lets each mode accumulate momentum from zero.
Alternating schedule: Avoid 1:1 alternation. Give the speculator longer contiguous update blocks:
| Schedule | Avg Hit@1 | Avg Success | |---|---|---| | 1:1 | 31.8 | 10.3 | | 2:2 | 41.8 | 22.9 | | 4:4 | 50.3 | 24.4 | | 4:8 | 55.2 | 26.1 |
1:1 is worst; 4:8 is best. The speculator needs sustained signal to track the agent's current call distribution; short blocks keep it chasing rather than learning—analogous to how humans acquire skills better with focused practice than frequent switches.
Headline Numbers: 44.1 → 61.2
Evaluation covers agentic SearchQA (multi-hop QA + search tools) and τ-bench (airline/retail conversational tool calling):
| Model | Stage | Avg Hit@1 | Avg Task Success | |---|---|---|---| | Qwen3-4B | Base self | 29.1 | - | | | SFT warmup | 44.1 | - | | | Joint RL | 61.2 | 26.6 → 27.7 | | Qwen3.5-4B | Base self | 33.2 | - | | | SFT warmup | 48.9 | - | | | Joint RL | 66.3 | 49.2 → 50.6 |
Observations:
1. SFT warmup alone lifts Hit@1 from 29% to 44%: once the model learns the suffix → structured call mapping, its latent agent knowledge becomes usable. This confirms the speculator–agent gap—the 4B model already had the capability; it just needed to be told "now you are the speculator."
2. Joint RL adds another 17 points, 44.1 → 61.2, from on-policy training that keeps the speculator aligned with the agent's current preferences.
3. Task success barely changes (26.6 → 27.7, 49.2 → 50.6). This is the critical result: speculation gains do not cost task capability. Both roles coexist in one model.
4. Cross-domain transfer: a speculator trained on SearchQA still improves Hit@1 on τ-bench. The skill transfers—what the model learns is a meta-ability, "given this prefix, what tool will I call," not just task-specific habits.
Engineering Boundaries: What Transfers, What Doesn't
The paper is explicit: the method applies only to read-only tools. Search, retrieval, DB queries, read-only APIs—mispredictions are discarded with no side effects. Order placement, database writes, email sends, and irreversible workflows are unsafe for speculative execution and require dry-run modes, transaction rollback, or human confirmation.
This mirrors CPU branch prediction exactly. CPUs only speculatively execute pure-compute instructions, never writes to memory or I/O triggers, because those have side effects. Speculative execution is safe only when operations are reversible—a principle that holds from CPUs to LLM agents.
KV cache reuse is the resource-saving core. External speculators need two caches (agent + speculator), doubling memory. Self-speculation reuses the agent's prefix cache and only computes the cache for the short suffix tokens, making overhead negligible. Consistent with a "shared parameters, separated modes" philosophy: division of labor does not require division of the model.
Concept Lineage: A New Member
This paper extends a recurring theme of "solving problems by changing the level": octopus RNA editing, slime-mold externalized memory, SOPHIA role division, EvoThink atomic reasoning, Möbius RoPE topological intervention. The new member: don't treat tool-call latency as a fixed cost—switch to the prediction layer and hide it. More precisely, don't make tools faster; make the wait irrelevant. The serial "reason → call → wait → continue" becomes parallel "reason while pre-executing the next step."
A deeper read: prior work in this lineage argued "division beats unification." This paper looks like a counter-example (one model for both roles), but it is actually a more refined form: shared parameters, separated modes; shared data, separated updates. Division of labor does not require division of household. Roles can split; the carrier can stay unified—finer than both "fully divided" and "fully unified."
The on-policy design choice also articulates a general principle: a predictor's target should come from the predicted system's current behavior, not a fixed historical snapshot. This is the same idea as online learning in recommender systems and policy rollouts in RL—the predictor must co-evolve with the system it predicts. Wherever you forecast a moving target, this distinction holds.
Limitations and Open Directions
The paper lists:
Worth pursuing:
These questions outline a broader research space: speculative execution for LLM agents is just beginning. CPUs took 30 years to push branch prediction from 80% to 99%; the LLM agent story may still be on its first page.