Self-Speculation for LLM Agents: Hiding Tool-Call Latency by Letting Agents Predict Themselves
*Based on a Chinese forum post discussing the paper "Speculate While You Reason" (UC Santa Barbara & LinkedIn, July 2026, arXiv:2607.25816).*
The problem: the GPU idles while waiting for tools
Consider an LLM search agent answering a question. It decides to call web_search(...) and then waits — search APIs take 300 ms to 2 seconds, while token generation takes milliseconds. The paper quantifies this: a large fraction of an agent's wall-clock reasoning time is spent waiting on remote tool calls (search, database queries, code execution, sub-LLM calls).
The core insight borrows from CPU branch prediction: don't wait — speculatively guess the agent's next tool call while it is still reasoning, and issue it early. If the guess is right, results are ready when the real call arrives; if wrong, discard it. Modern CPUs exceed 95% branch-prediction accuracy this way.
Why external speculators fail: the speculator–agent gap
The obvious approach — a small external model predicting the big agent's calls — underperforms badly:
| Speculator | MuSiQue Hit@1 | τ-bench Hit@1 | |---|---|---| | Qwen3-0.6B (external) | 4.3% | 16.8% | | Qwen3-1.7B (external) | 14.7% | 25.4% | | Qwen3-4B self | 25.3% | 37.2% |
The paper names this the speculator–agent gap: a deployed agent develops idiosyncratic habits (search-then-filter, how it formats date arguments) that an external generic model cannot know. Worse, external speculators need separate weights, KV caches, and GPU parameter swaps — inference time went from 8.3s to 33.1s and memory from 8.7GB to 12.8GB in the paper's tests.
Key design: one model, two modes
The solution is self-speculation: the same 4B model, same parameters, plus a fixed "speculation suffix" prompt that switches it into prediction mode. The agent's prefix KV cache is fully reused; only a few extra tokens are computed. No gap exists because the speculator *is* the agent.
Joint Agent-Speculator RL
Training alternates updates on shared data: 1. Sample queries and roll out G=8 trajectories in agent mode. 2. Those trajectories serve both as agent training data (rewarded by final answer correctness) and as speculator training data (prefix + suffix → actual tool call as target). 3. Alternate updates between the two modes.
Speculator reward decomposes a structured call a = (name, args):
R_name = 1[name_hat = name]- Parameter token-F1 per key (macro-averaged), scored only if the name is correct
R_sp = R_name × R_args- SFT warmup alone lifts speculation from 29% to 44% — the capability was always there; the model just needed to be told "you are now a speculator."
- Joint RL adds ~17 points via on-policy alignment with the agent's current calling distribution.
- Task success is essentially unchanged — speculation training does not cost agent capability; the dual role can coexist.
- Cross-domain transfer: a speculator trained on SearchQA still improves Hit@1 on τ-bench, suggesting a general "how would I call tools given this prefix" meta-capability.
- Only 4B-scale models tested
- Only search QA and conversational tool calling (τ-bench); no code execution, web navigation, or multi-agent settings
- Read-only tools only
- Multi-step speculation: predicting two or three calls ahead (accuracy decays exponentially)
- Abstention: letting the speculator opt out when behavior is too uncertain
- Multi-agent speculation: one agent predicting another's calls — what form would the speculator–agent gap take?
- arXiv: 2607.25816
- HTML: https://arxiv.org/html/2607.25816v1
- Authors: Jiabao Ji, Yujian Liu, Li An, Rohit Jain, Gungor Polatkan, Siyu Zhu, Shiyu Chang
- Institutions: UC Santa Barbara, LinkedIn Inc.
- Code: not released (no GitHub link in the paper)
This aligns training with exact-match reuse (both name and args must match for the prefetched result to be used) while still giving partial gradient signal.
Three stabilization tricks
1. SFT warmup — teach the output format before RL, otherwise early RL is wasted learning format and speculation reward is unstable. 2. Optimizer reset — reset Adam state when switching modes so one mode's momentum doesn't push the other mode's updates off course. 3. Alternating schedule — give the speculator longer continuous update blocks; 4:8 (agent:speculator) beats 1:1:
| Schedule | Avg Hit@1 | Avg Success | |---|---|---| | 1:1 | 31.8 | 10.3 | | 2:2 | 41.8 | 22.9 | | 4:4 | 50.3 | 24.4 | | 4:8 | 55.2 | 26.1 |
Main results
| Model | Stage | Avg Hit@1 | Avg Task Success | |---|---|---|---| | Qwen3-4B | Base self | 29.1 | – | | | SFT warmup | 44.1 | – | | | Joint RL | 61.2 | 26.6 → 27.7 | | Qwen3.5-4B | Base self | 33.2 | – | | | SFT warmup | 48.9 | – | | | Joint RL | 66.3 | 49.2 → 50.6 |
Key observations:
Engineering boundaries
The method applies only to read-only tools — search, retrieval, DB reads, read-only APIs — where a wrong guess can simply be discarded. Writes, purchases, emails, and irreversible workflows would need dry-run modes, transaction rollback, or human confirmation. This is the same safety boundary as CPU branch prediction, which speculates only on pure computation, not on writes or I/O. The safety boundary of speculative execution is reversibility.
KV cache reuse is the other critical piece: the external-speculator design doubles cache memory, while self-speculation shares the agent's prefix cache and pays only for the suffix.
Discussion points from the original post
The author frames this as an instance of "solving a problem at a different level" — not making tools faster, but making waiting irrelevant by converting serial "reason → call → wait → reason" into parallel speculation. It also refines the usual "division of labor beats unification" heuristic: division of labor doesn't require separate carriers — parameters shared, modes separated; data shared, updates separated. And the on-policy design generalizes: a predictor of a dynamic system must track its target's current behavior, not a stale snapshot.
Limitations and open questions
From the paper:
Open directions raised in the post:
Paper links
*Member #12 in the post's conceptual lineage: don't optimize the latency itself — move to the prediction level and make the latency disappear. Parameters shared, modes separated: a design philosophy finer than either full division of labor or full unification.*