Self-Speculation for LLM Agents: Hiding Tool-Call Latency Like CPU Branch Prediction

The scenario: your GPU idles while waiting for search results

Suppose you build an LLM search agent and ask it: "What was the most counterintuitive physics finding in Nature in 2026?" It thinks for a few seconds, decides to call web_search("Nature 2026 physics counterintuitive")—and then waits. The search API takes anywhere from 300 ms to 2 seconds, and the model does nothing in the meantime.

This is not an isolated case. A team from UCSB and LinkedIn, in the July 2026 paper *Speculate While You Reason*, computed that a large share of LLM agent inference time is spent waiting on tool returns—search, database queries, code execution, calls to sub-LLMs, all remote calls. Token generation is millisecond-scale; tool calls are hundreds of milliseconds to seconds—an order of magnitude apart.

This recalls an old problem: in a CPU pipeline, a branch instruction must wait for its condition to be computed before knowing where to go next. The 1990s solution was branch prediction—guess the direction before the condition resolves and speculatively fetch and execute that path's instructions. Correct guess: free time. Wrong guess: throw it away and redo. Modern CPUs exceed 95% branch prediction accuracy, and this mechanism underpins pipeline efficiency.

The paper's core insight in one sentence: bring branch prediction into LLM tool calls. While the agent is still reasoning, predict which tool it will call next and with what arguments, and issue that call early. If the guess is right, the result is already ready; if wrong, discard it and wait for the real call.

But there is a key question: who does the guessing?

The problem with external predictors: the speculator-agent gap

The most intuitive approach is a small "predictor" model. A 4B main agent paired with a 0.6B model that sees the current trajectory prefix and predicts the next tool call—like combining simple and complex predictors inside a CPU.

The paper tested this first. The results were disappointing:

| Speculator | MuSiQue Hit@1 | τ-bench Hit@1 | |---|---|---| | Qwen3-0.6B (external) | 4.3% | 16.8% | | Qwen3-1.7B (external) | 14.7% | 25.4% | | Qwen3-4B self (guessing itself) | 25.3% | 37.2% |

The 4B guessing itself is more than twice as good as the external 1.7B. Why? The paper names this the speculator–agent gap: an external small model is trained as a "generic tool-use assistant" and predicts what a *typical* agent would do, but your deployed 4B agent has its own habits—it prefers searching then filtering, it splits date arguments into year and month fields, it has query preferences. These habits form only at deployment time; the external model knows nothing about them.

Worse, there is overhead. An external speculator needs its own model weights, its own KV cache, and parameter swaps on GPU at each switch. Measured: an external 1.7B speculator raised inference time from 8.3 s to 33.1 s and memory from 8.7 GB to 12.8 GB. To hide latency, more latency was introduced first—classic engineering irony.

Key design: one model, two modes

The paper's solution is almost counterintuitively simple: let the agent be its own speculator.

Concretely: the same 4B model, the same weights, plus a "speculation suffix"—a fixed prompt telling the model "enter prediction mode; do not reason, directly output the next tool call." The model sees the same trajectory prefix, just with this suffix appended. The KV cache is fully reused—no extra model, no extra memory.

The most elegant part: the speculator's context is exactly the agent's context. There is no speculator-agent gap, because the speculator *is* the agent. It knows its own preferences, its own argument conventions, when it will call what.

But can one model both solve tasks and predict its own next calls? In RL training, optimizing one objective often degrades the other.

Joint Agent-Speculator RL: taking turns

The core technical contribution is joint agent-speculator reinforcement learning: shared data, separated updates.

Each training round: 1. Sample a batch of queries; run G=8 trajectories in agent mode. 2. Those trajectories serve both as agent training data (reward from final answer correctness) and as speculator training data—each prefix before a tool call, plus the speculation suffix, is the speculator's input; the tool the agent actually called is the target. 3. Alternating updates: one step trains the agent, the next trains the speculator, back and forth.

A subtle detail: the speculator's target is on-policy—it predicts not some fixed "gold answer," but *what the current agent, under the current policy, would call*. Every time the agent changes, the speculator's target changes too. This avoids fitting an outdated snapshot of agent behavior.

The reward is carefully designed. Tool calls are structured a = (name, args), scored in two parts:

• Tool name: R_name = 1[name_hat = name]
• Arguments: given a correct name, token-F1 per argument key, macro-averaged
• Final: R_sp = R_name × R_args

Wrong name, all wrong; correct name earns partial credit on arguments. This matches the exact-match reuse requirement—pre-executed results are only reusable when name and arguments match exactly—but partial credit during training gives a gradient signal for "closeness."

Three stabilization tricks: don't let the objectives fight

The biggest risk in joint training is mode interference. Three tricks, each essential:

SFT warmup: before RL, briefly SFT on successful trajectories and speculation examples so the model learns the format "see speculation suffix → output structured tool call." Without it, early RL is wasted on format learning and the speculation reward is unstable.

Optimizer reset: reset optimizer state when switching modes. Adam's momentum and adaptive learning rates come from historical gradients; agent-mode momentum carried into speculator mode pushes updates the wrong way. Resetting lets each mode accumulate momentum from zero.

Alternating schedule: don't alternate 1:1; give the speculator longer continuous blocks. Four schedules tested:

| Schedule | Avg Hit@1 | Avg Success | |---|---|---| | 1:1 | 31.8 | 10.3 | | 2:2 | 41.8 | 22.9 | | 4:4 | 50.3 | 24.4 | | 4:8 | 55.2 | 26.1 |

1:1 is worst, 4:8 best. The speculator needs sustained signal to track the agent's current call distribution; short blocks keep it "chasing" rather than "learning."

Key numbers: from 44.1 to 61.2

Main experiments on agentic SearchQA (multi-hop QA + search tools) and τ-bench (conversational tool calling in airline/retail scenarios):

| Model | Stage | Avg Hit@1 | Avg Task Success | |---|---|---|---| | Qwen3-4B | Base self | 29.1 | - | | | SFT warmup | 44.1 | - | | | Joint RL | 61.2 | 26.6 → 27.7 | | Qwen3.5-4B | Base self | 33.2 | - | | | SFT warmup | 48.9 | - | | | Joint RL | 66.3 | 49.2 → 50.6 |

Observations:

1. SFT warmup alone lifts speculation from 29% to 44%—once the model learns "see suffix, output structured call," its existing agent knowledge transfers directly. This confirms the speculator-agent gap: the 4B's self-speculation ability was always there; nobody had told it "you are now the speculator."

2. Joint RL adds another 17 points (44.1 → 61.2), from on-policy training tracking the agent's current preferences.

3. Task success barely changes (26.6→27.7; 49.2→50.6)—the crucial result: speculation gains do not come at the cost of task ability. One model can wear both hats.

4. Cross-domain generalization: a speculator trained on SearchQA still improves Hit@1 on τ-bench, suggesting a meta-ability—"how I, as an agent, call tools given a certain prefix"—that transfers across tasks.

Engineering insight: what transfers, what doesn't

The paper is honest about boundaries: the method applies only to read-only tools. Search, retrieval, database reads, read-only APIs—wrong guesses can just be discarded. But ordering products, writing databases, sending email, triggering irreversible workflows: speculative execution there is dangerous, requiring dry-run modes, transactional rollback, human confirmation.

This mirrors CPU branch prediction exactly: CPUs only speculate on "pure computation," never on writes or IO with side effects. The safety boundary of speculation is reversibility—a principle that carries from CPUs to LLM agents.

Another engineering detail: KV cache reuse is what makes self-speculation cheap. An external speculator needs two KV caches, doubling memory. Self-speculation reuses the agent's prefix cache, computing only the few suffix tokens. Consistent with the philosophy: division of labor doesn't require separation.

Personal reflections: a new member of a conceptual lineage

This paper made me revisit a lineage I've been tracking—"solving the problem at a different level." From octopus RNA editing to slime mold externalized memory, from SOPHIA's division of labor to EvoThink's atomic reasoning, to Möbius RoPE topological intervention—each case avoids optimizing the original problem directly.

This paper is member #12: stop treating tool-call latency as an unavoidable cost—move to the prediction level and hide the latency. It doesn't make tools faster; it makes waiting irrelevant, turning the serial "reason → call → wait → reason" into parallel "reason while pre-executing the next step."

Deeper still: I've argued division of labor beats unification. This paper looks anti-division—merging speculator and agent—but it's actually a finer form of division: shared parameters, separated modes; shared data, separated updates. Division of labor doesn't require separate households. That's finer than both "complete division" and "complete unification."

Finally, the on-policy choice suggests a general principle: a predictor's target should come from the predicted system's current behavior, not a fixed historical snapshot—the same idea as online learning in recommender systems and policy rollout in RL. *To predict something that changes, the predictor must change with it.* The speculator's target isn't "what the agent should call" but "what this version of the agent will call."

Limitations and open roads

The paper's own limitations:

• Only 4B scale tested; optimization dynamics may differ at larger scales
• Only search QA and conversational tool calling; code execution, web navigation, multi-agent scenarios untested
• Read-only tools only

Directions worth pursuing:

• Multi-step speculation? Predicting two or three calls ahead could pay off more, but accuracy decays exponentially.
• Can the speculator learn to abstain? Under high uncertainty, guessing is worse than saying "not this time"—requiring an abstain option and reward design.
• Speculation across agents? Predicting another agent's next call—what form does the speculator-agent gap take?

No answers yet, but they point to a wide research space: speculative execution for LLM agents has just begun. CPUs took 30 years to push branch prediction from 80% to 99%; the LLM agent story may only be on its first chapter.

Paper links

• arXiv: 2607.25816
• HTML: https://arxiv.org/html/2607.25816v1
• Authors: Jiabao Ji, Yujian Liu, Li An, Rohit Jain, Gungor Polatkan, Siyu Zhu, Shiyu Chang
• Institutions: UC Santa Barbara, LinkedIn Inc.
• Code: not yet released (no GitHub link in the paper)

*Member #12 of the conceptual lineage: don't optimize latency itself—move to the prediction level and make it disappear. Division of labor doesn't require separation: shared parameters, separated modes—a design philosophy finer than both complete division and complete unification.*