Self-Speculating Agents: Eliminate Tool-Call Latency by Predicting Your Own Next Step

Forum topic · ✨步子哥 · 2026-08-03

Summary

In agentic LLM systems, most wall-clock time is spent waiting on remote tool APIs, not on model inference. Industry tool-call speculation uses a small draft model to predict the agent's next call, but a Speculator-Agent Gap arises because a separate model cannot faithfully mirror the agent's decision boundary. This article summarizes Ji et al. (2026, arXiv:2607.25816), who show that on Qwen3-4B the agent itself predicts its next tool call far better than Qwen3-0.6B or 1.7B external speculators (Hit@1 25.3 vs 14.7 on MuSiQue), with zero extra GPU memory. Their self-speculating agent runs in two modes that share prefix KV cache and a fixed speculative suffix. Joint Agent-Speculator RL with SFT warm-up, optimizer resets, and a 4:8 schedule lifts Hit@1 from 44.1 to 61.2 on Qwen3-4B and 48.9 to 66.3 on Qwen3.5-4B while task success rate holds or improves. Limitations: read-only tools, untested scales, single-agent tasks.

Key points

Agent latency is dominated by network I/O, not inference. Nichols et al. (2025) and Hooper et al. (2026) show agents spend most wall-clock time waiting on tool APIs. Token generation is fast; tool calls are slow.
Tool-call speculation has a fatal flaw: the Speculator-Agent Gap. A small external draft model cannot precisely predict a larger agent's next call. On MuSiQue and τ-bench, Qwen3-0.6B hits only 14.7/25.4 Hit@1, and Qwen3-1.7B hits 12.3/22.1, both with extra GPU memory.
The agent is its own best speculator. Qwen3-4B predicting itself reaches 25.3/37.2 Hit@1 with zero extra weights or KV cache. Same decision boundary means no guessing gap.
One model, two modes. Self-speculating agent shares prefix KV cache between Agent mode (reasons, calls tools) and Speculator mode (predicts next call via a fixed task-agnostic suffix: <think> Okay, let's see. The user provided what I need. I'll look it up. The next step is to make the tool call. ).
Joint Agent-Speculator RL training. Three stabilizers are required:
SFT warm-up to teach the prediction-mode output format before RL
Optimizer reset between mode switches to prevent momentum interference
Alternating 4:8 schedule (4 Agent steps, 8 Speculator steps); a 1:1 schedule collapses to 10.3% task success
Headline results. Hit@1 climbs from 44.1 to 61.2 on Qwen3-4B (+38.8%) and from 48.9 to 66.3 on Qwen3.5-4B. Task success rate does not drop and even improves slightly (26.6 → 27.7; 49.2 → 50.6).
Cross-domain transfer is asymmetric. Trained on SearchQA, evaluated on τ-bench: Hit@1 still rises (35.6 → 45.6), but task success drops (21.6 → 17.6). Prediction is a generalizable skill; task solving is domain-specific.
Design principle (unify vs divide). When two sub-tasks must share a decision boundary, unify; when they need different capabilities, divide. Self-speculation is the classic example where unification wins.
Analogy to CPU pipelining. Overlapping inference and tool I/O parallels instruction pipelining. The general principle: in any system with serial waits, ask "what can be done during the wait?"
Limitations.
Only safe for read-only tools (search, retrieval, DB queries); write actions need dry-runs or rollback
Validated only at 4B scale on Qwen3 / Qwen3.5
Tasks limited to multi-hop QA and conversational tool use

Open questions

Can the speculator also predict *which* of 10 tools will be chosen, not just the arguments?
Does parallel multi-speculator voting (Hit@k) further improve coverage?
Can feeding predictions back to the agent ("you predicted X, confirm?") improve decision quality?

Source

Paper: Ji, Liu, An, Jain, Polatkan, Zhu, Chang (UC Santa Barbara + LinkedIn). *Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent–Speculator RL.* arXiv:2607.25816. https://arxiv.org/abs/2607.25816

Why it matters

Self-speculation reframes agents from passive waiters into systems that overlap reasoning with I/O, without doubling memory. The deeper insight: no external model can predict you as well as you can predict yourself, because accurate self-prediction requires sharing the exact decision boundary.