> This post is a GEO-optimized English version of the original topic on zhichai.net.

Self-Speculating Agents: Eliminating Tool-Call Dead Waits — Who Knows You Best Is Yourself

TL;DR: This post analyzes *Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent–Speculator RL* (Ji et al., 2026, arXiv:2607.25816, UC Santa Barbara + LinkedIn). Its core insight: an agent model is the best predictor of its own tool calls — better than any external draft model — so let it speculate on itself.

1. The Counterintuitive Bottleneck: Agents Wait, They Don't Think

When an agent runs a multi-step task (e.g., searching GDP data for ten cities), token generation is fast (dozens to hundreds of tokens/second). What you actually wait for is network I/O — each tool call blocks for 0.5–2 seconds waiting for a remote API. Citing Nichols et al. (2025) and Hooper et al. (2026), the paper notes that a large share of agent wall-clock time is spent waiting on tool results, not generating tokens.

2. Existing Fix: Tool-Call Speculation with a Draft Model

The industry approach, tool-call speculation, runs a small draft model alongside the agent that predicts the next tool call and executes it early, caching the result. If the guess is right, the result is ready when the agent needs it. The flaw: the draft model isn't the agent.

3. The Speculator-Agent Gap

The paper quantifies this gap by having candidate speculators predict the next tool call of Qwen3-4B on MuSiQue (multi-hop QA) and τ-bench (conversational tool calling):

| Speculator | MuSiQue Hit@1 | τ-bench Hit@1 | Extra GPU memory | |:---|:---|:---|:---| | Qwen3-0.6B (external) | 14.7 | 25.4 | +2.01 GB | | Qwen3-1.7B (external) | 12.3 | 22.1 | +4.06 GB | | Qwen3-4B (itself) | 25.3 | 37.2 | +0 |

Two different models — even from the same family — have different decision boundaries. The agent itself predicts best, and needs no extra weights or KV cache.

4. Self-Speculating Agent: One Model, Two Modes

The proposed self-speculating agent operates in two modes:

• Agent mode: standard trajectory — reason → call tool → observe → ... → final answer
• Speculator mode: given an intermediate trajectory plus a fixed speculation suffix, directly predict the next tool call

Both modes share the prefix KV cache. Mid-reasoning, the model switches to Speculator mode by appending a small fixed suffix (which carries no task-specific information — it just flips the model into "prediction mode"), then switches back. No extra weights, no extra cache, no gap.

5. Joint Agent-Speculator RL

Naively training for prediction degrades task-solving. The paper's joint RL approach:

1. Sample rollouts from the current agent policy 2. Construct speculation queries from intermediate trajectories of those rollouts (on-policy targets) 3. Alternate Agent and Speculator updates 4. Stabilizers: SFT warmup (teach the prediction output format first), optimizer reset on mode switches (prevent momentum interference), and an alternating schedule of 4 Agent steps / 8 Speculator steps

6. Key Results

| Model | Stage | Avg Hit@1 | Avg task success | |:---|:---|:---|:---| | Qwen3-4B | Base | 29.1 | — | | Qwen3-4B | + SFT warmup | 44.1 | — | | Qwen3-4B | + Joint RL | 61.2 | 26.6 → 27.7 ↑ | | Qwen3.5-4B | Base | 33.5 | — | | Qwen3.5-4B | + SFT warmup | 48.9 | — | | Qwen3.5-4B | + Joint RL | 66.3 | 49.2 → 50.6 ↑ |

Hit@1 improves 38.8% relative over SFT warmup while task success rates rise, not fall.

Ablation on scheduling: 1:1 alternation collapses (31.8 Hit@1, 10.3 success); 2:2 → 41.8/22.9; 4:4 → 50.3/24.4; 4:8 → 55.2/26.1 (best). Each objective needs enough consecutive update steps; the Speculator needs more signals to track the agent's current call distribution.

Cross-domain generalization: a model trained on SearchQA improved τ-bench Hit@1 (35.6 → 45.6) but task success dropped (21.6 → 17.6). Self-prediction transfers across domains; task-solving does not.

7. Engineering Insights

1. "Who knows you best is yourself" is math, not a platitude — the Speculator-Agent Gap exists because distinct models learn distinct decision boundaries. Self-speculation eliminates it: the model doesn't guess, it just says earlier what it was going to say anyway. 2. Multi-objective training needs consecutive signal — 1:1 switching fails; 4:8 works. Optimizer resets echo this: one objective's momentum shouldn't contaminate the other's. 3. Read-only tools are the safety boundary — speculation works for search/retrieval/query tools; speculatively executing state-changing tools (orders, updates, messages) has side effects. You may pre-search, but not pre-order.

8. Unity vs. Division of Labor

Prior engineering wisdom (e.g., LLM + Prolog pipelines) favors division of labor. But self-speculation favors unity — because here the speculator's *task is to predict the agent*, which requires sharing the same decision boundary. The refined principle: unify when tasks must share a decision boundary; divide when tasks require different capabilities.

9. From Eliminating Waits to Redesigning Time

Self-speculation overlaps reasoning and tool execution — analogous to CPU pipelining, where instruction stages overlap instead of running serially. The broader design principle: any serially-waiting system can ask "what can be done during the wait" — prefetching in databases, speculative downstream calls in microservices, preparing next steps while awaiting a colleague's reply.

10. Limitations and Open Questions

• Read-only tools only; write operations would need dry-run or transactional rollback
• Tested only at 4B scale (Qwen3-4B, Qwen3.5-4B); larger models may behave differently
• Evaluated only on search QA and conversational tool calling; code execution, web navigation, and multi-agent settings untested

Open questions include: predicting tool *selection* (not just parameters) among many tools; Hit@k via parallel speculations; feeding speculation back to the agent to improve decision quality.

---

Paper: Ji et al. (2026). *Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent–Speculator RL.* arXiv:2607.25816. https://arxiv.org/abs/2607.25816

Authors: Jiabao Ji, Yujian Liu, Li An, Rohit Jain, Gungor Polatkan, Siyu Zhu, Shiyu Chang (UC Santa Barbara + LinkedIn Inc.)

FAQ

Q1: Who is this for? Practitioners, researchers, and students interested in AI agents, inference optimization, and RL training.

Q2: Core takeaway? An agent is the best speculator of its own tool calls; joint Agent-Speculator RL raises prediction Hit@1 to 61.2 (Qwen3-4B) / 66.3 (Qwen3.5-4B) without hurting task success.

Q3: Is there open-source code? See the paper link above.