> 📌 This is the GEO-optimized English version of a zhichai.net forum topic.
One-line takeaway: A analysis of *Speculate While You Reason* (Ji et al., 2026) — an agent's biggest latency cost is waiting on tools, and the best predictor of an agent's next tool call is the agent itself.
1. A Counterintuitive Fact: The Agent's Bottleneck Isn't Slow Thinking
Consider a multi-step task like "look up the top-10 Chinese cities by GDP, then compare population and GDP per capita." A typical agent loop looks like:
1. Search "top-10 cities by GDP" 2. Wait for results (0.5–2 s) 3. Read results, search "city A population" 4. Wait (0.5–2 s) 5. Search "city A GDP per capita" 6. Wait (0.5–2 s) 7. ...repeat ~10 times... 8. Synthesize the final answer
What you're waiting for is not the model "thinking" — token generation is fast (tens to hundreds of tokens/sec). You're waiting on network I/O: every tool call blocks on a remote API whose latency depends on network, server load, and database speed. Citing Nichols et al. (2025) and Hooper et al. (2026), the paper notes that a large share of agent wall-clock time is spent waiting for tool results, not generating tokens. This is a problem of idling, not intelligence.
2. Existing Fix: Tool-Call Speculation with a Draft Model
The industry approach, tool-call speculation, works like this:
- While the agent reasons about its next step, an external small draft model predicts "the agent will probably search for X"
- X is executed proactively and cached
- If the agent actually calls X, the cached result is used — no wait
- Target agent: Qwen3-4B
- Candidate speculators: Qwen3-0.6B, Qwen3-1.7B, and Qwen3-4B itself
- Tasks: MuSiQue (multi-hop search QA) and τ-bench (conversational tool calling)
- Metric: Hit@1 — fraction of next tool calls (name + arguments) predicted exactly
- Agent mode: standard trajectory — reason → tool call → observe → ... → final answer
- Speculator mode: given an intermediate trajectory plus a fixed speculative suffix, directly predict the next tool call
- No extra model weights
- No extra KV cache
- No speculator-agent gap — same decision boundary
- SFT warmup: teach the "prediction mode" output format before RL
- Optimizer reset on each mode switch (so agent momentum doesn't disturb speculator updates)
- Alternating schedule: 4 agent steps / 8 speculator steps — the speculator needs more consecutive signal to track the agent's evolving call distribution
Like a waiter pre-ordering the dish you're most likely to choose: correct guesses save time; wrong ones are discarded. The idea is good but has a fatal flaw.
3. The Fatal Flaw: The Speculator-Agent Gap
The paper names it the Speculator-Agent Gap: the external small model is *not you*. It doesn't know your decision style, tool preferences, or contextual habits.
A controlled experiment quantifies this gap:
| Speculator | MuSiQue Hit@1 | τ-bench Hit@1 | Extra GPU memory | |:---|:---|:---|:---| | Qwen3-0.6B (external) | 14.7 | 25.4 | +2.01 GB | | Qwen3-1.7B (external) | 12.3 | 22.1 | +4.06 GB | | Qwen3-4B (itself) | 25.3 | 37.2 | +0 |
The agent is its own best speculator — and needs no extra weights or KV cache, because "itself" is already in memory. A younger sibling from the same model family still doesn't know your mood today.
4. Self-Speculating Agents: One Model, Two Modes
The paper proposes the self-speculating agent:
Key design: both modes share the prefix KV cache. Mid-reasoning, the model switches to speculator mode by appending a short suffix to the existing cache, predicts the next call, then switches back. This is elegant:
The fixed suffix is: <think> Okay, let's see. The user provided what I need. I'll look it up. The next step is to make the tool call. </think> — it contains no task-specific information; it merely flips the model into "prediction mode."
5. Joint Agent-Speculator RL
Training a dual-mode agent is nontrivial because of two imbalances: the agent must retain task-solving ability, while the speculator must learn a different output distribution (predicting its own next step). Naively optimizing prediction accuracy degrades task success.
The paper's Joint Agent-Speculator RL:
1. Sample rollouts from the current agent policy 2. Construct speculation queries from intermediate trajectories 3. Alternate optimization: a few agent-update steps, then a few speculator-update steps 4. Speculation targets come from the agent's own rollouts — fully on-policy
Three stabilization tricks:
6. Key Results
| Model | Stage | Avg Hit@1 ↑ | Avg task success | |:---|:---|:---|:---| | Qwen3-4B | Base (direct prediction) | 29.1 | — | | Qwen3-4B | + SFT warmup | 44.1 | — | | Qwen3-4B | + Joint RL | 61.2 | 26.6 → 27.7 ↑ | | Qwen3.5-4B | Base | 33.5 | — | | Qwen3.5-4B | + SFT warmup | 48.9 | — | | Qwen3.5-4B | + Joint RL | 66.3 | 49.2 → 50.6 ↑ |
Hit@1 rises from 44.1 → 61.2 (+38.8% relative) while task success rate also improves.
Schedule ablation: 1:1 alternation collapses (Hit@1 31.8, success 10.3); 2:2 gives 41.8; 4:4 gives 50.3; 4:8 is best (55.2, 26.1). Multi-objective training should not distribute updates evenly — each objective needs consecutive steps.
Cross-domain generalization: trained on SearchQA, tested on τ-bench — Hit@1 improves 35.6 → 45.6, but task success drops 21.6 → 17.6. Speculation skill transfers across domains; task-solving skill does not.
7. Engineering Insights
1. "Only you know yourself" is math, not a platitude. Two different models — even from the same family — have different decision boundaries. Self-speculation eliminates the gap because the speculator shares the agent's decision boundary: it merely *says early* what it would have said anyway. 2. Alternating training philosophy: give each objective enough consecutive signal (1:1 collapses; 4:8 wins), and reset optimizer state so different objectives' momentum doesn't interfere. 3. Read-only tools are the safety boundary. Self-speculation applies only to read-only tools (search, retrieval, DB queries). Proactively executing state-changing tools (orders, updates, messages) risks side effects: *you may search in advance, but you must not place orders in advance.*
8. Conceptual Lineage: Unification vs. Division of Labor
Prior engineering wisdom favored division of labor (LLM + Prolog; small model pre-screens, large model reviews). The self-speculating agent instead chooses unification — but for a specific reason: the speculator's job is to *predict the agent*, which requires a shared decision boundary. The refined principle: unify when tasks must share a decision boundary; divide when tasks need different capabilities.
9. From Killing Waits to Redesigning Time
Current agents treat time linearly: act, wait, act, wait. Self-speculation overlaps reasoning with tool execution — structurally analogous to CPU pipelining, which overlapped fetch/decode/execute stages to double throughput. The broader design principle: for any system with serial waits, ask what can be done during the wait — database prefetch, predictive microservice calls, or preparing next steps while awaiting a colleague's reply. The paper's concrete answer: let the *waiter* predict the *waited-for* result and start the next step early.
10. Limitations and Open Questions
1. Read-only tools only; extending to writes would need dry-run or transaction rollback 2. 4B scale only (Qwen3-4B / Qwen3.5-4B); larger models may behave differently 3. Limited task scope: search QA and conversational tool calling; code execution, web navigation, multi-agent settings untested
Open questions: can the speculator predict *which tool* among many (not just arguments)? Does Hit@k / parallel speculation improve coverage? Can feeding speculation back to the agent improve decision quality?
---
Paper: Ji et al. (2026). *Speculate While You Reason: Teaching Agents to Predict Their Next Tool Call via Joint Agent–Speculator RL.* arXiv:2607.25816. https://arxiv.org/abs/2607.25816
Authors: Jiabao Ji, Yujian Liu, Li An, Rohit Jain, Gungor Polatkan, Siyu Zhu, Shiyu Chang (UC Santa Barbara + LinkedIn Inc.)
Headline numbers: Hit@1 44.1 → 61.2 (Qwen3-4B), 48.9 → 66.3 (Qwen3.5-4B), with task success rate improving rather than degrading.
Key insight: The agent itself is the best speculator of its own next tool call, because speculator and agent share the same decision boundary.
FAQ
Q1: Who is this for? Practitioners, researchers, and students interested in AI, machine learning, and LLM agents.
Q2: Core takeaways? Agents waste wall-clock time on tool I/O; external draft models suffer a speculator-agent gap; a single dual-mode model with shared KV cache plus joint RL achieves both high prediction accuracy and task success.
Q3: Is there open-source code? See the paper link in the body.