When the Model Calls but Doesn't Look
Imagine hiring a detective and giving him a telescope. He raises it, reports "I inspected the window" and "I magnified the license plate," and reaches a verdict. Replace the telescope lens with opaque glass: his accuracy barely changes. The telescope is not an observation tool — it is a prop.
This is the core finding of *The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images* (arXiv:2608.06270), authored by Zhiheng Wang, Bo Peng, Lai Wei, and Chaochao Lu from Shanghai AI Lab, Shanghai Jiao Tong University, and Shanghai Innovation Institute.
The Promise of Thinking with Images
Over the past year, multimodal LLMs have promoted thinking-with-images: rather than answering from a single full image, they actively invoke crop-and-zoom operations mid-reasoning. Representative systems include OpenAI o3/o4-mini, DeepEyes, Pixel Reasoner, Mini-o3, the Qwen3-VL family, and code-agent Thyme. Across five fine-grained perception benchmarks, however, the average accuracy gain from visual tool use over direct reasoning is only 0 to a few percentage points, while token consumption multiplies several-fold. In some cases, enabling the tool actually reduces accuracy.
A Three-Layer Causal Audit
Policy-level comparison (with vs. without tool) only answers whether the average outcome differs — not whether the tool's *visual information* caused the difference. The paper introduces a three-layer audit:
Layer 1 — Policy-level ATE
Average Treatment Effect at the policy level. On V*, DeepEyes scores 0.0, Pixel Reasoner +3.6, Qwen3-VL-4B +4.4, Qwen3-VL-8B +6.9, Mini-o3 +5.5. Promising-looking, but the number is misleading.Layer 2 — Trajectory-level corruption
The model still calls the tool, but every returned crop is replaced with Gaussian noise, a blank image, or an unrelated crop. If the model genuinely used the crops, accuracy must collapse. It largely doesn't — revealing that many calls carry no causal influence.Layer 3 — Step-level Visual Evidence Gain (VEG)
Holding all prior reasoning fixed, replace only the i-th tool's return with a counterfactual observation and measure how the answer distribution shifts. VEG > 0 means the call contributed real visual evidence; VEG ≈ 0 means the call was idle motion. The construction is inspired by Pearl's Natural Indirect Effect.Two Failure Modes
Mode 1 — Calling Without Looking (CWL)
The model emits a crop-and-zoom call, but the returned crop has zero causal impact on the answer (VEG ≈ 0). In trajectory-level corruption tests, some models' accuracy barely moves.Mode 2 — Looking Without Planning (LWP)
The model does consume visual evidence (VEG > 0) but its calling strategy is broken: it keeps calling after the answer probability already exceeds 0.95, repeatedly zooms into irrelevant regions, or exhausts the tool budget and gets truncated mid-reasoning.The Four-Bucket Decomposition
Every trajectory is classified into one of four buckets:
| Bucket | Meaning | |---|---| | No-call | No tool invocation | | Mode 1 (CWL) | Called, didn't look | | Calibrated | Called, looked, stopped correctly | | Mode 2 (LWP) | Looked, failed to plan |
On V*:
| Model | No-call | Mode 1 | Calibrated | Mode 2 | |---|---|---|---|---| | DeepEyes | 19.4% | 72.8% | 7.9% | 0.0% | | Qwen3-VL-4B | 13.1% | 58.6% | 24.1% | 4.2% | | Qwen3-VL-8B | 5.2% | 70.7% | 20.9% | 3.1% | | Mini-o3 | 10.5% | 34.6% | 45.0% | 9.9% |
ATE decomposition on the same benchmark:
| Model | No-call | Mode 1 | Calibrated | Mode 2 | Total ATE | |---|---|---|---|---|---| | DeepEyes | 0.0 | -0.5 | +0.5 | 0.0 | 0.0 | | Qwen3-VL-4B | -0.2 | +2.0 | +3.5 | -0.9 | +4.4 | | Qwen3-VL-8B | -0.5 | -0.2 | +7.6 | -0.1 | +6.9 | | Mini-o3 | -0.3 | +1.9 | +3.3 | +0.6 | +5.5 |
Calibrated is the only bucket that is positive across every model. The headline +5.5 pp or +6.9 pp gains come almost entirely from the 20–45% of trajectories that calibrate correctly; the remaining majority is idling, thrashing, or dragging accuracy down.
The RL-trap Hypothesis
The authors propose that all six models fall into the same trap because of outcome-only RL: training rewards the final answer, not the intermediate process. This produces three reinforcing failure mechanisms:
1. It reinforces the shortcut T → Y (tool call → correct answer) without forcing genuine extraction of visual evidence. 2. It imposes no penalty on redundant or harmful intermediate steps along otherwise correct trajectories. 3. It makes Calibrated and CWL trajectories reward-equivalent as long as the answer is right, removing any signal that would distinguish them.
Engineering Recommendations
- Training: replace outcome-only rewards with process-aware rewards that penalize CWL and reward calibrated behavior.
- Inference: bypass Mode 1 calls at runtime — they contribute nothing — and cut token cost without losing accuracy.
- Evaluation: stop reporting policy-level ATE alone; standardize on VEG and the four-bucket decomposition.
- Paper: https://arxiv.org/abs/2608.06270
- Code: https://github.com/OpenCausaLab/CauAudit
The classifier is deterministic, with five features (call count n, pre-call probability gap g₀, peak VEG V^max, budget-exhaustion flag HitMax, post-saturation over-call flag POER) and two thresholds (τ_sat = 0.95, ε = 0.01).