The Illusion of Visual Tool-Use: When Models Call Tools But Never Look
*English translation of a zhichai.net forum post discussing arXiv:2608.06270 — "The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images" (Zhiheng Wang et al., Shanghai AI Lab, August 2026).*
An Expensive Waste of Effort
Imagine hiring a detective and giving him a telescope. On complex cases he raises the telescope, looks around, and gives his verdict. You assume the telescope helps him see better — his reports are full of lines like "I checked the window with the telescope" and "I zoomed in on the license plate."
Then one day you secretly swap the lenses for opaque ones. The detective still raises the telescope, still writes the same reports, and his accuracy barely changes.
Only then do you realize: for him, the telescope isn't an observation tool — it's a performance prop.
That is precisely what the paper finds across six mainstream "image-thinking" multimodal models.
The Promise vs. Reality of "Thinking with Images"
Over the past year, thinking-with-images has been one of the hottest ideas in multimodal LLMs. Representative models include OpenAI's o3/o4-mini, DeepEyes, Pixel Reasoner, Mini-o3, the Qwen3-VL series, and the code-based Thyme. Instead of answering from the full image, these models actively invoke crop-and-zoom during reasoning: zoom into the corner to read a book title, crop the bottom-right to confirm a cup's brand.
But the paper reports an awkward number:
> Across five fine-grained perception benchmarks, visual tool use improves average accuracy by only 0 to a few points over direct inference, while token consumption is several times higher. Worse, some questions that direct inference answers correctly are answered incorrectly with tools enabled.
That is where the paper's "Illusion" comes from: the accuracy number looks like it rises — but is the rise actually caused by the tool?
Causal Audit: Not "Is It Accurate?" but "Did the Tool Actually Help?"
Standard evaluation toggles tools on and off and compares accuracy — what the paper calls policy-level intervention. That only tells you whether there's an overall difference, not whether the difference is caused by the tool's visual information. The paper's core contribution is a three-layer causal audit framework, each layer finer than the last.
Layer 1: Policy level — does toggling tools matter?
The coarsest layer. On the V* benchmark, policy-level ATE (Average Treatment Effect): DeepEyes 0.0, Pixel Reasoner +3.6, Qwen3-VL-4B +4.4, Qwen3-VL-8B +6.9, Mini-o3 +5.5. Looks positive — but this number lies.
Layer 2: Trajectory level — what if observations are replaced with noise?
The model calls tools as usual, but the returned crops are replaced with corrupted observations — Gaussian noise, blank images, or crops of irrelevant regions. If the model truly "looks" at the crops, accuracy should collapse; if it doesn't change, the model isn't using them. This underlies Visual Evidence Gain (VEG): quantifying how much visual information each tool call contributes by comparing answers under real vs. corrupted observations.
Layer 3: Step level — how much did each call contribute?
At step i of reasoning, keep everything else fixed and replace only the i-th tool call's result with a counterfactual observation, measuring how the answer distribution changes. This is VEG, inspired by Pearl's Natural Indirect Effect. VEG > 0 means the call delivered real visual evidence; VEG ≈ 0 means it idled.
Two Failure Modes: Calling Without Looking and Looking Without Planning
Mode 1: Calling Without Looking (CWL)
The model issues a crop-and-zoom, gets a crop back — but the crop has no causal influence on the final answer (VEG ≈ 0). Verified via trajectory-level intervention: replacing all observations with noise barely moves accuracy for some models. The opaque-telescope scenario in real life.
Mode 2: Looking Without Planning (LWP)
The model does use the visual information (VEG > 0), but the calling policy is a mess: calling after the answer is already settled (answer probability above 0.95, yet four or five more calls), repeatedly cropping the same irrelevant region, or exhausting the call budget and truncating reasoning.
Four-Group Decomposition: Gains Are Carried by a "Calibrated Minority"
Each trace is classified as No-call, Mode 1 (CWL), Calibrated, or Mode 2 (LWP). Distribution on V* (Table 4):
| Model | No-call | Mode 1 | Calibrated | Mode 2 | |---|---|---|---|---| | DeepEyes | 19.4% | 72.8% | 7.9% | 0.0% | | Qwen3-VL-4B | 13.1% | 58.6% | 24.1% | 4.2% | | Qwen3-VL-8B | 5.2% | 70.7% | 20.9% | 3.1% | | Mini-o3 | 10.5% | 34.6% | 45.0% | 9.9% |
DeepEyes raises the telescope most often — with opaque lenses. Even the best model, Mini-o3, achieves calibrated use in only 45% of traces.
ATE decomposition (Table 5):
| Model | No-call | Mode 1 | Calibrated | Mode 2 | Total ATE | |---|---|---|---|---|---| | DeepEyes | 0.0 | -0.5 | +0.5 | 0.0 | 0.0 | | Qwen3-VL-4B | -0.2 | +2.0 | +3.5 | -0.9 | +4.4 | | Qwen3-VL-8B | -0.5 | -0.2 | +7.6 | -0.1 | +6.9 | | Mini-o3 | -0.3 | +1.9 | +3.3 | +0.6 | +5.5 |
Calibrated is the only column positive across all models. The headline "+5.5 pp" or "+6.9 pp" comes almost entirely from the 20–45% calibrated traces; the rest idle, flail, or drag accuracy down.
> "Having a tool does not necessarily mean it is used properly."
"Can the model call visual tools?" and "Does the model actually think with images?" are two very different questions — and current models satisfy the former far more often than the latter.
The RL-trap Hypothesis: Why Every Model Falls Into the Same Pit
The paper explains why six models with different architectures, data, and tool APIs share the same failures: outcome-only RL. Training rewards only the final answer on tool-augmented traces, which simultaneously:
1. Creates Mode 1: rewards the act of calling tools, reinforcing the tool-call → answer shortcut without teaching genuine information extraction. 2. Creates Mode 2: never penalizes redundant or harmful intermediate steps on correct traces. 3. Confuses Calibrated with Mode 1: a trace that truly used visual evidence and one that ignored every observation get identical rewards if the answer is right.
The paper calls this the RL-trap: outcome-only RL raises benchmark scores, but the gains are causally unrelated to tool effectiveness. It is structurally analogous to the Regression Tax (skill libraries making agents worse) and Looping Is Not Reliability (correctness is not an absorbing state).
Practical Value
1. Training: replace outcome-only rewards with process-aware rewards — down-weight Mode 1 traces, up-weight Calibrated ones. 2. Inference: bypass tool calls predicted to be Mode 1, saving tokens without accuracy loss. 3. Evaluation: report VEG and the four-group decomposition as standard metrics. The diagnostic is deterministic — five features (call count n, pre-first-call probability gap g₀, peak VEG V^max, budget exhaustion HitMax, post-saturation overuse POER), two thresholds (τ_sat=0.95, ε=0.01), no trained classifier, no outcome labels.
Closing: A Detective Needs More Than a Telescope
The number of times a detective raises the telescope does not equal the number of times he sees a clue through it. Most current multimodal models are still at the "raising the telescope" stage — the motion is there, the performance is there, but the connection between lens and eye hasn't been built. If you work on visual tool use — training or evaluation — run the four-group decomposition and see whether your model is truly "thinking with images" or merely "calling image tools." Having a tool is not the same as using it well — a lesson that applies to humans, too.
---
Paper: arXiv:2608.06270 — *The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images* Authors: Zhiheng Wang, Bo Peng, Lai Wei, Chaochao Lu (Shanghai AI Lab / Shanghai Jiao Tong University / Shanghai Innovation Institute) Code: https://github.com/OpenCausaLab/CauAudit Paper link: https://arxiv.org/abs/2608.06270