The Illusion of Visual Tool-Use: When Models Wield a Magnifier But Don't Really Look

Forum topic · ✨步子哥 · 2026-08-08

Summary

A causal audit by Shanghai AI Lab, Shanghai Jiao Tong University, and Shanghai Innovation Institute exposes a structural flaw in 'thinking with images.' Across six open-source models (DeepEyes, Qwen3-VL, Mini-o3, Pixel Reasoner, Thyme) and five fine-grained benchmarks, aggregate accuracy gains from crop-and-zoom tool calls often come from a small minority of well-calibrated trajectories, while most calls are ritual or wasteful. The authors formalize visual tool-use as a causal graph with an observed-mediation path (tool → observation → answer) and an action-induced shortcut (tool action → answer), then run three interventions: policy-level on/off comparisons, trajectory-level replacement of observations with random crops, and step-level Visual Evidence Gain (VEG). Mini-o3 collapses from 87.8% to 23.6% on V* when observations are swapped; DeepEyes barely changes. Two failure modes emerge: Calling Without Looking (saturated priors or inactive calls) and Looking Without Planning (redundant or budget-exhausted calls). The paper proposes the RL-trap hypothesis: outcome-only RL reinforces shortcuts and cannot distinguish calibrated from wasteful behavior. The paper recommends distribution-level metrics, adaptive early stopping, and process-aware credit assignment.

The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images

A Counter-Intuitive Finding

You give a multimodal LLM a "magnifier" that lets it actively crop and zoom into regions of an image during reasoning. You expect it to become more accurate and more detailed.

Results across six open-source models and five fine-grained perception benchmarks tell a different story:

DeepEyes: on V*, tool-on and tool-off accuracy are both 83.3% (Δ = 0.0 pp). On HR-Bench-8K it actually drops by 1.4 points.
Qwen3-VL-4B: Δ = 0.0 pp on HR-Bench-4K, Δ = -1.6 pp on HR-Bench-8K.
Mini-o3: gains 21.3 pp on VisualProbe, but token consumption explodes and 84.8% of trajectories hit the tool-call cap.

In short: the model uses the magnifier, spends more tokens, and sometimes answers worse.

This is not an outlier. It is a structural phenomenon. Zhiheng Wang, Bo Peng, Lai Wei, and Chaochao Lu from Shanghai AI Lab, Shanghai Jiao Tong University, and Shanghai Innovation Institute name it the illusion of visual tool-use in their paper *The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images* (arXiv:2608.06270).

Paper: https://arxiv.org/abs/2608.06270
Code: https://github.com/OpenCausaLab/CauAudit

Why Aggregate Numbers Lie

Over the past year, "thinking with images" has become a hot paradigm in multimodal AI. OpenAI, Qwen, DeepEyes, Pixel Reasoner, Mini-o3, and Thyme all promote letting models insert crop-and-zoom operations into the reasoning chain to break the limit of "see the whole image at once."

Benchmark numbers rise accordingly. Mini-o3 jumps from 33.7% to 55.0% on VisualProbe, which looks great. But the authors ask a sharper question:

> Did the visual evidence returned by the tool causally affect the final answer?

Correlation is not causation. A model that calls a tool and gets the right answer may have been able to answer without the tool — the call was a ritual.

Answering this requires causal intervention, not accuracy alone.

Three Layers of Causal Intervention

The authors formalize visual tool-use as a causal graph with two paths:

Observed-mediation path T → O → Y: tool call T produces observation O whose visual content truly affects answer Y. This is "really looking."
Action-induced shortcut T → Y: the textual action of the tool call directly affects Y, while O does nothing. This is "fake looking."

Layer 1: Policy-level (use vs. don't use)

Directly compare tool-on vs. tool-off accuracy. The results are limited and uneven. DeepEyes is near zero, Mini-o3 and Qwen3-VL-8B gain the most (Mini-o3 +21.3 pp on VisualProbe), but at the cost of far more tokens. The Average Treatment Effect (ATE) mixes "really looking" with "fake looking."

Layer 2: Trajectory-level (replace observations with junk)

The harsher intervention: replace every observation O_i returned during a reasoning trajectory with a random crop (same size, same position, completely unrelated content). Measure how much accuracy drops.

| Model | V* accuracy (orig / intervened) | Δ | Hit-MaxT | |---|---|---|---| | DeepEyes | 83.3 / 83.8 | +0.4 | 0% | | Pixel Reasoner | 84.8 / 82.7 | -2.1 | 0% | | Mini-o3 | 87.8 / 23.6 | -64.2 | 84.8% | | Qwen3-VL-8B | 91.1 / 30.4 | -60.7 | 59.2% | | Qwen3-VL-4B | 86.4 / 38.2 | -48.2 | 50.3% | | Thyme | 83.2 / 82.2 | -1.0 | 0% |

Mini-o3 collapses by 64 points — its answers truly depend on observation content, i.e. it really looks. DeepEyes barely moves (+0.4 pp), and Thyme only loses 1.0 pp — for these models the returned observations contribute almost no causal signal.

Note the Hit-MaxT column: 84.8% of Mini-o3 trajectories hit the tool-call cap. It does not look once and move on; it looks again and again and cannot stop.

Layer 3: Step-level (per-call contribution)

The finest layer: under a fixed prefix, swap the real observation O_i^real of the i-th call for a counterfactual crop O_i^cf and measure the change in the model's probability of the correct answer. The metric is Visual Evidence Gain (VEG):

> VEG_i = ΔM_i^real − ΔM_i^cf

Large VEG means the call actually contributed visual evidence. VEG near zero means "fake looking" — the action text T_i directly influences the answer regardless of O.

VEG is the paper's central instrument. It separates "model called the tool" from "model actually used what the tool returned."

Two Failure Modes: Performing and Overworking

Based on VEG, the authors identify two classes of miscalibrated strategy.

Mode 1: Calling Without Looking (CWL)

The tool is called, but the visual content of the observation has no causal effect on the answer. Two sub-cases:

(a) Saturated prior: the model is already very confident (probability gap g_{i-1} > τ_sat = 0.95) before calling the tool, so any later observation is squashed by the ceiling. Qwen3-VL-8B is typical.
(b) Structurally inactive calls: across the entire trajectory, no call carries visual evidence; the action text T → Y carries everything. DeepEyes is typical.

> Analogy: a student highlights an entire textbook but never reads the highlighted words. The pen moves, the eyes do not.

Mode 2: Looking Without Planning (LWP)

The observations are informative (VEG is non-zero), but the calling plan is broken. Two sub-cases:

(a) Post-saturation continuation: the model has seen enough and the answer is settled, yet it keeps calling. In Mini-o3, per-position VEG decays with later positions while the harmful-call rate rises.
(b) Budget exhaustion: the model never stops by itself and runs into the tool-call cap. Mini-o3's 84.8% cap-hit rate is this case.

> Analogy: a doctor who has seen all the tests and reached a diagnosis keeps ordering more. Or worse — keeps ordering until closing time while the patient still has no answer.

Key Finding: A Minority of Calibrated Trajectories Carry All the Gain

The authors partition trajectories into four groups: CWL, LWP, No-call, and Calibrated (effective tool use). They then decompose:

> Calibrated is the only group with a positive ATE on every model, and it accounts for almost all of the non-zero ATE.

In other words: the accuracy gains visible at the aggregate level come almost entirely from a small set of trajectories that genuinely know how to use the tool. The bulk of tool-call trajectories are either CWL (performing) or LWP (overworking), contributing close to zero — or even negative — net gain.

This is the precise meaning of the illusion of visual tool-use:

> Aggregate accuracy is rising, but the tool is not being broadly and causally effective. The gains concentrate in a few calibrated trajectories; the rest of the calls are ritual or waste.

Two lines from the paper are worth remembering:

> "Having a tool does not necessarily mean it is used properly."

> "Does the model think with images?" is more important than "Does the model call visual tools?"

The RL-Trap Hypothesis

A striking observation: CWL and LWP appear across all six models, which differ in architecture, data, and tool interface. This looks less like an implementation bug and more like a shared cause.

The authors propose the RL-trap hypothesis:

> Outcome-only RL training simultaneously manufactures both failure modes.

The logic:

1. Reinforces tool-calling behavior: RL rewards trajectories that call the tool and answer correctly, which strengthens the T → Y shortcut (Mode 1). The model learns that the action "call tool" correlates with reward, without needing to actually use the observation. 2. Does not penalize redundant calls: on correct trajectories, outcome-only reward cannot tell whether intermediate steps were redundant or harmful. The model is not punished for ten useless tool calls as long as the final answer is right (Mode 2). 3. Cannot prefer calibrated trajectories: a Calibrated trajectory and a Mode-1 trajectory that both answer correctly receive identical reward. RL has no signal to prefer the former.

This aligns with recent process-aware training work: outcome-only rewards cannot distinguish useful, redundant, and misleading tool calls.

If the hypothesis holds, the defect lies in the training objective itself — not in model size or tool quality, but in the reward signal's inability to distinguish "really looking" from "fake looking."

Engineering Value: The Diagnostic Is Actionable

The paper offers three actionable directions.

1. Beyond-Accuracy Evaluation

Any new intervention (prompt, tool, or training method) should be evaluated at the distribution level:

Does the share Δf_Cal of Calibrated trajectories rise?
Does the combined share Δf_M1 + Δf_M2 of CWL + LWP fall?

Even if ATE_policy does not move, a migration from CWL/LWP toward Calibrated is real progress. Conversely, if ATE rises while Calibrated share stays flat, you are manufacturing more "fake looking."

2. Inference-Time Adaptation

Mode-1 trajectories: bypass tool calls to save tokens.
Mode-2 trajectories: apply early-stopping rules — once the answer is saturated, commit and stop.

3. Process-Aware Credit Assignment

Step-level VEG provides the per-step supervision signal that outcome-only reward lacks:

Up-weight Calibrated behavior.
Penalize wasteful Mode-1 calls.
Penalize harmful Mode-2 calls.

This points directly to a new RL objective: reward not only final correctness but also the per-step Visual Evidence Gain.

Limitations and Open Questions

The authors themselves list the limitations:

1. Model access: all experiments are on open-source models. Step-level VEG requires token-level probabilities; closed models like OpenAI o3 / o4-mini cannot be measured. Whether the illusion also exists in closed models is unknown. 2. Limited tool set: only crop-and-zoom is tested. OCR, segmentation, video-frame selection, and external search may exhibit different calibration properties. 3. RL-trap is a hypothesis: establishing the mechanism requires matched-training experiments that vary only the reward signal. This is left for future work.

Takeaways

When evaluating visual-tool-use models, do not rely on aggregate accuracy alone. Look at Calibrated share, VEG distributions, and Hit-MaxT ratios.
When training them, do not rely on outcome reward alone. Inject step-level VEG as a process reward.
At inference, bypass tools on Mode-1 trajectories and add early stopping on Mode-2 trajectories.

Final reminder: "Does the model think with images?" matters more than "Does the model call visual tools?" The lesson generalizes: calling a tool is surface competence; causally effective use of a tool is real competence.

---

Paper: The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images Authors: Zhiheng Wang, Bo Peng, Lai Wei, Chaochao Lu Affiliations: Shanghai AI Lab / Shanghai Jiao Tong University / Shanghai Innovation Institute arXiv: https://arxiv.org/abs/2608.06270 Code: https://github.com/OpenCausaLab/CauAudit License: CC BY 4.0