English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

The Illusion of Visual Tool-Use: Do Vision Models Really Look Through the Magnifying Glass?

Forum topic · ✨步子哥 · 2026-08-08

Summary

A causal audit study from Shanghai AI Lab, Shanghai Jiao Tong University, and Shanghai Innovation Institute (arXiv:2608.06270) reveals that multimodal models using visual tools like crop-and-zoom often gain little or nothing from actually seeing the returned image content. Across six open-source models and five fine-grained perception benchmarks, the authors find that aggregate accuracy gains hide a structural problem they call "the illusion of visual tool-use." Using three levels of causal intervention—policy-level, trajectory-level (swapping observations with random crops), and step-level Visual Evidence Gain (VEG)—they identify two failure modes: Calling Without Looking (CWL) and Looking Without Planning (LWP). Notably, nearly all accuracy improvement comes from a small minority of "calibrated" trajectories. The authors propose an "RL-trap hypothesis": outcome-only reinforcement learning rewards cannot distinguish genuine from ritual tool use. The paper offers practical remedies: distribution-level evaluation, inference-time adaptation, and process-aware credit assignment using VEG.

A Counterintuitive Finding

Give a multimodal large model a "magnifying glass"—the ability to actively crop and zoom into regions of an image during reasoning, like a detective leaning in on a clue. You expect it to become more accurate, more precise, smarter.

The result? Across six open-source models and five fine-grained perception benchmarks:

  • DeepEyes: On the V* benchmark, accuracy is 83.3% with tools on AND off (Δ = 0.0pp). On HR-Bench-8K it even drops 1.4pp.
  • Qwen3-VL-4B: Δ = 0.0pp on HR-Bench-4K, Δ = -1.6pp on HR-Bench-8K.
  • Mini-o3: Gains +21.3pp on VisualProbe, but at the cost of massively inflated token consumption, with 84.8% of trajectories hitting the tool-call limit.
  • In other words: the model used the magnifying glass, spent more tokens, and sometimes answered worse.

    This is not an isolated case but a structural phenomenon. Zhiheng Wang, Bo Peng, Lai Wei, and Chaochao Lu (Shanghai AI Lab, Shanghai Jiao Tong University, Shanghai Innovation Institute) name it the illusion of visual tool-use in the paper *The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images*.

    Paper: https://arxiv.org/abs/2608.06270 Code: https://github.com/OpenCausaLab/CauAudit

    Where the Problem Lies: Aggregate Numbers Lie

    Over the past year, "thinking with images" has become a buzzword in multimodal AI. OpenAI, Qwen, DeepEyes, Pixel Reasoner, Mini-o3, and Thyme all push this paradigm—inserting visual operations like crop-and-zoom into the reasoning chain, theoretically breaking the "look at the whole image once" limitation.

    Benchmark numbers do rise. Mini-o3 jumps from 33.7% to 55.0% on VisualProbe—looks great. But the authors ask a harsher question:

    > Does the visual evidence obtained from tool calls actually causally affect the final answer?

    Note "causal." Correlation is not causation—the model called a tool and answered correctly, but that doesn't mean the tool helped. The tool call may be pure ritual.

    Answering this requires causal intervention, not just accuracy.

    Three Levels of Causal Intervention

    The authors formalize visual tool-use as a causal graph, distinguishing two paths:

  • Observation-mediated path T → O → Y: the tool call T produces an observation O whose visual content genuinely affects the answer Y. "Real looking."
  • Action-induced shortcut T → Y: the text of the tool call itself directly affects Y, with O doing nothing. "Fake looking."
  • Level 1: Policy-level intervention (tools on vs. off)

    Compare accuracy with tools on vs. off (the Δ values above). Findings: improvements are limited and extremely uneven. DeepEyes is nearly zero; Mini-o3 and Qwen3-VL-8B gain the most, but with heavy token costs—and this ATE conflates real and fake looking.

    Level 2: Trajectory-level intervention (swap observations with garbage)

    Replace every observation O_i returned by tool calls with a randomly cropped image (same size and position, irrelevant content) and measure the accuracy drop:

    | Model | V* accuracy (orig / intervened) | Δ | Hit-MaxT | |---|---|---|---| | DeepEyes | 83.3 / 83.8 | +0.4 | 0% | | Pixel Reasoner | 84.8 / 82.7 | -2.1 | 0% | | Mini-o3 | 87.8 / 23.6 | -64.2 | 84.8% | | Qwen3-VL-8B | 91.1 / 30.4 | -60.7 | 59.2% | | Qwen3-VL-4B | 86.4 / 38.2 | -48.2 | 50.3% | | Thyme | 83.2 / 82.2 | -1.0 | 0% |

    Mini-o3 crashes 64 points—its answers genuinely depend on observation content. But DeepEyes barely moves (+0.4pp) and Thyme drops only 1.0pp—their observations have almost no causal contribution to answers. Note Hit-MaxT: 84.8% of Mini-o3 trajectories hit the tool-call limit. It doesn't "look once and stop"—it "keeps looking compulsively."

    Level 3: Step-level intervention (VEG)

    The finest level: under a fixed prefix, swap the real observation O_i^real of the i-th tool call with a counterfactual crop O_i^cf, and measure the change in the model's probability of the correct answer. This metric is Visual Evidence Gain (VEG):

    > VEG_i = ΔM_i^real - ΔM_i^cf

    Large VEG means the observation contributed real visual evidence; VEG near zero means "fake looking"—the action text T_i drove the answer, and O could be anything. VEG is the paper's core tool: it separates "the model called the tool" from "the model actually used the visual content."

    Two Failure Modes

    Based on VEG, the authors identify two forms of policy miscalibration:

    Mode 1: Calling Without Looking (CWL)

    Tool calls whose observations carry no causal contribution to the answer. Two subtypes:

  • (a) Saturated prior: the model is already confident before calling (probability gap g_{i-1} > τ_sat = 0.95), capping any observation's contribution near zero. Typical: Qwen3-VL-8B.
  • (b) Structurally inactive calls: no call in the entire trajectory carries visual evidence; the T → Y shortcut carries everything. Typical: DeepEyes.
  • > Analogy: a student highlighting an entire textbook without reading the highlighted text. The pen moves; the eyes don't.

    Mode 2: Looking Without Planning (LWP)

    Observations are informative (nonzero VEG), but the call plan is terrible:

  • (a) Post-saturation continuation: the model has seen enough and the answer is settled, yet keeps calling tools. Mini-o3's per-position VEG decays with position while harmful-call rate rises.
  • (b) Budget exhaustion: the model never stops on its own, calling until the limit. 84.8% of Mini-o3 trajectories hit the cap.
  • > Analogy: a doctor who has completed all tests and made the diagnosis but keeps ordering more—until end of shift, with no conclusion delivered.

    Key Finding: A Few "Calibrated" Trajectories Carry All the Gains

    The authors partition trajectories into CWL, LWP, No-call, and Calibrated. The decomposition:

    > Calibrated is the only group with positive ATE on all models, and accounts for the vast majority of nonzero ATE.

    Aggregate accuracy improvement comes almost entirely from a small minority of trajectories that use tools properly. Most tool-use trajectories are either CWL (ritual) or LWP (wasteful), contributing near-zero or negative net gain.

    This is the precise meaning of the illusion:

    > Rising aggregate accuracy does not mean tools are being widely and causally effectively used. The gains concentrate in a few calibrated trajectories; the rest are ceremony or waste.

    A memorable line: "Having a tool does not necessarily mean it is used properly." "Does the model *think* with images?" matters more than "Does the model *call* visual tools?"

    The RL-Trap Hypothesis

    Why do all six models—differing in architecture, data, and tool interfaces—show CWL and LWP? The authors propose the RL-trap hypothesis:

    > Outcome-only RL training produces both failure modes simultaneously.

    1. Reinforcing tool-call behavior: RL rewards trajectories that call tools and answer correctly, reinforcing the T → Y shortcut (Mode 1)—"calling" becomes associated with reward regardless of whether observations are used. 2. Not penalizing redundant calls: on correct trajectories, outcome-only rewards can't tell redundant or harmful steps from useful ones (Mode 2). Calling ten useless tools is fine as long as the final answer is right. 3. No signal favoring calibrated trajectories: a Calibrated trajectory and a Mode 1 trajectory that both answer correctly get identical reward.

    This aligns with recent process-aware training research: outcome-only rewards cannot distinguish useful, redundant, and misleading tool calls. If the hypothesis holds, the training objective itself is flawed—not the model size, not the tools, but the reward signal failing to separate real from fake looking.

    Engineering Value: Operational Diagnostics

    The paper offers three actionable directions:

    1. Beyond accuracy in evaluation

    Any new intervention (new prompt, tool, or training method) should be checked for distributional shifts:

  • Does the Calibrated share Δf_Cal rise?
  • Do CWL + LWP shares Δf_M1 + Δf_M2 fall?
  • Even if ATE_policy is flat, migration from CWL/LWP toward Calibrated is real improvement. Conversely, rising ATE with flat Calibrated share means you're manufacturing more fake looking.

    2. Inference-time adaptation

  • Mode 1 trajectories: bypass tool calls entirely, saving tokens.
  • Mode 2 trajectories: apply early-stopping rules—commit once the answer saturates.
  • 3. Process-aware credit assignment

    Step-level VEG provides the per-step supervision that outcome-only rewards lack: upweight Calibrated behavior, penalize wasteful Mode 1 calls and harmful Mode 2 calls. This points directly to new RL objectives rewarding per-step visual evidence gain, not just final correctness.

    My Take: Another Case of the Evaluation Blind-Spot Law

    This paper fits a recurring theme—the evaluation blind-spot law: whatever you measure is what models optimize; whatever you don't measure is where problems hide.

  • Epanorthosis: models systematically reproduce the ancient rhetorical device of self-correction, because RLHF rewards confident emphasis.
  • TokenBudget: a bimodal 96.5% vs 11.5% fate is encoded at layer-20 token-150, invisible to standard loss.
  • QuantiBias: quantization introduces 24–27% bias inside the blind spots of standard safety checks.
  • Progressive Cramming: 99% token accuracy masking 100% generation failure.
  • Regression Tax: skill libraries make agents worse, with 59% of gains offset by regressions that average pass rates can't see.
  • This paper lands another precise strike on the same theme: aggregate accuracy is the most-used metric and the biggest blind spot. It blends "a few trajectories genuinely using tools" with "most trajectories faking or flailing" into a false impression of tool effectiveness.

    The remedy follows the same direction: from aggregate to distributional metrics, from outcome to process metrics. Don't ask "how much did the average improve"—ask "which trajectory class produced the gain, and what is the visual evidence gain per call."

    Another Association: Granularity Isomorphism

    This paper also evokes the granularity isomorphism principle—the granularity of optimization should match the granularity of the optimized object.

  • The actual working unit of thinking-with-images is a single tool call (each producing an observation that may or may not contribute evidence).
  • But outcome-only RL's reward granularity is the whole trajectory.
  • This mismatch means trajectory-level rewards cannot distinguish step-level effectiveness.
  • In the agent era, the optimization unit must upgrade from call to trajectory to step-level evidence. VEG is the quantitative metric for that step-level evidence.

    Broader Implications of the RL Trap

    The RL trap applies wherever intermediate steps are observable but rewards are outcome-only:

  • Code-generation agents: which tool calls (search, file reads, test runs) were genuinely useful when only the final code is rewarded?
  • Search agents: five queries and a correct answer—but were the first four redundant or progressive?
  • Reasoning chains: ten reasoning steps and a correct answer—but which steps were key, which were filler?
  • Whenever reward granularity is coarser than behavior granularity, there is room for "fake doing" and "flailing." This is a structural defect of outcome-only RL, not a domain-specific quirk.

    Limitations and Open Questions

    1. Model access: all experiments are on open-source models. Step-level VEG requires token-level probabilities; closed models (OpenAI o3/o4-mini) couldn't be tested. Do they share the illusion? Unknown. 2. Limited tool set: only crop-and-zoom tested. OCR, segmentation, video frame selection, and external search may have different calibration properties. 3. RL trap is a hypothesis: establishing the mechanism requires matched training experiments varying only the reward signal—left as future work.

    Conclusion: From "Can Use Tools" to "Actually Uses Tools"

    This paper turns an intuitive suspicion ("is the model really using the magnifying glass?") into an operational causal framework (three intervention levels + VEG), then systematically exposes a structural problem across six models.

    The most striking finding: aggregate accuracy rises, but the gains concentrate in a minority of calibrated trajectories. Most tool calls are ritual or waste. This is not a capability problem but a training-objective problem—outcome-only RL cannot separate real from fake looking.

    Practical takeaways:

  • When evaluating, don't just look at accuracy—look at Calibrated share, VEG distributions, Hit-MaxT rates.
  • When training, don't just give outcome rewards—use step-level VEG as process reward.
  • At inference, bypass tools for Mode 1 trajectories and apply early stopping for Mode 2.
One last line: "Does the model think with images?" matters more than "Does the model call visual tools?" This applies to all tool-augmented agents—calling a tool is surface capability; causally effective tool use is real capability.

---

Paper: The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images Authors: Zhiheng Wang, Bo Peng, Lai Wei, Chaochao Lu Institutions: Shanghai AI Lab / Shanghai Jiao Tong University / Shanghai Innovation Institute arXiv: https://arxiv.org/abs/2608.06270 Code: https://github.com/OpenCausaLab/CauAudit License: CC BY 4.0

Tags

#multimodal-models#visual-tool-use#causal-analysis#reinforcement-learning#evaluation-benchmarks#thinking-with-images#ai-agents#rl-reward-design

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178603074