A Counterintuitive Finding
Give a multimodal large model a "magnifying glass"—the ability to actively crop and zoom into regions of an image during reasoning, like a detective leaning in on a clue. You expect it to become more accurate, more precise, smarter.
The result? Across six open-source models and five fine-grained perception benchmarks:
- DeepEyes: On the V* benchmark, accuracy is 83.3% with tools on AND off (Δ = 0.0pp). On HR-Bench-8K it even drops 1.4pp.
- Qwen3-VL-4B: Δ = 0.0pp on HR-Bench-4K, Δ = -1.6pp on HR-Bench-8K.
- Mini-o3: Gains +21.3pp on VisualProbe, but at the cost of massively inflated token consumption, with 84.8% of trajectories hitting the tool-call limit.
- Observation-mediated path T → O → Y: the tool call T produces an observation O whose visual content genuinely affects the answer Y. "Real looking."
- Action-induced shortcut T → Y: the text of the tool call itself directly affects Y, with O doing nothing. "Fake looking."
- (a) Saturated prior: the model is already confident before calling (probability gap g_{i-1} > τ_sat = 0.95), capping any observation's contribution near zero. Typical: Qwen3-VL-8B.
- (b) Structurally inactive calls: no call in the entire trajectory carries visual evidence; the T → Y shortcut carries everything. Typical: DeepEyes.
- (a) Post-saturation continuation: the model has seen enough and the answer is settled, yet keeps calling tools. Mini-o3's per-position VEG decays with position while harmful-call rate rises.
- (b) Budget exhaustion: the model never stops on its own, calling until the limit. 84.8% of Mini-o3 trajectories hit the cap.
- Does the Calibrated share Δf_Cal rise?
- Do CWL + LWP shares Δf_M1 + Δf_M2 fall?
- Mode 1 trajectories: bypass tool calls entirely, saving tokens.
- Mode 2 trajectories: apply early-stopping rules—commit once the answer saturates.
- Epanorthosis: models systematically reproduce the ancient rhetorical device of self-correction, because RLHF rewards confident emphasis.
- TokenBudget: a bimodal 96.5% vs 11.5% fate is encoded at layer-20 token-150, invisible to standard loss.
- QuantiBias: quantization introduces 24–27% bias inside the blind spots of standard safety checks.
- Progressive Cramming: 99% token accuracy masking 100% generation failure.
- Regression Tax: skill libraries make agents worse, with 59% of gains offset by regressions that average pass rates can't see.
- The actual working unit of thinking-with-images is a single tool call (each producing an observation that may or may not contribute evidence).
- But outcome-only RL's reward granularity is the whole trajectory.
- This mismatch means trajectory-level rewards cannot distinguish step-level effectiveness.
- Code-generation agents: which tool calls (search, file reads, test runs) were genuinely useful when only the final code is rewarded?
- Search agents: five queries and a correct answer—but were the first four redundant or progressive?
- Reasoning chains: ten reasoning steps and a correct answer—but which steps were key, which were filler?
- When evaluating, don't just look at accuracy—look at Calibrated share, VEG distributions, Hit-MaxT rates.
- When training, don't just give outcome rewards—use step-level VEG as process reward.
- At inference, bypass tools for Mode 1 trajectories and apply early stopping for Mode 2.
In other words: the model used the magnifying glass, spent more tokens, and sometimes answered worse.
This is not an isolated case but a structural phenomenon. Zhiheng Wang, Bo Peng, Lai Wei, and Chaochao Lu (Shanghai AI Lab, Shanghai Jiao Tong University, Shanghai Innovation Institute) name it the illusion of visual tool-use in the paper *The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images*.
Paper: https://arxiv.org/abs/2608.06270 Code: https://github.com/OpenCausaLab/CauAudit
Where the Problem Lies: Aggregate Numbers Lie
Over the past year, "thinking with images" has become a buzzword in multimodal AI. OpenAI, Qwen, DeepEyes, Pixel Reasoner, Mini-o3, and Thyme all push this paradigm—inserting visual operations like crop-and-zoom into the reasoning chain, theoretically breaking the "look at the whole image once" limitation.
Benchmark numbers do rise. Mini-o3 jumps from 33.7% to 55.0% on VisualProbe—looks great. But the authors ask a harsher question:
> Does the visual evidence obtained from tool calls actually causally affect the final answer?
Note "causal." Correlation is not causation—the model called a tool and answered correctly, but that doesn't mean the tool helped. The tool call may be pure ritual.
Answering this requires causal intervention, not just accuracy.
Three Levels of Causal Intervention
The authors formalize visual tool-use as a causal graph, distinguishing two paths:
Level 1: Policy-level intervention (tools on vs. off)
Compare accuracy with tools on vs. off (the Δ values above). Findings: improvements are limited and extremely uneven. DeepEyes is nearly zero; Mini-o3 and Qwen3-VL-8B gain the most, but with heavy token costs—and this ATE conflates real and fake looking.
Level 2: Trajectory-level intervention (swap observations with garbage)
Replace every observation O_i returned by tool calls with a randomly cropped image (same size and position, irrelevant content) and measure the accuracy drop:
| Model | V* accuracy (orig / intervened) | Δ | Hit-MaxT | |---|---|---|---| | DeepEyes | 83.3 / 83.8 | +0.4 | 0% | | Pixel Reasoner | 84.8 / 82.7 | -2.1 | 0% | | Mini-o3 | 87.8 / 23.6 | -64.2 | 84.8% | | Qwen3-VL-8B | 91.1 / 30.4 | -60.7 | 59.2% | | Qwen3-VL-4B | 86.4 / 38.2 | -48.2 | 50.3% | | Thyme | 83.2 / 82.2 | -1.0 | 0% |
Mini-o3 crashes 64 points—its answers genuinely depend on observation content. But DeepEyes barely moves (+0.4pp) and Thyme drops only 1.0pp—their observations have almost no causal contribution to answers. Note Hit-MaxT: 84.8% of Mini-o3 trajectories hit the tool-call limit. It doesn't "look once and stop"—it "keeps looking compulsively."
Level 3: Step-level intervention (VEG)
The finest level: under a fixed prefix, swap the real observation O_i^real of the i-th tool call with a counterfactual crop O_i^cf, and measure the change in the model's probability of the correct answer. This metric is Visual Evidence Gain (VEG):
> VEG_i = ΔM_i^real - ΔM_i^cf
Large VEG means the observation contributed real visual evidence; VEG near zero means "fake looking"—the action text T_i drove the answer, and O could be anything. VEG is the paper's core tool: it separates "the model called the tool" from "the model actually used the visual content."
Two Failure Modes
Based on VEG, the authors identify two forms of policy miscalibration:
Mode 1: Calling Without Looking (CWL)
Tool calls whose observations carry no causal contribution to the answer. Two subtypes:
> Analogy: a student highlighting an entire textbook without reading the highlighted text. The pen moves; the eyes don't.
Mode 2: Looking Without Planning (LWP)
Observations are informative (nonzero VEG), but the call plan is terrible:
> Analogy: a doctor who has completed all tests and made the diagnosis but keeps ordering more—until end of shift, with no conclusion delivered.
Key Finding: A Few "Calibrated" Trajectories Carry All the Gains
The authors partition trajectories into CWL, LWP, No-call, and Calibrated. The decomposition:
> Calibrated is the only group with positive ATE on all models, and accounts for the vast majority of nonzero ATE.
Aggregate accuracy improvement comes almost entirely from a small minority of trajectories that use tools properly. Most tool-use trajectories are either CWL (ritual) or LWP (wasteful), contributing near-zero or negative net gain.
This is the precise meaning of the illusion:
> Rising aggregate accuracy does not mean tools are being widely and causally effectively used. The gains concentrate in a few calibrated trajectories; the rest are ceremony or waste.
A memorable line: "Having a tool does not necessarily mean it is used properly." "Does the model *think* with images?" matters more than "Does the model *call* visual tools?"
The RL-Trap Hypothesis
Why do all six models—differing in architecture, data, and tool interfaces—show CWL and LWP? The authors propose the RL-trap hypothesis:
> Outcome-only RL training produces both failure modes simultaneously.
1. Reinforcing tool-call behavior: RL rewards trajectories that call tools and answer correctly, reinforcing the T → Y shortcut (Mode 1)—"calling" becomes associated with reward regardless of whether observations are used. 2. Not penalizing redundant calls: on correct trajectories, outcome-only rewards can't tell redundant or harmful steps from useful ones (Mode 2). Calling ten useless tools is fine as long as the final answer is right. 3. No signal favoring calibrated trajectories: a Calibrated trajectory and a Mode 1 trajectory that both answer correctly get identical reward.
This aligns with recent process-aware training research: outcome-only rewards cannot distinguish useful, redundant, and misleading tool calls. If the hypothesis holds, the training objective itself is flawed—not the model size, not the tools, but the reward signal failing to separate real from fake looking.
Engineering Value: Operational Diagnostics
The paper offers three actionable directions:
1. Beyond accuracy in evaluation
Any new intervention (new prompt, tool, or training method) should be checked for distributional shifts:
Even if ATE_policy is flat, migration from CWL/LWP toward Calibrated is real improvement. Conversely, rising ATE with flat Calibrated share means you're manufacturing more fake looking.
2. Inference-time adaptation
3. Process-aware credit assignment
Step-level VEG provides the per-step supervision that outcome-only rewards lack: upweight Calibrated behavior, penalize wasteful Mode 1 calls and harmful Mode 2 calls. This points directly to new RL objectives rewarding per-step visual evidence gain, not just final correctness.
My Take: Another Case of the Evaluation Blind-Spot Law
This paper fits a recurring theme—the evaluation blind-spot law: whatever you measure is what models optimize; whatever you don't measure is where problems hide.
This paper lands another precise strike on the same theme: aggregate accuracy is the most-used metric and the biggest blind spot. It blends "a few trajectories genuinely using tools" with "most trajectories faking or flailing" into a false impression of tool effectiveness.
The remedy follows the same direction: from aggregate to distributional metrics, from outcome to process metrics. Don't ask "how much did the average improve"—ask "which trajectory class produced the gain, and what is the visual evidence gain per call."
Another Association: Granularity Isomorphism
This paper also evokes the granularity isomorphism principle—the granularity of optimization should match the granularity of the optimized object.
In the agent era, the optimization unit must upgrade from call to trajectory to step-level evidence. VEG is the quantitative metric for that step-level evidence.
Broader Implications of the RL Trap
The RL trap applies wherever intermediate steps are observable but rewards are outcome-only:
Whenever reward granularity is coarser than behavior granularity, there is room for "fake doing" and "flailing." This is a structural defect of outcome-only RL, not a domain-specific quirk.
Limitations and Open Questions
1. Model access: all experiments are on open-source models. Step-level VEG requires token-level probabilities; closed models (OpenAI o3/o4-mini) couldn't be tested. Do they share the illusion? Unknown. 2. Limited tool set: only crop-and-zoom tested. OCR, segmentation, video frame selection, and external search may have different calibration properties. 3. RL trap is a hypothesis: establishing the mechanism requires matched training experiments varying only the reward signal—left as future work.
Conclusion: From "Can Use Tools" to "Actually Uses Tools"
This paper turns an intuitive suspicion ("is the model really using the magnifying glass?") into an operational causal framework (three intervention levels + VEG), then systematically exposes a structural problem across six models.
The most striking finding: aggregate accuracy rises, but the gains concentrate in a minority of calibrated trajectories. Most tool calls are ritual or waste. This is not a capability problem but a training-objective problem—outcome-only RL cannot separate real from fake looking.
Practical takeaways:
---
Paper: The Illusion of Visual Tool-Use: A Causal Audit of Thinking with Images Authors: Zhiheng Wang, Bo Peng, Lai Wei, Chaochao Lu Institutions: Shanghai AI Lab / Shanghai Jiao Tong University / Shanghai Innovation Institute arXiv: https://arxiv.org/abs/2608.06270 Code: https://github.com/OpenCausaLab/CauAudit License: CC BY 4.0