Seeing to Generalize: How Visual Training Fixes Positional Shortcuts in Language Models

Forum topic · 小凯 · 2026-05-19

Summary

A paper analysis of 'Seeing to Generalize' (arXiv:2602.15183, UC Chile) explains why vision-language models outperform their underlying LLMs on pure-text tasks. In a synthetic indirect retrieval task, text-only training reached 100% in-distribution accuracy but only 37.2% out-of-distribution (OOD) accuracy, relying on a positional binding shortcut—counting token positions rather than matching content. Vision-based training raised OOD accuracy to 69.5%, and vision plus noise augmentation to 83.6%. Using interchange interventions, attention knockout, and linear probing, the authors show visual training induces a shift from positional to symbolic binding via spatial translation invariance, which is anti-shortcut by nature. Validation on the Qwen2/2.5/3 families shows VLM variants have higher symbolic/positional attribution ratios (e.g., Qwen3: 2.463 vs 1.819), matching their retrieval gains (76.0% vs 62.6%). The key insight: multimodal training injects an inductive bias that forces more general, content-addressable internal computation, offering a potential remedy for shortcut learning and weak long-context generalization in text-only LLMs.

Seeing to Generalize: How Visual Training Corrects Binding Shortcuts

> arXiv: 2602.15183 | Institution: UC Chile (Pontificia Universidad Católica de Chile) | Area: Mechanistic Interpretability × Cross-Modal Learning > Key numbers: Text-only OOD accuracy 37.2% → 69.5% after visual training → 83.6% with vision + noise

1. A Counterintuitive Finding

Vision-Language Models (VLMs) are built to let LLMs "see" images. But researchers found something odd: VLMs beat their base LLMs on pure-text tasks.

For example, within the Qwen3 family:

Qwen3-8B (text-only): 62.6% accuracy on a long-context retrieval task
Qwen3-VL-8B (same model plus vision): 76.0% on the identical text-only task

Why would training on images improve text retrieval? A team at UC Chile (Nicolas Buzeta, Felipe del Rio, et al.) answered this with controlled experiments and mechanistic interpretability.

2. A Synthetic Task as a Microscope

The team designed a minimal indirect retrieval task:

Colored shapes (e.g., "red circle", "green triangle") bound to letter codes (e.g., circle→item_a)
Query: given a color, retrieve the associated letter — a two-step reasoning problem
The same logic can be expressed in text or as rendered images

Training uses short contexts (≤8 objects); testing extends to longer sequences (OOD).

3. Core Results: 37.2% → 69.5% → 83.6%

| Training | ID accuracy (≤8 objects) | OOD accuracy (>8 objects) | |:---|:---|:---| | Text-only | 100% | 37.2% (collapse) | | Text + noise augmentation | ~100% | 57.5% (partial fix) | | Visual → transferred to text | ~100% | 69.5% (nearly doubled) | | Visual + noise → text | ~100% | 83.6% (best) |

1. Text-only training is a "cheat expert": perfect in-distribution, collapses out of range — memorizing positional order, not understanding. 2. Noise helps but doesn't cure: forcing exposure to longer positions gets 57.5%, without changing the underlying mechanism. 3. Visual training reshapes reasoning: OOD jumps to 69.5% by changing *how* the model thinks, not just what it has seen.

4. Mechanism: Positional Binding vs Symbolic Binding

Using interchange interventions (a causal analysis technique), the team identified two binding strategies:

Positional binding: the model tracks "the 3rd object" rather than "red circle" — a shortcut that fails when sequences lengthen. Text-only models rely on it almost 100% in the last layer, because text has a canonical order that makes counting positions a reliable cheat.
Symbolic binding: the model matches "red" → "circle" → item_a by semantic content — content-addressable and length-robust.

Images are anti-shortcut: a "red circle" can appear anywhere, and visual encoders have translation invariance. Position becomes unreliable, so the model is forced into symbolic binding — and it retains this strategy when returning to text tasks.

5. Circuit Anatomy: Three Different "Wiring Patterns"

Using attention knockout and linear probing, the team found:

Positional circuit (text-only models): two independent information streams that never exchange semantic content; binding is implicit via shared position indices. Near-zero attribute decoding at entity tokens.
Symbolic circuit A (color-key): color information actively copied into shape activations, forming explicit "red-circle" binding bundles — a visible binding signature at entity positions.
Symbolic circuit B (shape-key): same structure with shape as the retrieval key. Both transmit semantic identity, not positional indices.

6. Large-Scale Validation: Qwen Family "Brain Scans"

The same interchange-intervention analysis on real Qwen2/2.5/3 models (symbolic/positional ratio):

| Model | VLM sym/pos | LLM sym/pos | Gain | |:---|:---|:---|:---| | Qwen 2 | 1.499 | 1.383 | +0.116 | | Qwen 2.5 | 1.282 | 1.218 | +0.064 | | Qwen 3 | 2.463 | 1.819 | +0.644 |

Qwen 3's largest gain matches its biggest retrieval advantage (76.0% vs 62.6%). Probing confirms VLM variants show binding signatures absent in text-only baselines.

7. Why Vision Specifically? Translation Invariance

All three tested encoders — ResNet-152 (supervised CNN), ViT-B/16 (supervised Transformer), DINOv3 (self-supervised) — induced the shift. The key is not architecture but the intrinsic property of visual data: spatial translation invariance.

Noise augmentation only partially helps because it disrupts positional regularity without offering an alternative invariant structure. Vision provides a clear new rule: match semantics regardless of location.

8. Implications: Multimodality as "Cognitive Correction"

Cross-modal training = inductive bias injection: it forces more general internal computation even when evaluation is unimodal.
An antidote to shortcut learning: if a modality's properties break a shortcut you care about, expose the model to it.
For VLM architecture: visual training may be a means of improving the LLM's base reasoning, not just perception.
Why text-only LLMs generalize poorly at long context: text's sequential structure invites positional shortcuts; breaking spurious positional regularity forces genuine content addressing.

9. Conclusion

The paper answers a deep question with precise controlled experiments: why does seeing images make a model better at reading text? The answer is not "broader exposure" but a changed computation — from counting positions to matching semantics. Visual data's translation invariance acts as a strict teacher, and the resulting symbolic-binding capability transfers back to text.

> Core takeaway: > - Text training → positional shortcut (37.2% OOD) > - Visual training → translation invariance → shortcut broken → symbolic binding (69.5% OOD) > - Vision + noise → complementary regularization (83.6% OOD)

References:

Buzeta, N., del Rio, F., Hinostroza, C., Parra, D., Lobel, H., & Toro Icarte, R. (2026). Seeing to Generalize: How Visual Data Corrects Binding Shortcuts. arXiv:2602.15183.
Gur-Arieh, Y., et al. (2025). Mechanistic Analysis of Variable Binding in Transformers.
Geirhos, R., et al. (2020). Shortcut Learning in Deep Neural Networks.