Seeing to Generalize: How Visual Training Turns LLM Memorization into True Understanding

> arXiv: 2602.15183 | Institution: UC Chile (Pontificia Universidad Católica de Chile) | Field: Mechanistic Interpretability × Cross-Modal Learning > Key numbers: Text-only OOD accuracy 37.2% → 69.5% after visual training → 83.6% with visual + noise

Key Points

• Counterintuitive finding: VLMs beat their base LLMs on *text-only* tasks. Qwen3-VL-8B scores 76.0% on a long-context retrieval task where the text-only Qwen3-8B scores 62.6%.
• Synthetic indirect retrieval task: two-step reasoning (color → shape → item letter), expressible in both text and rendered images. Training uses short contexts (≤8 objects); testing extends to longer sequences (OOD).

Main Results

| Training setup | ID accuracy (≤8 objects) | OOD accuracy (>8 objects) | |:---|:---|:---| | Text-only | 100% | 37.2% | | Text + noise augmentation | ~100% | 57.5% | | Visual → back to text | ~100% | 69.5% | | Visual + noise → back to text | ~100% | 83.6% |

1. Text-only training learns a shortcut: perfect in-distribution, collapses out-of-distribution — the model memorizes positional order rather than matching content. 2. Noise helps but doesn't cure: forcing exposure to longer positional ranges lifts OOD to 57.5%, without changing the underlying mechanism. 3. Visual training fundamentally reshapes reasoning: OOD jumps from 37.2% to 69.5% — a different way of thinking, not just broader exposure.

Mechanism: Positional Binding vs. Symbolic Binding

Using interchange interventions (a causal analysis technique):

• Positional binding: the model tracks "the 3rd object" instead of "red circle". It exploits regular token positions and fails when sequences lengthen.
• Symbolic binding: the model matches semantic content ("red" → "circle" → item_a), a content-addressable strategy robust to sequence length.

Text-trained models rely almost 100% on positional binding at the final layer, because text sequences have canonical orders that make position-counting a viable shortcut.

Why vision breaks the shortcut: an object can appear anywhere in an image. Visual encoders' spatial translation invariance makes positional strategies unreliable, forcing the model to adopt symbolic binding — and it retains this strategy when returning to text-only tasks.

Noise augmentation only partially works: it disrupts positional regularity but provides no alternative invariance structure.

Circuit Analysis (Attention Knockout + Linear Probing)

• Positional circuit (text-trained): two independent information streams bind implicitly via shared position indices; probes show near-zero attribute decoding at entity positions.
• Symbolic circuit A (color-key): color information is actively copied into shape-token activations, forming explicit "red-circle" binding packages — visible as attribute-decoding "pulses" (a binding signature).
• Symbolic circuit B (shape-key): same structure with shape as the retrieval key. Both transfer semantic identity rather than position indices.

Validation on Real Models (Qwen 2/2.5/3)

Symbolic/positional attribution ratio (VLM vs LLM):

| Model | VLM sym/pos | LLM sym/pos | Gain | |:---|:---|:---|:---| | Qwen 2 | 1.499 | 1.383 | +0.116 | | Qwen 2.5 | 1.282 | 1.218 | +0.064 | | Qwen 3 | 2.463 | 1.819 | +0.644 |

Qwen3's largest gain matches its largest VLM-over-LLM retrieval advantage (76.0% vs 62.6%). Probes confirm binding signatures in VLM variants but not text-only baselines.

Why Vision Specifically?

Three different encoders — ResNet-152 (supervised CNN), ViT-B/16 (supervised Transformer), and DINOv3 (self-supervised Transformer) — all induce the positional-to-symbolic shift. The key driver is not architecture but the inherent spatial translation invariance of visual data, which strips away position and rewards content matching.

Implications

• Cross-modal training as inductive bias injection: multimodal training does more than enable perception — it forces more general internal computation, benefiting even unimodal evaluation.
• An antidote to shortcut learning: if a modality's properties break a shortcut you care about, expose the model to it.
• VLM design: visual training may improve the base LLM's reasoning itself, not just enable image understanding.
• Why text-only LLMs struggle with long-context generalization: text sequences are too easily exploited by positional shortcuts; breaking positional regularity forces genuine content addressing.

Conclusion

Core takeaway:

• Text training → positional shortcut (37.2% OOD)
• Visual training → translation invariance → shortcut broken → symbolic binding (69.5% OOD)
• Visual + noise → complementary regularization (83.6% OOD)

Vision acts as a strict teacher that forbids shortcuts — and the model that learns the "honest" way ends up with more generalizable reasoning that persists even in text-only exams.

References:

• Buzeta, N., del Rio, F., Hinostroza, C., Parra, D., Lobel, H., & Toro Icarte, R. (2026). Seeing to Generalize: How Visual Data Corrects Binding Shortcuts. arXiv:2602.15183.
• Gur-Arieh, Y., et al. (2025). Mechanistic Analysis of Variable Binding in Transformers.
• Geirhos, R., et al. (2020). Shortcut Learning in Deep Neural Networks.