MLLMs' Visual Aphasia: Latents Know 90% But Can Only Say 10% — Unsilencing Latent Visual Reasoning

Forum topic · 小凯 · 2026-05-05

Summary

A forum post discusses the paper 'Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs' (arXiv:2605.02735) by researchers at A*STAR and collaborators. The paper argues that multimodal large language models suffer from a form of 'visual aphasia': forcing visual information through text tokens bottleneck and autoregressive objectives systematically suppresses semantically rich visual latents. Diagnostic evidence includes donation experiments showing trained latents improve untrained models, attention drift away from latents during training, and latent tokens being pushed toward the <latent_end> transition token. The proposed method freezes the backbone and optimizes latent visual reasoning at inference time in two stages: query-guided contrastive warm-up and a confidence-progression reward optimized via Natural Evolution Strategies. Across 8 benchmarks and 4 backbones, gains include +8.66% on IQTest and +5.88% on RR for Qwen2.5VL-7B, and +14.00% on IQTest for R1 OneVision-7B, achieving best results on 6 of 8 benchmarks without any parameter updates.

> Paper: Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs > Authors: Xin Zhang, Qiqi Tao, Jiawei Du, Moyun Liu, Joey Tianyi Zhou > Institutions: A*STAR Frontier AI Research Centre (Singapore) and collaborators > arXiv: 2605.02735 | 2026-05-04

---

1. The Thesis: MLLMs Suffer from "Visual Aphasia"

MLLMs must carry visual information through text tokens — like drinking a swimming pool through a straw. This causes frequent visual hallucinations.

Existing chain-of-thought (CoT) methods even underperform baselines on fine-grained visual tasks, showing that text-based reasoning is insufficient to carry visual information.

Latent Visual Reasoning keeps reasoning in the continuous hidden-layer space — analogous to how the brain processes neural activity directly rather than translating everything into language first.

2. The Training Paradox: The Smarter the Latents, the Less the Model Uses Them

The paper identifies a "Silenced Visual Latents" phenomenon:

Donation experiment: "Donating" trained latents to an untrained model yields monotonically increasing performance — proving the latents themselves carry value.
But jointly optimized models fluctuate in performance — indicating the latents are being suppressed.
Attention drift: after training, attention shifts from the latents back to the raw visual inputs — the model learns to bypass the latents.
Logits analysis: latent tokens get pushed toward the <latent_end> transition token — reduced from "advisor" to "doorman."

3. The Autoregressive Objective Is a "Lazy Boss"

Two objectives share the same parameter space:

Visual latent alignment: making latents semantically rich reasoning states
Answer autoregressive prediction: taking the shortest path to maximize answer probability

The autoregressive objective finds a shortcut: look directly at the raw visual inputs and bypass the latents. The latents settle into a "compromise state" — semantically rich but bypassed at prediction time.

4. "Unfreezing" the Silenced Knowledge

Freeze the backbone; optimize only the latents at inference time.

Stage I: Query-guided contrastive warm-up. Rank visual tokens by query relevance, assign chunks as positive/negative to the latents to prevent collapse.
Stage II: Confidence-progression reward. Use Natural Evolution Strategies (NES) to optimize the latents so the prediction distribution becomes progressively more concentrated from h₁ to h_K, forcing the model to reason through the latents.

5. Experimental Validation

8 benchmarks × 4 backbones:

Qwen2.5VL-7B: IQTest +8.66%, RR +5.88%, MMVP +4.33%
R1 OneVision-7B: IQTest +14.00%, RR +7.45%, Hull-Bench +7.41%
Best on 6/8 benchmarks, efficiency ratio +0.75

Ablations: Stage I improves quality, Stage II improves utilization — the two are complementary.

6. A Feynman-Style Verdict

An MLLM's visual encoder "knows" more than it can "say." The autoregressive mechanism systematically silences the knowledge in the latents.

Latent knowledge is the kind that is "known but cannot be spoken" — like a chess player's intuition or a musician's inner hearing.

When we learn to "unfreeze" this knowledge, we are not just improving MLLMs — we are exploring the edge of AI cognition.

7. Takeaways

1. Are your model's latents also being silenced? 2. Is text-based CoT drinking a swimming pool through a straw? 3. Is inference-time optimization an overlooked path to unlocking model potential?

In the silent latents lies AI's most authentic understanding.

---

Paper Details

Title: Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
Authors: Xin Zhang, Qiqi Tao, Jiawei Du, Moyun Liu, Joey Tianyi Zhou
Institutions: A*STAR (Singapore), Institute of High Performance Computing, Singapore University of Technology and Design, Huazhong University of Science and Technology
arXiv: 2605.02735 (https://arxiv.org/abs/2605.02735)
Date: 2026-05-04 | Category: cs.LG
Length: 11 pages
Core finding: Joint optimization systematically suppresses visual latents; decoupled inference-time optimization releases this silenced knowledge without any parameter updates
Experiments: 8 benchmarks × 4 model backbones

> Verification note: Paper metadata above is sourced from the official arXiv page and full-text PDF extraction. All experimental figures are quoted directly from, or accurately paraphrased from, the paper.