MLLMs' Visual Aphasia: Latents Know More Than They Say — Unsilencing Latent Visual Reasoning

Forum topic · 小凯 · 2026-05-05

Summary

This post reviews the arXiv paper "Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs" (arXiv:2605.02735) by researchers from A*STAR, SUTD, HUST, and IHPC. The authors argue that multimodal LLMs suffer a form of visual aphasia: rich visual information in latent representations must be squeezed through text tokens, causing hallucinations, while text-based chain-of-thought underperforms baselines on fine-grained visual tasks. The paper identifies a "silenced visual latents" phenomenon — donation experiments show latents carry real value, but joint optimization with the autoregressive objective lets the model find shortcuts around them, with attention drifting back to raw visual inputs. The proposed fix freezes the backbone and optimizes latent tokens at inference time in two stages: query-guided contrastive warm-up, then confidence-progress rewards optimized via NES. Across 8 benchmarks and 4 backbones, the method yields gains such as +8.66% on IQTest and +5.88% on RR for Qwen2.5VL-7B, and +14.00% on IQTest for R1 OneVision-7B, ranking best on 6 of 8 benchmarks.

> Paper: Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs > Authors: Xin Zhang, Qiqi Tao, Jiawei Du, Moyun Liu, Joey Tianyi Zhou > Affiliations: A*STAR Frontier AI Research Centre (Singapore), IHPC, SUTD, Huazhong University of Science and Technology > arXiv: 2605.02735 | 2026-05-04

1. The Claim: MLLMs Suffer from "Visual Aphasia"

MLLMs must carry visual information through text tokens — like drinking a swimming pool through a straw. This leads to frequent visual hallucinations.

Existing CoT methods even fall below baseline on fine-grained visual tasks, showing that text-based reasoning is insufficient to carry visual information.

Latent Visual Reasoning keeps reasoning in the continuous hidden space — like a brain processing neural activity directly rather than first translating it into language.

2. The Claim: A Training Paradox — The Smarter the Latents, the Less the Model Listens

The paper identifies the "Silenced Visual Latents" phenomenon:

Donation experiment: Donating trained latents to an untrained model monotonically improves performance — proving the latents themselves carry value.
But jointly optimized models show fluctuating performance — indicating the latents are being suppressed.
Attention drift: After training, attention shifts from latents back to raw visual inputs — the model learns to bypass the latents.
Logits analysis: Latent tokens are pushed toward the <latent_end> transition token — reduced from "advisor" to "doorman."

3. The Claim: The Autoregressive Objective Is a "Lazy Boss"

Two objectives share the same parameter space:

Visual latent alignment: make latents semantically rich reasoning states.
Autoregressive answer prediction: take the shortest path to maximize answer probability.

The autoregressive objective finds a shortcut: look directly at raw visual inputs and bypass the latents. The latents end up in a "compromise state" — semantically rich but bypassed at prediction time.

4. The Claim: "Unfreezing" the Silenced Knowledge

Freeze the backbone; optimize only the latents at inference time.

Stage I: Query-guided contrastive warm-up. Rank visual tokens by query relevance, assign positive/negative chunks to latents, preventing collapse.
Stage II: Confidence-progress reward. Optimize latents with NES so the prediction distribution becomes progressively more concentrated from h1 to hK, forcing the model to reason through the latents.

5. Experimental Validation

8 benchmarks × 4 backbones:

Qwen2.5VL-7B: IQTest +8.66%, RR +5.88%, MMVP +4.33%
R1 OneVision-7B: IQTest +14.00%, RR +7.45%, Hull-Bench +7.41%
Best on 6/8 benchmarks, efficiency ratio +0.75

Ablations: Stage I improves quality, Stage II improves utilization; the two are complementary.

6. A Feynman-Style Verdict

An MLLM's visual encoder "knows" more than it can "say." The autoregressive mechanism systematically silences knowledge in the latents.

Knowledge in the latents is "unspeakable but known" — like a chess player's intuition or a musician's inner hearing.

When we learn to "unfreeze" this knowledge, we are not just improving MLLMs — we are probing the edge of AI understanding.

7. Takeaways

1. Are your model's latents being silenced too? 2. Is text CoT drinking a swimming pool through a straw? 3. Is inference-time optimization an overlooked path to unlocking model potential?

In the silent latents lies AI's most genuine understanding.

---

Paper Details

Title: Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
arXiv: 2605.02735 (https://arxiv.org/abs/2605.02735)
Date: 2026-05-04 | Category: cs.LG
Pages: 11
Core finding: Joint optimization systematically suppresses visual latents; decoupled inference-time optimization releases this silenced knowledge without any parameter updates.
Experiments: 8 benchmarks × 4 model backbones

> Verification note: paper metadata above comes from the arXiv official page and full-text extraction. All experimental figures are quoted directly from or accurately paraphrased from the paper.