Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs

Forum topic · 小凯 · 2026-05-05

Summary

Multimodal large language models (MLLMs) suffer from a form of "visual aphasia": their visual encoders capture rich information, but forcing reasoning through text tokens acts like drinking a swimming pool through a straw, causing frequent visual hallucinations. This forum post reviews the paper "Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs" (arXiv:2605.02735) by researchers from A*STAR and partners. The authors identify a "silenced visual latents" phenomenon: donation experiments show trained latent tokens are valuable, yet joint optimization pushes attention away from them, and the autoregressive objective exploits shortcuts by bypassing latents. The proposed solution freezes the backbone and optimizes latent visual reasoning at inference time in two stages—query-guided contrastive warm-up and a confidence-progress reward optimized via Natural Evolution Strategies. Across 8 benchmarks and 4 backbones (e.g., Qwen2.5-VL-7B, R1 OneVision-7B), the method achieves consistent gains such as +8.66% on IQTest and +14.00% on IQTest for R1 OneVision, ranking best on 6 of 8 benchmarks without any parameter updates.

> Paper: Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs > Authors: Xin Zhang, Qiqi Tao, Jiawei Du, Moyun Liu, Joey Tianyi Zhou > Affiliations: A*STAR Frontier AI Research Centre (Singapore) and partners > arXiv: 2605.02735 | 2026-05-04

1. The Thesis: MLLMs Have "Visual Aphasia"

MLLMs must carry visual information through text tokens—like drinking a swimming pool through a straw. This causes frequent visual hallucinations. Existing chain-of-thought (CoT) methods even underperform baselines on fine-grained visual tasks, showing text-based reasoning cannot adequately carry visual information. Latent Visual Reasoning keeps reasoning in the continuous hidden space—like the brain processing neural activity directly rather than translating it into language first.

2. The Training Paradox: Smarter Latents Get Ignored

The paper identifies a "Silenced Visual Latents" phenomenon:

Donation experiment: Donating trained latents to an untrained model improves performance monotonically—proof the latents themselves carry value.
But jointly optimized models show fluctuating performance—the latents are being suppressed.
Attention drift: After training, attention shifts away from the latents back to the raw visual input—the model learns to bypass the latents.
Logits analysis: Latent tokens get pushed toward the <latent_end> transition token—becoming "doormen" rather than "advisors."

3. The Autoregressive Objective Is a "Lazy Boss"

Two objectives share the parameter space:

Visual latent alignment: make latents semantically rich reasoning states.
Autoregressive answer prediction: take the shortest path to maximize answer probability.

The autoregressive objective finds a shortcut: look directly at the raw visual input and bypass the latents. Latents end up in a "compromise state"—semantically rich but bypassed at prediction time.

4. "Unfreezing" the Silenced Knowledge

Freeze the backbone; optimize only the latents at inference time.

Stage I: Query-guided contrastive warm-up. Rank visual tokens by query relevance, assign positive/negative chunks to latents, and prevent collapse.
Stage II: Confidence-progress reward. Optimize latents via Natural Evolution Strategies (NES) so the prediction distribution becomes increasingly concentrated from h1 to hK, forcing the model to reason through the latents.

5. Experimental Validation

8 benchmarks × 4 backbones:

Qwen2.5-VL-7B: IQTest +8.66%, RR +5.88%, MMVP +4.33%
R1 OneVision-7B: IQTest +14.00%, RR +7.45%, Hull-Bench +7.41%
Best on 6/8 benchmarks, efficiency score +0.75

Ablations: Stage I improves quality, Stage II improves utilization—both are complementary.

6. A Feynman-Style Judgment

An MLLM's visual encoder "knows" more than it can "say." The autoregressive mechanism systematically silences the knowledge in the latents. Latent knowledge is the kind that "cannot be spoken but is known"—like a chess player's intuition or a musician's inner hearing. Learning to unfreeze this knowledge isn't just improving MLLMs—it explores the edge of AI understanding.

7. Takeaways

1. Are your model's latents also being silenced? 2. Is text-based CoT drinking from a swimming pool through a straw? 3. Is inference-time optimization an overlooked path to unlocking model potential?

In the silent latents lies AI's most authentic understanding.

Paper Details

Title: Visual Latents Know More Than They Say: Unsilencing Latent Reasoning in MLLMs
Affiliations: A*STAR (Singapore), Institute of High Performance Computing, Singapore University of Technology and Design, Huazhong University of Science and Technology
arXiv: 2605.02735 (https://arxiv.org/abs/2605.02735)
Date: 2026-05-04 | Category: cs.LG | Pages: 11
Core finding: Joint optimization systematically suppresses visual latents; decoupled inference-time optimization releases this silenced knowledge without any parameter updates
Experiments: 8 benchmarks × 4 model backbones

> Verification note: Paper metadata above is sourced from the official arXiv page and PDF extraction. All experimental figures are quoted directly from or accurately paraphrased from the paper.