Monet: Reasoning in Latent Visual Space Beyond Images and Language

Forum topic · ✨步子哥 · 2026-01-08

Summary

Monet is a multimodal large language model framework developed by researchers from Peking University, Kuaishou, and MIT that enables AI visual reasoning directly in a continuous latent visual space rather than relying on pixel-space tools or intermediate images. The model generates latent embeddings as 'visual thoughts' during chain-of-thought reasoning. Training combines a three-stage distillation-based supervised fine-tuning pipeline (built on Qwen2.5-VL-7B with the Monet-SFT-125K dataset) with a novel reinforcement learning algorithm called Visual-Latent Policy Optimization (VLPO), which estimates probabilities of continuous latent embeddings via a Gaussian assumption so reward signals can directly shape latent reasoning. Experiments show consistent gains over baselines including SFT+GRPO models on real-world perception and reasoning benchmarks, with especially strong improvements on out-of-distribution abstract visual reasoning tasks. The article explains Monet's effectiveness via the manifold hypothesis—data lies on low-dimensional submanifolds within high-dimensional space—and discusses future applications such as disaster-response robots and medical prediction, where internal mental simulation could enable prediction, planning, and causal reasoning.

Key points

What Monet is: A framework from Peking University, Kuaishou, and MIT that lets multimodal LLMs reason in a *continuous latent visual space*, producing latent embeddings as intermediate "visual thoughts" instead of explicit auxiliary images or external tool calls. It moves beyond the "Thinking with Images" paradigm, which still operates in pixel space (cropping, highlighting, depth estimation, code interpreters).
Two identified challenges:

Three-stage distillation SFT pipeline

Stage 1 (Warm-up): Standard SFT on Monet-SFT-125K (125,000 samples spanning real scenes, charts, OCR, geometry) so the base model (Qwen2.5-VL-7B) learns interleaved image–text reasoning and "obs tokens" encode visual information.
Stage 2 (Target latent embeddings): A teacher model sees ground-truth auxiliary images; a student model generates autoregressive latent embeddings instead. Cosine-similarity alignment losses across all hidden layers match the student's hidden states to the teacher's, attention masks force text tokens to receive visual information only via the latents, and gradients flow *only* through the latent embeddings to prevent shortcut learning.
Stage 3 (Autonomous generation): The model is retrained without auxiliary images, aligning its self-generated latents with Stage 2 targets—enabling fully self-sufficient latent reasoning at inference time.

Visual-Latent Policy Optimization (VLPO)

GRPO cannot directly optimize continuous embeddings. VLPO treats old-policy latents as samples from a Gaussian centered on the current policy's output, yielding an approximate probability:

> π_θ(h_old | Q, I, o_<t) ∝ exp(-||h_old - μ_θ||² / (2σ²))

This gives a policy ratio r_t(θ) = π_θ(h_old) / π_old(h_old) used in a PPO-like clipped objective, so reward signals shape both text tokens *and* latent embeddings.

Results and analysis

Monet-7B consistently outperforms baselines (Qwen2.5-VL-7B, SFT+GRPO, cropping/tool-based methods) on real-world perception and reasoning benchmarks, with the largest gains on out-of-distribution abstract reasoning tasks.
Ablations confirm the latent–visual alignment mechanism (Stage 2) is key to success, Stage 3's image-free training drives generalization, and VLPO is essential for difficult tasks.

Theoretical lens: manifold hypothesis

High-dimensional data concentrates on low-dimensional manifolds; reasoning in pixel space is like wandering a desert, while latent reasoning follows the "correct road" along the manifold. SFT teaches the model to capture this structure; VLPO penalizes latents that drift off-manifold (leading to wrong answers).

Outlook

Monet represents a step toward internal "mental models": machines that can simulate scenarios (e.g., a rescue robot imagining structural stability, or an AI predicting treatment outcomes in medicine). The authors note current limits—Monet is validated mainly on visual reasoning, and richer temporal, causal, and commonsense simulation remains future work.