Monet: Reasoning in Latent Visual Space Beyond Images and Language

Monet, proposed by a joint team from Peking University, Kuaishou, and MIT, aims to give multimodal large language models (MLLMs) a human-like "mind's eye" — the ability to perform visual reasoning internally in a continuous latent visual space, rather than merely describing pixels or calling external tools.

Background

Current "Thinking with Images" methods inject visual evidence into chain-of-thought (CoT) reasoning via auxiliary images, cropping, highlighting, or external tools (depth estimation, code interpreters). These approaches remain confined to pixel space and lack flexible, internal visual abstraction. AI today largely *describes* images rather than truly *imagining* them.

Framework Overview

Monet's core idea: the model generates continuous embeddings as intermediate Visual Thoughts instead of explicit auxiliary images. Training has two stages:

1. Supervised fine-tuning (SFT) — a three-phase distillation pipeline (built on Qwen2.5-VL-7B). 2. Reinforcement learning — via Visual-Latent Policy Optimization (VLPO).

Three-Phase Distillation SFT

Two challenges motivate the design: (1) latent-visual alignment is computationally expensive, and (2) standard next-token prediction under-supervises the latent embeddings, while GRPO only optimizes discrete text tokens.

• Phase 1 (Warm-up): Standard SFT on Monet-SFT-125K (125,000 samples spanning real scenes, charts, OCR, geometry) so the model adapts to interleaved image-text reasoning and "obs tokens" encode visual information.
• Phase 2 (Target latent embeddings): A teacher-student setup. The teacher sees ground-truth auxiliary images; the student generates autoregressive latent embeddings that can attend to the original image via modified attention masks, while downstream text tokens cannot. Alignment uses cosine similarity loss across all hidden layers, and gradients for the alignment loss flow only through the latent embeddings, preventing shortcut behavior.
• Phase 3 (Autonomous generation): Re-initialized from Phase 1, the model learns to generate latent embeddings *without* auxiliary images, aligned to the Phase 2 targets.

Visual-Latent Policy Optimization (VLPO)

GRPO cannot compute probabilities for continuous embeddings. VLPO treats rollouts of latent embeddings as samples from a Gaussian centered at the current policy's output, giving the approximate policy probability:

> π_θ(h_old | Q, I, o_<t) ∝ exp(-||h_old - μ_θ||² / (2σ²))

This yields a policy ratio r_t(θ) that plugs into a PPO-style clipped objective, so reward signals directly shape latent embeddings as well as text tokens. When the advantage A_t > 0, the model is encouraged to produce embeddings closer to those leading to correct answers.

Experimental Results

• On real-world perception and reasoning benchmarks, both Monet-SFT and Monet+VLPO consistently outperform baselines (Qwen2.5-VL-7B, SFT+GRPO, crop/tool-based methods).
• Gains are especially large on out-of-distribution (OOD) abstract visual reasoning tasks, demonstrating generalization of latent-space reasoning.
• Ablations confirm that Phase 2's latent-visual alignment is key to success, Phase 3 (dropping auxiliary images) drives generalization, and VLPO is essential for hard tasks.

Illustrative chart values from the post (approximate):

| Model | Standard tasks (%) | OOD abstract tasks (%) | |---|---|---| | Baseline (SFT+GRPO) | 48.5 | 22.0 | | Monet (SFT) | 53.1 | 28.5 | | Monet (VLPO) | 54.5 | 33.7 |

The Manifold Hypothesis Perspective

The post explains Monet's effectiveness via the manifold hypothesis: high-dimensional data actually lies on low-dimensional manifolds — oases in a high-dimensional desert. SFT forces embeddings to capture the essential structure of the data distribution, so the model learns to move along the manifold. VLPO then penalizes embeddings that drift off-manifold (they yield wrong answers) and rewards on-manifold ones, keeping mental simulations within valid regions.

Outlook: Machines with Mental Models

If machines can mentally simulate in latent space, they gain internal causal models — imagining "what happens if I move this object?" Promising directions include:

• Disaster-response robotics: internally rehearsing movement consequences in unstable structures to plan safer paths.
• Medical prediction: simulating disease progression or treatment outcomes to support clinical decisions.

The post cautions that Monet is only an early step: it validates latent visual reasoning but does not yet cover temporal, causal, or commonsense simulation at scale. Extending the approach to broader multimodal, long-horizon planning is future work.