Monet: Reasoning in Latent Visual Space for Multimodal AI

Forum topic · ✨步子哥 · 2026-01-08

Summary

Monet is a multimodal large language model (MLLM) framework developed by a joint team from Peking University, Kuaishou, and MIT that enables visual reasoning directly in a latent visual space rather than relying on explicit pixel-level image generation. Grounded in the manifold hypothesis, Monet performs continuous 'mental simulation' along low-dimensional manifolds in high-dimensional space, avoiding the curse of dimensionality. The training pipeline combines supervised fine-tuning (SFT) with distillation in three stages—warm-up on interleaved image-text reasoning, obtaining high-quality target latent embeddings, and generating embeddings autonomously without auxiliary images—followed by VLPO, a reinforcement learning policy optimization method that incorporates continuous latent variables into policy gradients to optimize 'visual intuition' from reward signals. On the poster's reported benchmarks, Monet (SFT+VLPO) achieves 54.5% accuracy on standard reasoning tasks versus 48.5% for an SFT+GRPO baseline, and 33.7% on out-of-distribution (OOD) abstract reasoning versus 22.0%, also surpassing baselines such as GPT-4V. Potential applications include disaster-response robots simulating complex environments for safe path planning and medical AI simulating disease progression to support diagnosis.

Monet: Reasoning in Latent Visual Space

A joint team from Peking University, Kuaishou, and MIT presents Monet, a framework that lets multimodal large language models (MLLMs) move beyond "describing what they see" and instead perform reasoning inside a latent visual space — a kind of "mind's eye" for AI.

Core Concept

Rather than simple pixel recognition, Monet conducts continuous mental simulation in a high-dimensional latent visual space. The approach is grounded in the manifold hypothesis: data in high-dimensional space concentrates on low-dimensional manifolds. Monet performs "mental simulation" on these low-dimensional manifolds, avoiding the curse of dimensionality.

Technical Architecture

The training pipeline consists of two components:

1. SFT (Distilled Supervised Fine-Tuning) — three stages:

Stage 1: Warm-up adaptation to interleaved image-text reasoning
Stage 2: Acquiring high-quality target latent embeddings
Stage 3: Autonomously generating embeddings without auxiliary images

2. VLPO (Variational Latent Policy Optimization):

Incorporates continuous latent variables into the reinforcement learning policy gradient, directly optimizing the model's "visual intuition" from reward signals.

Experimental Results

Monet significantly outperforms baseline models (including GPT-4V) on both standard reasoning tasks and out-of-distribution (OOD) abstract reasoning, as shown in the poster's reported accuracies:

| Model | Standard Reasoning | OOD Abstract Reasoning | |---|---|---| | Baseline (SFT + GRPO) | 48.5% | 22.0% | | Monet (SFT + VLPO) | 54.5% | 33.7% |

Future Outlook and Applications

When machines possess a "mental model," they can rehearse the consequences of actions in their minds before acting — opening a new chapter for AI in the physical world:

Disaster-response robotics: simulating complex environments to plan safe paths
Medical prediction: simulating disease progression to assist clinical decision-making

*Source: Monet Research Team, 2025.*