Monet: Reasoning in Latent Visual Space — A Breakthrough for Multimodal AI

Forum topic · ✨步子哥 · 2026-01-08

Summary

Monet is a multimodal large language model that performs visual reasoning in latent visual space rather than over raw pixels. Developed jointly by Peking University, Kuaishou, and MIT, the system enables continuous mental simulation inside a high-dimensional latent space using the manifold hypothesis, allowing efficient “visual imagination” without the curse of dimensionality. The framework combines a three-stage supervised fine-tuning pipeline (distillation warm-up, target latent acquisition, and autonomous embedding generation) with VLPO, a reinforcement-learning algorithm that incorporates continuous latent variables into policy-gradient optimization. Experiments report substantial gains over strong baselines such as GPT-4V on both standard reasoning benchmarks and out-of-distribution abstract tasks, with reported accuracy of 54.5% versus 48.5% on in-distribution tests and 33.7% versus 22.0% on OOD abstract reasoning. Potential applications span robotics, disaster response, and medical decision support.

Monet: Reasoning in Latent Visual Space

Overview

Monet is a multimodal large language model (MLLM) designed to move beyond simple pixel recognition and image captioning. Jointly developed by Peking University, Kuaishou, and MIT, the system endows AI with an "eye of imagination," enabling continuous mental simulation in a high-dimensional latent visual space.

Core Concept: Beyond Pixels

Monet challenges the conventional "see-then-describe" paradigm of vision-language models. Instead of operating on raw images, it performs reasoning directly on learned latent representations, allowing the model to *imagine* visual transformations.

Manifold Hypothesis

Underlying the approach is the manifold hypothesis: high-dimensional visual data concentrates on a low-dimensional manifold. Monet follows this "oasis path through the desert," conducting mental simulation along the manifold to avoid the curse of dimensionality and to produce semantically coherent visual thoughts.

Technical Architecture

The training framework combines Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).

SFT — Distillation Fine-Tuning (three stages)

1. Stage 1 (Warm-up): adaptation to interleaved image-text reasoning. 2. Stage 2 (Target acquisition): obtain high-quality target latent embeddings. 3. Stage 3 (Autonomous generation): generate embeddings without auxiliary images, simulating internalized visual imagination.

VLPO — Variational Latent Policy Optimization

A reinforcement learning approach that integrates continuous latent variables into the policy gradient. Instead of relying only on text, VLPO lets the reward signal directly optimize a model's "visual intuition."

Experimental Results

Monet was evaluated against strong baselines (including a GPT-4V-style SFT+GRPO setup) on standard reasoning benchmarks and on out-of-distribution (OOD) abstract reasoning tasks.

| Benchmark | Baseline (SFT+GRPO) | Monet (VLPO) | |---|---|---| | Standard reasoning | 48.5% | 54.5% | | OOD abstract reasoning | 22.0% | 33.7% |

Monet significantly outperforms baselines on both in-distribution and OOD abstract tasks, suggesting that latent-space reasoning generalizes better than pixel-conditioned reasoning.

Applications

Robotics & disaster response: simulate complex, hazardous environments and plan safe navigation paths before physical execution.
Medical AI: model disease progression and support diagnostic decision-making.
Embodied agents: preview the consequences of actions in a learned world model.

When machines acquire a mental "world model," they can rehearse actions internally—opening a new chapter for AI operating in the physical world.

References

Project poster summary, Monet Research Team, 2025.
Affiliations: Peking University · Kuaishou · MIT.