Imagine closing your eyes and clearly seeing an ordinary wooden chair slowly rotating in your mind — the curve of the backrest, the texture of the seat, the shadows of four legs, round and round, like a private film screening. This is not magic, but the most natural human "mental simulation." For artificial intelligence, however, it has long been an elusive holy grail. Until Monet came along.
This paper comes from a joint team at Peking University, Kuaishou, and MIT, titled simply yet ambitiously: *Monet: Reasoning in Latent Visual Space*. It is no longer satisfied with "describing images," but attempts to give AI a true "imaginative eye" — performing visual reasoning directly in an abstract, high-dimensional mathematical space, just as humans silently rehearse the physical world in their minds.
The Thought Experiment of the Chair: Where the Human-AI Gap Begins
Close your eyes and imagine a chair rotating. You can do it almost instantly: the chair turns from front to side, the legs visually shorten, the curve of the backrest gradually appears, and you can even predict it will fully block the seat from behind. No verbal description needed — pure intuition.
But give the task to today's top multimodal models like GPT-4V, and what happens? They typically first translate the image into text ("This is a four-legged wooden chair, facing forward..."), then reason in language ("If rotated 90 degrees, the left legs would shorten..."), and finally translate text back into an image. The whole process is like walking with a crutch — accurate, but clumsy, and prone to losing detail in translation.
Monet's ambition: throw away the crutch and rotate the chair directly in "visual latent space." It doesn't speak; it simulates. The result is smoother reasoning closer to human intuition.
> What is latent space? Simply put, it compresses an image into a high-dimensional vector (say, a few thousand numbers). In this space, similar images cluster together, and different angles of the same chair form a smooth curve. Moving the vector is equivalent to rotating the object — no need to pass through language as an intermediary.
Why Visual Reasoning Needs a Hidden Garden
The visual world is too noisy. A 512×512 color image has over 780,000 pixels — 780,000 dimensions. If all possible images were placed in this high-dimensional space, the vast majority of positions would be meaningless noise: grotesque, physically implausible, nightmarish frames.
Real-world images occupy only a tiny fraction — like a narrow highway across the Sahara. Drivers must strictly follow the road or be instantly lost. This "highway" is the mathematical manifold.
The manifold hypothesis holds that natural images are not uniformly scattered in high-dimensional space, but concentrated on low-dimensional, curved substructures. Monet's insight: rather than painstakingly navigating pixel space, first jump to latent space — a "hidden garden" where manifolds are clearer, operations more continuous, and reasoning more reliable. Think of an astronaut searching for habitable planets: flying randomly through 3D space is inefficient, but in a compressed coordinate system, planetary orbits become clean ellipses and navigation becomes a hundred times easier. Monet is that "star map" for AI.
Latent Distillation: Training a Blind Painter
How do you teach a model to think in latent space? The team designed Latent Distillation — a teacher-student game without language.
The teacher is a powerful vision-language model (VLM) that can describe images. The student is a purely visual policy network that initially knows nothing. The teacher doesn't tell the student "what happens when the chair rotates 90 degrees"; instead, it shows many image sequences (like video frames of a chair gradually rotating) for the student to imitate those trajectories in latent space.
The student keeps trying to generate intermediate latent vectors, and the teacher uses reward signals (like scoring) to say "closer" or "off track." Over time, the student learns to predict the next step purely through visual intuition, without words. It's like teaching a blind painter: you don't say "the nose you drew is crooked" — you hold their hand and trace outlines repeatedly until they can draw portraits independently.
VLPO: Solving RL in Continuous Space
Traditional RLHF succeeds in language because language is discrete — each word is a clear action, and rewards are easy to assign. But visual latent space is continuous, with no clear "steps." You can't say "adding 0.3 to the 5th dimension is the correct action." Standard policy gradients fail due to variance explosion or vanishing gradients.
The team therefore invented VLPO (Visual-Latent Policy Optimization) — RL designed for continuous vision. Instead of rewarding discrete tokens, it directly evaluates the physical plausibility and task completion of an entire latent trajectory. If old RL was like a written exam with clear right/wrong answers, VLPO is like judging gymnastics: fluidity, difficulty, and landing stability all get scored. The judge is a vision-based reward model trained on human preferences that can tell "is this rotation natural? Is the sense of gravity right?"
Experiments show VLPO significantly outperforms traditional methods on multi-step visual reasoning, especially in continuous transformation scenarios like object collisions and fluid simulation.
Beyond Numbers: Handling the Unseen World
Monet was tested on multiple benchmarks:
- CLEVRER (causal reasoning in videos): Monet beats GPT-4V by 10-15%.
- A-OKVQA (open-ended visual QA): more stable on compositional questions.
- Most striking: out-of-distribution (OOD) generalization — with unseen object colors, materials, and combinations, Monet stays robust while text-dependent models collapse due to "vocabulary bias."
| Dimension | Monet (latent reasoning) | GPT-4V (text CoT) | Traditional VLM (e.g., CLIP) | |---|---|---|---| | CLEVRER accuracy | 85-95% | 70-80% | 60-70% | | OOD generalization | High | Medium | Low | | Compute cost (visual tasks) | Lower | Higher (text conversion) | Medium | | Continuity handling | Native | Discrete simulation | Almost none | | Robotics potential | Strong | Medium | Weak |
Looking Ahead
If approaches like Monet scale up: a rescue robot in earthquake rubble could locally simulate in latent space — "if I push aside this concrete slab, will the person below be harmed again?" — instead of uploading frames, translating to text, and planning paths. In medicine, AI might "rehearse" surgeries in latent space: how vessels bend, how tissue deforms, all in silent mathematical flow. Combined with 3D diffusion and video generation, AI might build a complete "inner world" — an interactive physics simulator.
Monet is still a beginning: the current version is trained mainly on synthetic data, and the real world is noisier with more complex manifolds. Compute cost, data bias, and interpretability remain hurdles. But the direction is exciting: from "translating vision into language" to "computing vision directly" — like the key leap in human cognition from needing to speak aloud to think, to silent intuitive insight.
When AI can close its eyes and see a chair rotating — and even more complex physics — we may truly be entering a new era of intelligence.
References
1. Monet: Reasoning in Latent Visual Space — arXiv preprint: https://arxiv.org/abs/2410.23335 2. Project Monet GitHub Repository — https://github.com/project-monet/monet 3. MIT CSAIL News: New AI model can reason about the visual world — https://www.csail.mit.edu/news/new-ai-model-can-reason-about-visual-world 4. Peking University AI Lab Paper Announcement — https://ai.pku.edu.cn/research/papers/monet 5. Kuaishou Research Technical Report — https://research.kuaishou.com/papers/monet-latent-reasoning