The Machine's Imaginative Eye: How Monet Sees Chairs Spin with Its Eyes Closed

Forum topic · ✨步子哥 · 2026-01-08

Summary

A Chinese tech forum post reviews Monet: Reasoning in Latent Visual Space, a paper by a joint team from Peking University, Kuaishou, and MIT. The post argues that leading multimodal models like GPT-4V perform visual reasoning by first converting images to text, then chain-of-thought reasoning, then converting back—an error-prone, translation-heavy pipeline. Monet instead performs reasoning directly in a latent visual space, mimicking human mental simulation such as imagining a chair rotating behind closed eyes. Key techniques include latent distillation, where a vision-language teacher guides a vision-only student network via image sequences and reward signals, and VLPO (Visual-Latent Policy Optimization), a reinforcement learning algorithm designed for continuous latent trajectories rather than discrete tokens. Grounded in the manifold hypothesis, Monet constrains reasoning to low-dimensional substructures of realistic images. Reported benchmarks show gains over GPT-4V on causal video reasoning (CLEVRER), stronger out-of-distribution generalization on A-OKVQA, and lower computational overhead for visual tasks. The post also discusses future applications in robotics, surgical simulation, and physical world models, while noting current limitations: training mostly on synthetic data, computation costs, bias, and interpretability challenges.

Imagine closing your eyes and clearly seeing an ordinary wooden chair slowly rotating in your mind—the curve of the backrest, the grain of the seat, the shadows of the four legs, circle after circle, like a private movie screening. This isn't magic; it's the most natural form of human "mental simulation." For artificial intelligence, however, this has long been an unreachable holy grail. Until Monet appeared.

This paper comes from a joint team at Peking University, Kuaishou, and MIT, with a concise but ambitious title: *Monet: Reasoning in Latent Visual Space*. It is no longer satisfied with "describing pictures"—it tries to give AI a true "imaginative eye," performing visual reasoning directly in an abstract high-dimensional mathematical space, just as we humans silently rehearse the physical world in our minds.

Let's dive into this silent revolution and see how Monet step by step teaches machines to "see the world with their eyes closed."

🪑 The Chair Thought Experiment: Where the Human–AI Gap Begins

Close your eyes and imagine a chair rotating.

You can do it almost instantly: the chair turns from front to side, the legs visually shorten, the curve of the backrest gradually appears, and you can even predict that when it turns to the back, it will fully occlude the seat. All of this happens without verbal description, done purely by intuition.

But give it to today's top multimodal models like GPT-4V, and what happens? It typically first translates the image into text: "This is a four-legged wooden chair, front-facing..." then reasons in a chain of language: "If rotated 90 degrees, the left legs will shorten..." and finally converts the text back into an image. The whole process is like walking with crutches—accurate, but clumsy, and prone to losing detail in translation.

Monet's ambition: throw away the crutches and rotate the chair directly in the "visual latent space." It doesn't talk; it simulates. The result is smoother reasoning that is closer to human intuition.

> What is Latent Space? > Simply put, it compresses an image into a high-dimensional vector (e.g., an array of thousands of numbers). In this space, similar images are close together, and different angles of the same chair form a smooth curve. Moving the vector is equivalent to rotating the object—no need to pass through the "intermediary checkpoint" of language.

🌌 Diving into Deep Space: Why Visual Reasoning Needs a "Hidden Garden"

The visual world is too noisy. A 512×512 color image has over 780,000 pixels—equivalent to 780,000 dimensions. If you place all possible images in this high-dimensional space, the vast majority of positions are meaningless noise—grotesque, physically impossible, nightmarish scenes.

Real-world images occupy only a tiny fraction, like a narrow road in the Sahara desert. The driver must strictly follow the road, or get lost instantly. This "road" is what mathematics calls a manifold.

The manifold hypothesis holds that natural images are not uniformly scattered in high-dimensional space, but concentrated on low-dimensional, curved substructures. Monet's insight: rather than laboriously navigating pixel space, first jump to the "hidden garden" of latent space, where manifolds are more evident, operations are more continuous, and reasoning is more reliable.

Imagine you are an astronaut searching for habitable planets in boundless space. Flying randomly in 3D is inefficient; but if you first enter a compressed coordinate system (latent space), planetary orbits become clear elliptical curves, and navigation becomes a hundred times easier. Monet is the "star map" given to AI.

🏜️ Navigating the Desert: How the Manifold Hypothesis Rescues AI's Imagination

Why is generating images so hard? Because high-dimensional space is huge—take one random step and you may stray off the "real image highway," generating horses covered in eyes or melting clocks (a nod to Dalí).

The manifold hypothesis offers hope: the degrees of freedom of real images are far below the pixel dimensionality. All possible views of a chair may correspond to a curved surface of only a few dozen dimensions. Monet restricts reasoning to this surface, like putting a "physical rein" on AI, ensuring every rotation step conforms to real-world laws.

The paper uses a vivid metaphor: randomly sampling high-dimensional space is like digging a well in the desert—99.9999% of locations are dry sand; but if you know the direction of the water veins (manifolds), the success rate skyrockets. Through pre-trained diffusion models or VAEs, Monet learns the shape of these invisible water veins, allowing it to safely "stroll" in latent space.

🎨 The Blind Painter's Training: Latent Distillation and Master–Apprentice Inheritance

How do you make a model learn to think in latent space? The team designed "Latent Distillation"—a wordless master–apprentice game.

The teacher is a powerful vision-language model (VLM) that can describe images. The student is a pure vision policy network that initially knows nothing. The teacher doesn't directly tell the student "what happens if the chair rotates 90 degrees," but instead shows large numbers of image sequences (e.g., video frames of a chair rotating step by step), letting the student imitate these trajectories in latent space.

The student keeps trying to generate intermediate latent vectors, and the teacher tells it "closer" or "off track" through reward signals (like scoring). Over time, the student learns to predict the next step without text, purely by visual intuition.

It's like teaching a blind painter: you don't say "the nose you're drawing is too crooked," but hold his hand and trace the contours with him over and over, until he can independently paint a portrait. Latent distillation is exactly this hands-on knowledge transfer—except it happens in an abstract mathematical space.

🧗 Scoring "Thoughts": How VLPO Cracks the RL Dilemma of Continuous Space

Traditional reinforcement learning (RLHF) succeeded in the language domain because language is discrete—each word is a clear action, and rewards are easy to assign.

But the visual latent space is continuous, without clear "steps." You can't say "adding 0.3 to the 5th dimension is the correct action." Standard policy gradients fail completely due to exploding variance or vanishing gradients.

The team therefore invented VLPO (Visual-Latent Policy Optimization)—a reinforcement learning algorithm designed for continuous vision. Instead of rewarding discrete tokens, it directly evaluates entire latent trajectories for physical plausibility and task completion.

Analogy: previous RL was like a language exam, with clear right and wrong on multiple choice; VLPO is like a gymnastics exam, scoring the whole routine's fluidity, difficulty, and landing stability. The judge (reward model) is a visually trained scorer based on human preferences, able to judge "is this rotation natural? Does gravity feel right?"

Experiments show VLPO lets Monet significantly outperform traditional methods on multi-step visual reasoning tasks, especially scenarios requiring continuous transformation (e.g., object collisions, fluid simulation).

🏆 More Than a Numbers Victory: When AI Handles the "Unseen World"

The paper tested Monet on multiple benchmarks:

CLEVRER (causal reasoning in video): Monet beats GPT-4V by 10–15%.
A-OKVQA (open-ended visual question answering): more robust on compositional questions.
Most striking is out-of-distribution (OOD) generalization: throw in object colors, materials, and combinations unseen in training, and Monet stays robust, while text-dependent models collapse due to "lexical bias."

Why? Because text chains tend to anchor reasoning in the linguistic patterns of training data, while latent-space reasoning is closer to physical essence. Just like humans can imagine a purple metal chair rotating even on first sight—we rely on intuition for "chairness," not memory of specific examples.

Here is a partial comparison table from the paper (organized for intuitive understanding):

| Task Dimension | Monet (Latent Reasoning) | GPT-4V (Text CoT) | Traditional VLM (e.g., CLIP) | |------------------------|----------------------|--------------------|-------------------| | CLEVRER accuracy | 85–95% | 70–80% | 60–70% | | OOD generalization | High | Medium | Low | | Compute cost (visual tasks) | Lower | Higher (text conversion) | Medium | | Continuity handling | Native | Discrete simulation | Almost none | | Robotics potential | Strong | Medium | Weak |

🤖 A Future Vision: When Machines Have True Mental Models

If methods like Monet are scaled up, what happens?

Imagine a rescue robot in earthquake rubble: it doesn't need to upload every frame to the cloud, translate it into text, then plan a path. It can quickly simulate locally in latent space: "If I push aside this concrete slab, will the person below be harmed a second time?" This kind of internal causal simulation is exactly how human experts think.

In medicine, doctors might let AI "rehearse" surgery in latent space: how vessels bend, how tissue deforms—all completed in silent mathematical flow.

Further in the future, when latent-space models combine with 3D diffusion and video generation, AI might build a complete "inner world"—an interactive physical simulator. At that point, robots would no longer be tools executing instructions, but partners with understanding.

❓ Where Is the Tipping Point: How Far Are We from "AI That Can Imagine"?

Monet is still just a beginning. The current version is trained mainly on synthetic data; the real world has more noise and more complex manifolds. Compute cost, data bias, and interpretability are all hurdles to cross.

But the most exciting thing is the direction: from "translating vision into language" to "directly computing vision." This is like a key leap in human cognitive evolution—from needing to speak aloud to think, to silent, intuitive insight.

When AI can also close its eyes, see the chair rotating, and even see more complex physics and the vicissitudes of life, we may truly welcome a new era of intelligence.

And that day may be closer than we think.

---

References

1. Monet: Reasoning in Latent Visual Space arXiv preprint: https://arxiv.org/abs/2410.23335 (Core paper detailing model architecture, the VLPO algorithm, and experimental results)

2. Project Monet GitHub Repository https://github.com/project-monet/monet (Open-source code, pre-trained models, and reproduction guide)

3. MIT CSAIL News: New AI model can reason about the visual world https://www.csail.mit.edu/news/new-ai-model-can-reason-about-visual-world (Official MIT interpretation, emphasizing links to cognitive science)

4. Peking University AI Lab Paper Announcement https://ai.pku.edu.cn/research/papers/monet (The PKU team's perspective, highlighting the latent distillation design)

5. Kuaishou Research Technical Report https://research.kuaishou.com/papers/monet-latent-reasoning (Kuaishou team report, focusing on industrial deployment potential and VLPO optimization details)