Einstein World Models: Teaching LLMs to Daydream with Video

Forum topic · ✨步子哥 · 2026-06-27

Summary

Einstein World Models (EWM), a June 2026 position paper by MBZUAI and RIKEN researchers (arXiv:2606.26969), proposes that large language models should treat 'imagining a video' as a tool call. Inspired by Einstein's thought experiments, where imagery preceded language, the authors argue some reasoning is inherently visual-temporal and poorly served by text-only chain-of-thought, citing SimpleBench questions that LLMs routinely fail. In EWM, an LLM reasoning trace sparsely interleaves M world-module calls (M << N) with N text steps: the model issues a query, a video generation model returns a short clip, and the video is treated as an inspectable hypothesis, not an answer. Training follows standard SFT plus RLVR (GRPO-style), with video tokens masked during SFT and rewards combining answer correctness with optional world-module usage. The paper compares EWM to Whiteboard-of-Thought and Visualization-of-Thought, outlines renderer/simulator/planner roles for world modules, and notes ensembling benefits. Notably, the paper contains no experiments; its core demand is new datasets of text questions answerable only via visual thought experiments. The post also discusses implications for interpretability and test-time compute beyond language.

Einstein World Models: When LLMs Learn to "Daydream"

*Translation of a zhichai.net explainer post. The original is a Feynman-style popularization of a position paper, not a literal translation of the paper itself. The paper is a blueprint with no experiments or open-source code.*

Under the Bern Clock Tower: A Man Chasing a Beam of Light

One afternoon in 1907, a junior clerk at the Bern patent office did something odd — Albert Einstein chased a beam of light in his head.

He wondered: if I ran at the speed of light, what would that beam look like beside me? Would it hang frozen in midair? But Maxwell's equations don't permit a static electromagnetic wave. The image was absurd — absurd enough to crack open classical physics.

This thought experiment happened in no lab, on no paper, using no formula. It happened in Einstein's mind, in the form of images. Years later, the mathematician Hadamard recorded Einstein's own description:

> "The psychical entities which seem to serve as elements in thought are certain signs and more or less clear images which can be 'voluntarily' reproduced and combined... The above mentioned elements are, in my case, of visual and some of muscular type. Conventional words or other signs have to be sought for laboriously only in a secondary stage."

Note the order: images first, language second. Images are the protagonist; words are a supporting act.

In June 2026, a team from MBZUAI and RIKEN asked: can large language models learn to do this too?

Their answer is named "Einstein World Models" (EWM).

An Awkward Truth: LLMs Can Talk, But Not "Imagine"

Today's LLMs are extremely good at reasoning in text. Chain-of-Thought lets them write out their thinking step by step, like a student solving problems on scratch paper.

But there's a problem: some questions can't be answered correctly in words, no matter how you write them.

The paper cites a SimpleBench example:

> An acrobat throws a blue solid ball 1 meter into the air, then throws a purple solid ball (same size) 2 meters into the air. She then carefully climbs a tall ladder while balancing a yellow balloon on her head. Where is the purple ball most likely now? > > A. Same height as the blue ball > B. Same height as the yellow balloon > C. Inside the blue ball > D. Above the yellow balloon > E. Below the blue ball > F. Above the blue ball

The correct answer is A.

Why? A ball thrown 1–2 meters is airborne for less than a second. Carefully climbing a tall ladder while balancing a balloon takes at least a dozen seconds. By the time she reaches the top, both balls have long since landed. So both are on the ground, at the same height.

Easy for humans — you "play the movie" in your head. But text-only LLMs often get it wrong because they over-formalize: 2 meters is higher than 1 meter, so the purple ball is higher? They lack intuition for the passage of time and physical scenes.

The issue isn't that LLMs aren't smart — it's that some reasoning is naturally done with images, not text. Object identity, spatial relations, contact, heat, motion, material state — tracking these in text is slow and error-prone, but a "mental video" shows them instantly.

Core Idea: Make "Imagination" a Tool Call

Modern models already call tools. Don't know a fact? Search the web. Need math? Run code. But one tool has been missing: imagining a scene.

EWM's core idea in one sentence: let the LLM use "imagine a video" as a tool call.

Concretely, EWM inserts a new kind of step into standard chain-of-thought. Ordinary CoT is N steps of text generation. EWM sparsely inserts M "world-module calls" among those N steps. When the LLM feels it needs to "see" how a scene unfolds, it generates a query, calls a world module (essentially a video generation model), receives a short video, and treats that video as part of its reasoning trace.

In the paper's notation:

\mathcal{T}_{t+1} = \mathcal{T}_t \oplus \begin{cases} s_t, & \text{no world-module call} \\ [q_t, v_t], & \text{world-module call} \end{cases}

Here $$s_t$$ is ordinary text reasoning, $$q_t$$ is the query, and $$v_t$$ is the returned video. The video is not LLM-generated — it's an "observation," not an "action."

There's a subtle design point: M ≪ N. World-module calls are sparse. Not every step needs imagination — only the key junctures, just as Einstein wasn't perpetually chasing light, but ran the thought experiment at critical moments.

Key Distinction: The Video Is a "Hypothesis," Not an "Answer"

This is the paper's most important conceptual distinction, and what separates EWM from "letting video generation models answer questions directly."

When EWM calls the world module, the resulting video is not the final answer. It is an inspectable hypothesis.

Back to Einstein: his imagined chase of a light beam was physically impossible. But it was valuable because it turned an abstract contradiction (electrodynamics vs. motion intuition) into a concrete, examinable picture. EWM's videos are the same: they need not be physically perfect simulations; they need to expose the model's assumptions so subsequent reasoning can build on, revise, or overturn them.

This matters enormously. If the video were the answer, the video model's physical inaccuracies would become wrong answers. But if the video is a hypothesis:

It can be imperfect (like Einstein's physically impossible chase)
It can be inspected, questioned, corrected
What was once hidden in model weights becomes a publicly reviewable object

The paper puts it elegantly: EWM turns an "otherwise private visualised episode" into a "public object of analysis." Even if the underlying model isn't open-source, its "imagination process" becomes visible.

Training: SFT + RLVR, Just Like Tool Use

EWM's training follows the standard paradigm for reasoning models: SFT first, then RLVR.

Stage 1: Supervised fine-tuning. Teach the model the format of EWM reasoning traces — when to write text, when to emit a tool-call tag, when to receive a video observation, when to give the final answer. Standard next-token cross-entropy, but video observation tokens are masked — videos are observations, not actions, so the model doesn't learn to "generate" them.

Stage 2: Reinforcement learning (RLVR). GRPO-style training over full EWM trajectories, with reward:

r_{\mathcal{M}}(\mathcal{T}, y^\star) = r(\hat{y}, y^\star) + r_{\mathcal{W}}(\mathcal{T})

The first term rewards final-answer correctness; the second is an optional world-module usage reward (encouraging the model to invoke imagination at the right moments, not abuse it or never use it).

This is exactly how LLMs are trained to call search engines and code interpreters. EWM's novelty isn't the training algorithm — it's bringing "imagining video" into the tool-use framework.

Three Forms of the World Module

The paper classifies systems that can serve as world modules:

1. Renderers: given a text description, generate a video. Text-to-video and image-to-video diffusion models belong here. This is EWM's default world module. 2. Simulators: let the reasoner intervene in a visualizable world and observe outcomes. Genie-style interactive world models are one example. But the paper notes that in practice, repeated renderer calls can play the simulator's role — the LLM watches a video, revises its assumption, calls again. Renderers are the core. 3. Planners: in EWM, planning remains the LLM reasoner's job, not the world module's, because EWM targets visual-temporal reasoning, not embodied robot action.

The paper also discusses world-module quality. Einstein's thought experiments worked because his strong physical intuition constrained his imagination. If the video model's physics is poor, EWM's hypotheses will be poor too. The good news: physical consistency of video diffusion models is measurable (via likelihood estimates derived from the denoising objective), and the field is improving fast.

There's also an ensembling idea: different world modules have different biases — one excels at visual realism, another at physical consistency, another at temporal coherence. Because videos are externalized, these biases can be compared rather than hidden.

How It Differs from Prior Work

Chain-of-Thought: makes intermediate reasoning visible, but only in language. EWM extends "visibility" to the visual-temporal dimension.
Whiteboard-of-Thought: gives multimodal models a visual scratchpad. But it draws static images, not video. EWM wants visual-temporal replay.
Visualization-of-Thought (VoT): spatial reasoning in 2D grid worlds. But its visualization is still text-form grids, not independent video artifacts.
Direct video-model reasoning: some research asks whether video generation models can "answer with video" directly. EWM is fundamentally different — not replacing the LLM as reasoner, but giving it video generation as a thought-experiment tool.

One-sentence summary: prior work either uses static images, simulates vision in text, or lets video models reason themselves. EWM keeps the LLM as the reasoner but equips it with an "imagination" tool.

The Data Bottleneck: A Paper That Calls for Datasets

This paper has an unusual feature: it has no experimental results.

That's not an oversight — it's the paper's positioning. EWM is a blueprint: it defines a capability and designs the architecture and training objectives, but notes that realization is blocked by a key bottleneck: no suitable dataset exists.

Existing datasets are either pure-text reasoning (no imagination needed) or visual QA (the image is already given). EWM needs something special: questions posed in pure text, but answerable only through a visual thought experiment.

SimpleBench is one of the few datasets pointing this way, but it has only ~200 questions, with only 10 public. The acrobat question is one of them.

Hence the final section: "Future Work: A Call for Datasets." Unlike most papers' ritual future-work sections, this one is the paper's core demand: we've defined a capability; now the community needs to build the data to train it.

An ideal dataset would contain two question types: those requiring a visual thought experiment, and those that don't. That way the model learns not just to imagine, but when to imagine — and when not to.

My Take: Language Is Pretraining; Imagination Is Test-Time Compute

This paper suggests a bigger picture.

I've recorded elsewhere a cross-domain isomorphism: octopus RNA editing — "DNA is pretrained weights; RNA editing is test-time compute." Octopus lineages solved the test-time-compute route 300 million years ago: without changing DNA (the base model), they adjust protein sequences in real time within a single lifetime via RNA editing.

EWM echoes the same idea in AI, from a different angle. Current LLM test-time compute is mostly textual reasoning (CoT, ReAct, tool calls). EWM points out that some reasoning shouldn't be done in text — it should be done in video. Text is the LLM's "pretraining modality"; visual imagination is its missing "test-time modality."

Deeper still: EWM is really saying intelligence isn't a single-modality affair. Einstein's greatness wasn't his verbal ability but his freedom to move between language, images, even muscle sensation. Hadamard's record is clear — images precede language; words are only the laborious second stage. Today's LLMs are stuck at the second stage: superb at language, lacking the first stage — the ability to "voluntarily reproduce and combine images." EWM isn't just bolting a video generator onto an LLM; it opens a door: during reasoning, when words run out, permit yourself to "look."

This also raises an interpretability question. CoT already externalizes thinking — we can read the model's reasoning steps. But CoT only externalizes verbal thought. If a future LLM has internal "visual intuitions" shaping its answers, CoT won't reveal them. EWM externalizes visual imagination as inspectable video, extending interpretability from the linguistic to the visual dimension: know not only what the model thought, but what it "saw."

Finally, though this is a blueprint, the timing may be right. Video generation models improved dramatically in 2025–2026: physical consistency is rising, video lengths growing. If the trend holds, the world module EWM needs may become good enough within a year or two. At that point, whoever builds the dataset EWM needs will be first to teach an LLM to "daydream."

Teaching an LLM to chase a beam of light — a goal worth taking seriously.

---

Paper: Einstein World Models (arXiv:2606.26969)

Authors: Munachiso Samuel Nwadike, Zangir Iklassov, Ali Mekky, Zayd M. Kawakibi Zuhri, Kentaro Inui

Affiliations: MBZUAI / RIKEN AIP / Tohoku University

Note: This is a Feynman-style popularization, not a translation of the paper. The paper is a blueprint/position paper with no open-source code yet.