ReWorld: An Interactive World Model with Long-Horizon Memory
> *"Memory is the currency of the soul."* — Jorge Luis Borges
Introduction: A Philosophical Experiment About Forgetting
Imagine walking into a giant library where you can only read the books on your current floor. When you move up, previous floors lock forever. This is how traditional AI models handle long sequences—they have a fixed "context window," and new information pushes old information out.
Now imagine a second library, where instead you carry a magic notebook. On each floor, you sketch its outline—not every book, just enough to find your way back. This is the world ReWorld sets out to build.
Chapter 1: The Three-Way Dilemma of World Models
What Is a "World Model"?
A world model gives AI a virtual environment it can: 1. Understand structurally (there's a table here, a chair there) 2. Predict action consequences (if I walk forward, I hit the table) 3. Generate plausible sensory inputs (what the table looks like from this angle)
Unlike game engines with hand-programmed rules, world models learn "intuitive physics" from data.
The Tension Between Real-Time Control and Memory
A truly useful interactive world model must satisfy three seemingly contradictory requirements:
1. 🎮 Real-time control — inference must complete within ~100 ms to avoid perceived lag 2. 🧠 Long-horizon memory — the model must remember the starting room even after ten minutes of roaming 3. 🎬 High-quality generation — outputs must be realistic, coherent, and artifact-free
These are structurally in conflict:
- Real-time control favors short horizons
- Long-horizon memory favors unbounded history
- High-quality generation needs sufficient context for consistency
- Sliding window: keep only the last N frames → amnesia
- Full attention: keep everything → too slow, memory explosion
- Local heads (the majority): attention limited to the recent past (e.g., last 16 frames) — fast, suited for real-time control
- Global heads (a small minority): attend to the entire history — responsible for long-range memory and consistency
- Landmark extraction: salient video frames are compressed into compact landmark embeddings
- Pose indexing: each landmark is tagged with the camera's 3D position and orientation — like GPS coordinates
- Nearest-neighbor retrieval: given the current camera pose, the model retrieves the closest landmarks to "refresh" its memory
- Fixed-budget KV cache: memory use stays constant regardless of video length (12 chunks in the paper) — a notebook with fixed pages holding only the most important sketches
- The base model stays unchanged
- One backbone supports both:
- High-quality mode: multi-step generation
- Real-time mode: 4-step generation
- Rotation error: 11.95 degrees (state of the art)
- Camera motion consistency: best in class
- Chen, Z., Wang, L., Shen, G., et al. (2026). *ReWorld: An Interactive World Model with Long-Horizon Memory*. arXiv preprint.
- Ha, D., & Schmidhuber, J. (2018). *World Models*. arXiv:1803.10122.
- Vaswani, A., et al. (2017). *Attention Is All You Need*. NeurIPS 2017.
Traditional compromises fail either way:
ReWorld's answer: have both.
Chapter 2: ReWorld's Key Innovations
1. Mixed Per-Head Attention Windows (the "frequency splitter")
Like an orchestra where most musicians play the current movement while a few hold the full score:
Random head routing: at each training step, which heads are local vs. global is randomized, preventing any head from "coasting" and forcing all heads to develop both capabilities.
2. Pose-Indexed Landmark Bank
The most elegant mechanism. Like an explorer sketching landmarks at interesting spots:
3. Random Chunk Dropout — Making Sparsity the Norm
During training, models see continuous clips; at inference with the landmark bank, history is sparse with large gaps. ReWorld randomly drops video chunks during training, forcing the model to reconstruct scenes from sparse evidence and treat incomplete history as normal rather than anomalous. Like training a detective given only screenshots from key moments.
4. Metric-Scale-Aligned Data Engine
Different data sources have different physical scales (Unreal Engine flythroughs vs. game walkthroughs vs. real-world footage). Mixing them raw makes the model "schizophrenic"—the same keypress meaning different motions. ReWorld's data engine calibrates 8 data sources to a single physical scale:
> One "forward" keypress moves the camera the same distance in every data source.
Palindrome trajectories: the data includes out-and-back paths, forcing the model to memorize the starting point because it must eventually regenerate it.
5. Distribution-Matching Distillation via LoRA Adapter
High-quality video diffusion typically needs ~50 steps; real-time interaction allows only 4. ReWorld uses distribution-matching distillation to compress the multi-step model into a 4-step one, confined to a LoRA adapter:
Like one camera that shoots both high-resolution photos and smooth video by switching modes.
Chapter 3: Experimental Results — Does It Actually Remember?
ReWorld was evaluated on a three-axis protocol:
Axis 1: Action Following
Axis 2: Long-Horizon Recall
The headline test: roam a virtual environment for 64 seconds (384 latent frames), then return to the starting point. Sliding windows have long discarded the start; full KV attention would explode memory. With its fixed 12-chunk cache, ReWorld successfully recalls and regenerates the starting scene — proof the landmark bank works.Axis 3: Generation Quality
Compared against 6 recent interactive world models, ReWorld achieves best-level quality, supporting three styles — photorealistic, game-style, and stylized — all via the same backbone with switchable LoRA adapters.Epilogue: The Nature of Memory
ReWorld evokes Borges' story *Funes the Memorious*, whose protagonist's perfect memory prevents him from thinking—because thinking requires forgetting and abstraction.
ReWorld's landmark bank mirrors human selective memory: we don't remember every step of a walk, but we remember "turn left at that corner and there's a coffee shop." Landmarks are enough to find our way next time.
AI doesn't need perfect memory. It needs good-enough memory—compressed, retrievable, sufficient for the task. ReWorld shows one path: pose-indexed landmarks, mixed attention, and dropout training deliver seemingly unbounded long-horizon memory under a fixed compute budget.
Perhaps true intelligence lies not in remembering everything, but in knowing what is worth remembering and how to retrieve it when needed.