ReWorld: An Interactive World Model with Long-Horizon Memory
> *"Memory is the currency of the soul."* — Jorge Luis Borges
Introduction: A Philosophical Experiment About Forgetting
Imagine walking into a vast library where you can only read books on your current floor. When you move to a new floor, the previous ones lock forever. This is how traditional AI models handle long sequences: a fixed context window where new information pushes out the old.
Now imagine a second library — equally vast, but this time you carry a magic notebook. As you pass each floor, you sketch its outline — not every book, just the shape. When you need to recall, you flip through the sketches and decide whether to return for a closer look.
This is the world ReWorld sets out to build.
The Three-Way Dilemma of World Models
A world model gives an AI an internal simulator of an environment: it should understand spatial structure, predict the consequences of actions, and generate plausible sensory input — not from hand-programmed rules, but from learned "intuitive physics" in data.
ReWorld's authors argue a truly useful interactive world model must satisfy three structurally conflicting requirements:
1. Real-time control — responses within ~100ms to feel responsive, favoring short horizons. 2. Long-horizon memory — remembering environments visited minutes ago, favoring unbounded horizons. 3. High-quality generation — coherent, artifact-free video, requiring sufficient context.
Conventional solutions compromise: sliding windows cause amnesia; full attention causes memory blow-up and slowness. ReWorld's answer: have both.
The Four (Five) Key Inventions
1. Mixed Per-Head Attention Windows
Like an orchestra where most musicians play the current movement while a few hold the full score, ReWorld splits Transformer attention heads into two groups:
- Local heads (the majority): attend only to the recent past (e.g., the last 16 frames), enabling fast real-time control.
- Global heads (a small fraction): attend to the entire history, responsible for long-range consistency.
- Extracts landmarks: identifies frames worth remembering and compresses them into compact landmark embeddings.
- Indexes by pose: tags each landmark with the camera's 3D position and orientation — like GPS coordinates.
- Retrieves by nearest neighbor: when the model needs to recall, it queries the current camera pose and fetches the closest landmarks to "refresh" memory.
- Action following: best-in-class rotation error of 11.95 degrees, with the most coherent camera motion.
- Long-horizon recall: the extreme test — roam a virtual environment for 64 seconds (384 latent frames), then return to the start. Sliding windows long forgot the start; full attention would exhaust memory. ReWorld's fixed 12-chunk cache successfully recalled and regenerated the starting location's appearance.
- Generation quality: best-or-equal versus six recent interactive world models, supporting photorealistic, game-style, and stylized outputs — all from one model via LoRA switching.
- Chen, Z., Wang, L., Shen, G., et al. *ReWorld: An Interactive World Model with Long-Horizon Memory*. arXiv preprint.
- Ha, D., & Schmidhuber, J. (2018). *World Models*. arXiv:1803.10122.
- Vaswani, A., et al. (2017). *Attention Is All You Need*. NeurIPS 2017.
Random head routing randomly reassigns which heads are local vs. global at each training step, preventing heads from lazily specializing and forcing every head to master both capabilities.
2. Pose-Indexed Landmark Bank
Like an explorer sketching landmarks at interesting junctions, ReWorld:
Memory stays under a fixed budget: regardless of video length, only a fixed number of landmarks (12 chunks in the paper) are kept.
3. Random Chunk Dropping
Training data is typically dense and continuous, but at inference with the landmark bank the model sees a sparse history with gaps. ReWorld randomly drops video chunks during training, forcing the model to reconstruct scenes from sparse evidence — like training a detective from screenshots rather than full surveillance footage.
4. Metric-Scale-Aligned Data Engine
Different data sources have different physical scales: pressing "forward" may move the camera 10 meters in Unreal Engine renders but 1 meter in game walkthrough footage. ReWorld's data engine calibrates eight data sources to a single physical scale, so the same action means the same motion everywhere.
It also generates palindrome trajectories — travel out, then retrace your path — forcing the model to remember the starting point because it must regenerate it on return.
5. Distribution-Matching Distillation via LoRA
High-quality video generation typically needs many diffusion steps (~50), but real-time interaction allows only 4. ReWorld uses distribution-matching distillation confined to a LoRA adapter: the base model stays unchanged, and a lightweight adapter enables a 4-step real-time mode, while multi-step high-quality mode remains available on the same backbone.
Experimental Results
ReWorld is evaluated on a three-axis protocol:
Epilogue: The Nature of Memory
ReWorld evokes Borges' story *Funes the Memorious*, whose protagonist's perfect memory prevents him from thinking — because thinking requires forgetting and abstraction. ReWorld's landmark bank instead mimics human selective memory: we don't remember every step, but we remember "turn left at that corner and there's a coffee shop."
AI doesn't need perfect memory. It needs *good-enough* memory — compressed, retrievable, sufficient for the task. ReWorld shows one path: pose-indexed landmarks, mixed attention, and random-dropout training can deliver seemingly unbounded long-horizon memory under fixed compute. Perhaps true intelligence lies not in remembering everything, but in knowing what is worth remembering and how to retrieve it.