GIM-World: Building a 3D Physical Space in AI's Mind — A Geometric Memory Revolution for Long Video Generation
> Paper: Geometry-Aware Implicit Memory for Video World Models > Institutions: School of Intelligence Science and Technology, Nanjing University; Kuaishou Kling team; Tsinghua University > arXiv: 2606.02436 > Project page: https://gim-world.github.io/
The Nightmare of Long Video Generation: Walk Far, Forget Everything
Imagine playing an open-world game. You walk down a corridor, open a door, pass through several rooms, then walk back. The corridor should still be the same corridor; the door should be in place; the furnishings unchanged.
For AI video generation models, this is extremely hard. When generating long videos, models face three nightmares:
1. Geometric inconsistency — walls bend, floors tilt, objects drift. Leave a room and return: the layout has changed. 2. Memory drift — each frame of autoregressive generation introduces tiny errors that accumulate until the frame collapses after hundreds of steps. 3. Structural collapse — scene landmarks (doors, windows, furniture) gradually vanish or deform into abstraction.
Traditional solutions fall into two camps:
Explicit memory: store historical frames in a database and retrieve them during generation. Problems — retrieval by visual similarity often picks wrong reference frames; redundant appearance information fills memory, limiting history length; reconstruction errors accumulate.
Implicit memory: compress history into a compact state (like an RNN hidden state). Problems — no geometric constraint, so what gets compressed is appearance, not 3D structure; cross-view scene geometry is not explicitly encoded; long-range dependencies still fade.
GIM-World's core insight: Geometry should be an attribute of memory, not just an input to the generator.
The Three-Layer Design
1. Implicit Memory Encoder: Compressing History into Fixed-Size Tokens
A lightweight Transformer encoder compresses variable-length history into a fixed-size set of Memory Queries (e.g., 64 learnable queries). These queries read the latent tokens of history frames via attention; the updated queries form the memory state m_t, whose size never grows with history.
The encoder contains two Compact Self-Attention + FFN layers: the Compact stage compresses spatial token blocks to reduce attention cost, and the Expand stage restores dimensions to preserve expressiveness. Total runtime is under 0.3% of the diffusion backbone — meaning you can remember thousands of frames at negligible memory cost.
2. Camera-Queryable Geometric Supervision: Distilling 3D Structure into Memory
If implicit memory only compresses appearance, it learns surface textures, not spatial structure. GIM-World's fix: make the memory answer geometric queries — given a camera pose, predict the 3D features of that view.
- A frozen 3D foundation model (e.g., VGGT) serves as a "geometry teacher."
- During training, a historical camera pose is sampled; its ray map queries the memory state.
- A lightweight Geometry Head predicts features for that view, compared against the teacher's features via a loss.
- Explicit methods (FramePack, CaM) retain frames/tokens but lack a cross-view compact world state, yielding low reprojection scores.
- Pure implicit methods (SSM) are compact but lack geometric supervision, scoring poorly on both metrics.
- GIM-World combines the compactness of implicit memory with the structural power of explicit geometry.
- Wei, Z., et al. (2026). "Geometry-Aware Implicit Memory for Video World Models." arXiv:2606.02436
- Project page: https://gim-world.github.io/
- Related work: VMem (2025), SSM (2025), VideoSSM (2025), FramePack (2025), CaM (2025)
Key points: the 3D teacher is used only during training and completely discarded at inference; geometric supervision acts on the memory state, not the generator; the memory is forced to encode cross-view scene structure.
3. Information-Guided Pruning: Choosing What to Keep
Even with fixed-size memory tokens, reading history grows with time. Before encoding, GIM-World prunes redundant observations using Krause et al.'s mutual-information sensor-placement criterion, greedily selecting the subset of frames that best predicts the discarded ones. Only the most informative frames are kept.
Experimental Results: Consistency Over a Thousand Frames
Dataset: MIND
MIND (Multi-view Interactive Video Dataset) contains first-person and third-person long-video scenes testing memory consistency and action controllability.
| Method | Type | Memory Consistency | 3D Geometry (1st-person) | 3D Geometry (3rd-person) | |--------|------|--------------------|--------------------------|--------------------------| | FramePack | Explicit | Medium | Fair | Fair | | CaM | Explicit | Medium | Fair | Fair | | SSM | Implicit | Poor | Poor | Poor | | GIM-World | Implicit + Geometry | Best | 81.70 | 87.10 |
(3D geometric consistency measured by Normalized Reprojection Score; higher is better.)
Key findings:
Qualitative Results: 400-Frame Comparison
Baselines begin drifting within 100 frames — walls and landmarks deform and vanish. In third-person scenes, baselines lose the controllable character or place it inconsistently. GIM-World shows only minor per-frame viewpoint offsets while preserving scene geometry, layout, landmarks, and character poses.
Stress Test: 1000-Frame Loop-Back Consistency
In a single first-person scene spanning 1000 frames, two similar camera poses at frames 100 and 750 — separated by over 600 frames — show small objects (highlighted in green and red boxes) retaining the same spatial positions, relative arrangement, and texture details on revisit. The implicit memory stores a stable 3D scene representation, not just short-term context propagation.
Why the Design Works
Geometry-as-Memory vs. Geometry-as-Input
Existing methods inject geometry into the generator: direct conditioning (concatenating geometric tokens or point-map projections), feature supervision (aligning backbone features with 3D features), or post-training alignment (DPO/RL from 3D-derived rewards). GIM-World inverts this: geometry is an attribute of memory itself. The generator reads geometry-consistent conditions from memory, separating responsibilities, giving memory explicit geometric meaning, and adding zero inference overhead.
The Power of Camera Queryability
Unlike NeRF or 3DGS, GIM-World requires no explicit reconstruction, point clouds, or meshes. Given any camera pose, view-dependent features are decoded directly from memory — turning "rendering" into "querying."
Information-Guided Pruning Intuition
Adjacent frames are highly redundant; static regions persist for hundreds of frames. Pruning keeps frames that provide the most new information and discards frames predictable from existing ones, maximizing information within finite-capacity memory tokens.
Industry Implications
1. Foundation for interactive virtual worlds. Long video generation is a stepping stone to embodied AI training environments, open-world game engines, and persistent VR/AR spaces. GIM-World demonstrates thousand-frame-scale spatially stable generation — a leap from "video generation" toward "world simulation." 2. For Kuaishou Kling. The Kling team are core authors, suggesting next-generation Kling models may integrate this technology, moving from "appearance realism" toward "physical consistency." 3. Comparison with related work:
| Work | Approach | Strengths | Weaknesses | |------|----------|-----------|------------| | VMem (Oxford) | Explicit geometric memory (Surfel) | Interpretable, pluggable | Requires reconstruction, complex engineering | | SSM | Pure implicit state | Compact, fast | No geometric constraint | | GIM-World | Implicit + geometric supervision | Compact + geometrically consistent | Relies on 3D teacher during training |
GIM-World takes the practical middle path: implicit-memory efficiency with explicit-geometry structure.
Limitations and Future Directions
1. Training depends on a 3D foundation model — high-quality teacher, camera-pose annotations, and camera estimation for pose-free video data are required. 2. Dynamic objects — the paper focuses on static-scene geometry; pruning may discard key frames containing dynamic changes. This is future work. 3. From video to world model — GIM-World is a video generation framework, not a full world model (lacking physical interaction and causal reasoning). Next steps: action-physics feedback loops, multi-agent interaction, and RL integration for embodied AI.
Conclusion: A Qualitative Leap in Memory
GIM-World's core contribution is a paradigm shift: from remembering appearance to remembering the 3D world. Walk far away and return — the scene is consistent. View the same object from another angle — the geometry is correct. Thousands of autoregressive frames — no collapse.
With inference overhead under 0.3% of the diffusion backbone, the technique is fully practical: no extra geometric plugins, no expensive reconstruction pipelines — just a "geometry teacher" during training. For long video generation, interactive virtual worlds, and embodied AI simulation, GIM-World offers a practical, scalable, geometrically consistent memory framework.
> "Building a 3D physical space in AI's mind" — this vision is becoming reality.
References