GIM-World: Building a 3D Physical Space in an AI's Mind — A Geometric Memory Revolution for Long Video Generation

> Paper: *Geometry-Aware Implicit Memory for Video World Models* > Authors: Zhengxuan Wei, Xu Guo, Xinghui Li, Xunzhi Xiang, Min Wei, Yiran Zhu, Qiulin Wang, Xintao Wang, Pengfei Wan, Xiangwang Hou, Qi Fan > Institutions: School of Intelligence Science and Technology, Nanjing University; Kuaishou Kling team; Tsinghua University > arXiv: 2606.02436 > Project page: https://gim-world.github.io/

---

1. The Nightmare of Long Video Generation: Walk Far Enough and the Model Forgets

Imagine playing an open-world game. You walk down a corridor, open a door, pass through several rooms, and then return the way you came.

The corridor should still be the same corridor. The door should still be where it was. The furnishings should be unchanged.

For AI video generation models, this is extremely hard. When generating long videos, models face three nightmares:

1. Geometric inconsistency: walls bend, floors tilt, object positions drift. Leave a room and come back — the layout has changed. 2. Memory drift: each frame of autoregressive generation introduces small errors that accumulate; after hundreds of frames the picture collapses entirely. 3. Structural collapse: landmarks in the scene (doors, windows, furniture) gradually vanish or deform, and the video becomes an abstraction.

Traditional solutions fall into two camps:

Explicit memory: store past frames in a database and retrieve them during generation. Problems:

• Visual-similarity retrieval often picks the wrong reference frames
• Redundant appearance information fills memory, limiting stored history
• Reconstruction errors accumulate

Implicit memory: compress history into a compact state (like an RNN hidden state). Problems:

• No geometric constraint; what is compressed is appearance, not 3D structure
• Cross-view scene geometry is not explicitly encoded
• Long-range dependencies are still forgotten

GIM-World's core insight: geometry should be a property of memory, not merely an input to the generator.

---

2. GIM-World's Three-Layer Design

2.1 Implicit Memory Encoder: Compressing History into a Fixed-Size Token Set

Problem: the number of history frames keeps growing; memory cannot expand indefinitely.

Solution: a lightweight Transformer encoder compresses variable-length history into a fixed-size set of memory tokens.

How it works:

• A set of learnable memory queries (fixed count, e.g., 64)
• These queries read the latent tokens of history frames via attention
• The updated queries form the memory state m_t
• Memory size never grows with history length

Key design: the encoder contains Compact Self-Attention + FFN layers:

• Compact: compresses spatial token blocks to reduce attention cost
• Expand: restores dimensions to preserve expressiveness
• Total runtime is under 0.3% of the diffusion backbone

> Implication: you can remember thousands of frames of history at nearly negligible memory cost.

2.2 Camera-Queryable Geometric Supervision: "Distilling" 3D Structure into Memory

Problem: if implicit memory only compresses appearance, it learns surface textures, not spatial structure.

Solution: make memory answer geometric queries — given a camera pose, predict the 3D features of that view.

How it works:

• A frozen 3D foundation model (e.g., VGGT) acts as a "geometry teacher"
• During training, a historical camera pose is sampled; it produces a ray map that queries the memory state
• A lightweight geometry head predicts features for that view
• These are compared against features extracted by the 3D teacher to compute a loss

Key insight:

• The 3D teacher is used only in training and fully discarded at inference
• Geometric supervision applies directly to the memory state, not the generator
• Memory is forced to encode cross-view scene structure

> Analogy: a geometry tutor trains the memory to understand space during training. Once learning is complete, the tutor leaves and the memory recalls spatial relations on its own.

2.3 Information-Guided Pruning: Choosing What to Keep Intelligently

Problem: the longer the history, the higher the encoding cost — even with fixed-size memory tokens, reading history scales with time.

Solution: prune redundant past observations based on information theory before encoding.

How it works:

• Uses Krause et al.'s mutual-information sensor placement criterion
• Selects the subset of observations that best predicts the discarded frames
• Optimized with a greedy algorithm

Effect:

• Keeps only the most informative history frames
• Discards redundant, similar-view frames
• Keeps encoding cost bounded

---

3. Experimental Results: Consistency Over a Thousand Frames

Dataset: MIND

MIND (Multi-view Interactive Video Dataset) contains long-video scenes with first-person and third-person views, testing memory consistency and action control of video world models.

Quantitative Comparison

| Method | Type | Memory consistency | 3D geometry consistency (first-person) | 3D geometry consistency (third-person) | |--------|------|--------------------|----------------------------------------|----------------------------------------| | FramePack | Explicit | Medium | Average | Average | | CaM | Explicit | Medium | Average | Average | | SSM | Implicit | Poor | Poor | Poor | | GIM-World | Implicit + geometry | Best | 81.70 | 87.10 |

(3D geometric consistency measured by Normalized Reprojection Score; higher is better.)

Key findings:

• Explicit memory methods (FramePack, CaM): retain frames or tokens but lack a compact cross-view world state, yielding low reprojection scores
• Pure implicit memory (SSM): compact but without geometric supervision, poor on both metrics
• GIM-World: combines the compactness of implicit memory with the structured capability of explicit memory

Qualitative Results: 400-Frame Comparison

Figures 2 and 3 show 400-frame comparisons on first-person and third-person scenes:

• Baseline: scene layout begins drifting within 100 frames; walls and landmarks deform and vanish; predictions quickly diverge from the original environment
• Third-person view: baseline either loses the controllable character or renders it inconsistently positioned
• GIM-World: only occasional per-frame viewpoint shifts; overall scene geometry, layout, landmark structure, and character poses stay consistent with ground truth

Stress Test: Loop-Closure Consistency Over 1000 Frames

Figure 4 shows generation over 1000 frames in a single first-person scene:

• Frames 100 and 750 are marked with two nearly identical camera poses
• These views are separated by 600+ frames
• Small objects in the green and red boxes retain, on revisit:
• The same spatial position
• The same relative arrangement
• The same texture details

> Implication: implicit memory stores a stable 3D scene representation rather than merely propagating short-term context. World geometry remains globally consistent across a thousand frames of autoregressive generation.

---

4. Technical Deep Dive: Why This Design Works

4.1 Geometry-as-Memory vs Geometry-as-Input

Existing methods inject geometry into the generator:

• Direct conditioning: concatenating geometric tokens or projected point maps with generator inputs
• Feature supervision: aligning generator backbone features with 3D features during training
• Post-training alignment: deriving rewards or preferences from 3D features for DPO/RL

GIM-World's inverted view:

• Geometry is not an input signal to the generator but a property of memory
• The memory state should encode a world model that is geometrically consistent
• The generator reads conditions from memory and naturally receives geometry-consistent information

Why is this better?

• Separation of concerns: the generator no longer handles geometry and appearance simultaneously
• Memory is explicitly constrained as a "queryable 3D representation" with clear geometric meaning
• The geometry teacher is discarded at inference — zero extra cost

4.2 The Power of Camera Queryability

Traditional 3D representations (NeRF, 3DGS) require reconstructing a scene and then rendering. GIM-World does not:

• Given any camera pose, view-specific features are decoded directly from memory
• No explicit reconstruction, no point clouds or meshes stored
• The decoder is lightweight and fast

This turns "rendering" into "querying" — extracting view-dependent information directly from the memory state rather than generating images from a 3D model.

4.3 The Intuition Behind Information-Guided Pruning

History frames contain massive redundancy:

• Adjacent frames have similar viewpoints and nearly identical content
• Static regions are unchanged for hundreds of frames
• Only dynamic regions truly need remembering

The essence of information-guided pruning:

• Keep frames that provide the most new information
• Discard frames predictable from existing ones
• Approximate the optimal subset greedily

This lets finite-capacity memory tokens carry maximized information.

---

5. Industry Implications

5.1 A Foundation for Interactive Virtual Worlds

Long video generation is not just "making longer videos" — it is a necessary step toward interactive virtual worlds:

• Embodied AI: robots trained in simulated environments need long-term consistent world models
• Game engines: open-world games need scenes to stay consistent no matter how far the player travels
• VR/AR: persistent virtual spaces cannot "forget when you walk away"

GIM-World is the first to demonstrate spatially stable generation at the thousand-frame scale — a key leap from "video generation" to "world simulation."

5.2 What It Means for Kuaishou Kling

The Kuaishou Kling team is among the paper's core authors. This suggests:

• Kling's next-generation models may integrate GIM-World techniques
• The "geometric consistency" problem of long video generation may be getting solved
• Kling's technical depth is shifting from "photorealistic appearance" to "physical consistency"

5.3 Comparison with Other Work

| Work | Method | Strengths | Weaknesses | |------|--------|-----------|------------| | VMem (Oxford) | Explicit geometric memory (surfels) | Interpretable, pluggable | Requires reconstruction, engineering-heavy | | SSM | Pure implicit state | Compact, fast | No geometric constraint | | GIM-World | Implicit + geometric supervision | Compact + geometry-consistent | Relies on a 3D teacher for training |

GIM-World takes a "middle path": the efficiency of implicit memory plus the structured nature of explicit geometry — currently the most practical option.

---

6. Limitations and Future Directions

6.1 Training Depends on a 3D Foundation Model

Geometric supervision requires a frozen 3D teacher (e.g., VGGT). This demands:

• A high-quality 3D teacher
• Training data with camera-pose annotations
• Camera estimation preprocessing for videos without pose information

6.2 The Challenge of Dynamic Objects

The paper focuses on geometric consistency in static scenes. For dynamic objects:

• The queryable geometry head must handle moving objects
• Information-guided pruning may drop key frames containing dynamic changes
• This is a direction for future extension

6.3 From Video to World Model

GIM-World is currently a video generation framework, not a full world model (lacking physical interaction and causal reasoning). Next steps:

• Integrating action–physics feedback loops
• Supporting multi-agent interaction
• Combining with RL training for embodied AI

---

7. Conclusion: A Qualitative Change in Memory

GIM-World's core contribution is not any single module but a paradigm shift:

> From "remembering appearance" to "remembering the 3D world"

Conventional video generation models remember "arrangements of pixels." GIM-World remembers "spatial structure." The qualitative change:

• Walk far away and return — the scene is still consistent
• View the same object from another angle — the geometric relations are correct
• Thousands of frames of autoregressive generation without collapse

With inference overhead under 0.3% of the diffusion backbone, the technique is fully feasible in engineering. No extra geometric plugins, no costly reconstruction pipelines — just add a "geometry teacher" during training.

For long video generation, interactive virtual worlds, and embodied AI simulation, GIM-World offers a practical, scalable, geometry-consistent memory framework.

> "Building a 3D physical space inside an AI's mind" — this vision is becoming reality.

---

References

• Wei, Z., et al. (2026). "Geometry-Aware Implicit Memory for Video World Models." arXiv:2606.02436
• Project page: https://gim-world.github.io/
• Related work: VMem (2025), SSM (2025), VideoSSM (2025), FramePack (2025), CaM (2025)