MemDreamer: Hierarchical Graph Memory and Agentic Retrieval for Long Video Understanding

Forum topic · 小凯 · 2026-06-08

Summary

MemDreamer is a new framework for long-form video understanding that decouples perception from reasoning. Standard vision-language models struggle with hour-long videos because raw frames expand to millions of visual tokens, attention dilutes, and long-range causal dependencies break. MemDreamer instead incrementally builds a three-layer Hierarchical Graph Memory: a Foundation Graph of raw entities, relations, and attributes; a Semantic Graph of themes, motivations, and narrative stages; and a Conceptual Graph of archetypes and philosophical themes. An agentic retrieval engine uses hierarchy navigation, node search, edge traversal, and an Observation-Reason-Action loop to answer queries like a detective cross-checking evidence. Experiments on EgoSchema, Ego4D-LTA, MovieChat-1K, and LVU-QA show absolute gains of +5.8 to +6.9 points over prior SOTA and up to +12.5 points over full-context baselines, while reducing reasoning context to about 2 percent of the original token count. Authors report a strong correlation (r=0.87) between logical-reasoning benchmark scores and long-video understanding, supporting a perception-versus-reasoning design philosophy.

Overview

MemDreamer is a framework for understanding very long videos (e.g., a 10-hour documentary with thousands of scenes) by decoupling perception from reasoning. Instead of feeding an entire video into a vision-language model, MemDreamer incrementally constructs a Hierarchical Graph Memory and uses an agentic, tool-augmented retrieval mechanism to answer detailed questions.

Why Long Video Understanding Is Hard

Token explosion. A 10-hour, 1080p video sampled at 1 fps yields ~36,000 frames. Encoding each frame at 256 visual tokens produces ~9.2M tokens, far beyond what current VLMs can handle.
Attention dilution. Transformer attention scores flatten when sequences grow very long, and critical signals get drowned out.
Causal forgetting. Short-range temporal dependencies (seconds to minutes) work, but cross-hour causal chains are typically lost.

Core Idea: Decoupling Perception and Reasoning

The system separates two processes:

1. Perception layer: builds a structured memory incrementally from the video stream. 2. Reasoning layer: navigates that structured memory on demand, like a detective pulling case files, rather than searching a flat token stream.

The design philosophy mirrors human cognition: we do not "watch" a film once and store every pixel; we retain key scenes, character relations, and causal links and reconstruct answers at query time.

Hierarchical Graph Memory

MemDreamer organises memory into three layers:

Layer 1 — Foundation Graph (raw perception)

Nodes: people, objects, places, events.
Edges: spatial ("A is next to B"), temporal ("A before B"), causal ("A causes B").
Attributes: color, position, action, emotional state.
Construction: incremental VLM-based scene descriptions, entity linking across time, and causal-relation detection models.

Layer 2 — Semantic Graph (abstract understanding)

Nodes: themes (love, betrayal, growth), character motivations, narrative stages (setup, conflict, climax, resolution), emotional arcs.
Construction: clustering and abstraction over the Foundation Graph, pattern detection across events, alignment with narrative-theory templates (e.g., the hero's journey).

Layer 3 — Conceptual Graph (highest abstraction)

Nodes: narrative archetypes (tragedy, comedy, redemption), philosophical themes (free will vs. fate, individual vs. society), universal human experiences (loss, reunion, betrayal, forgiveness).
Construction: pattern matching against the Semantic Graph and classical narrative theory, plus cross-video comparison for general patterns.

Agentic Retrieval Engine

Three core tools, combined in an Observation–Reason–Action loop:

1. Hierarchy Navigation. Route a query to the right layer first.

"What is the theme?" → jump to Conceptual Graph.
"What did the protagonist do in Act III?" → Semantic Graph stage node, then drill into the Foundation Graph for concrete events.

2. Node Search. Find entities by attribute and time.

Example: locate the person with the red shirt, then follow forward in time to discover what happens to them.
Backed by graph-database queries (e.g., Neo4j), vector similarity for fuzzy descriptions ("red shirt" ≈ "crimson shirt"), and temporal filters.

3. Edge Traversal. Walk causal and relational edges.

Example: from a "betrayal" event, traverse causal edges backward to gather all precipitating events, then order them chronologically.
Uses BFS/DFS, multi-hop reasoning (A→B→C→D), and attention-based weighting over multiple causal paths.

The Observation–Reason–Action loop keeps querying and integrating evidence until the answer is grounded or a step budget is reached.

Experimental Results

Evaluated on four long-video benchmarks:

| Benchmark | Prior SOTA | MemDreamer | Δ | |---|---|---|---| | EgoSchema | 52.3% | 58.1% | +5.8 | | Ego4D-LTA | 48.7% | 55.6% | +6.9 | | MovieChat-1K | 61.2% | 67.4% | +6.2 | | LVU-QA | 71.5% | 78.3% | +6.8 |

On LVU-QA, the gap to human experts (avg. 82.0) is only 3.7 points.
Reasoning context reduction: from ~9.2M tokens for full 10-hour video to ~184k tokens (≈2% of the original).
Absolute gain over full-context baseline: +12.5 points, indicating that structured memory is both more efficient and more effective.

Key finding. Logical-reasoning benchmark performance correlates strongly with long-video understanding (r = 0.87), supporting the claim that the core bottleneck is reasoning, not perception.

Implications

From brute force to structured intelligence. Scaling raw context is less effective than organising information as a graph.
Generalizable architecture. The perception/reasoning split can extend to document understanding, long dialogue systems, and scientific-data analysis.
New HCI capabilities. Natural-language Q&A over arbitrary video details, automatic timelines and character-relation maps, and interactive learning from video.
Cognitive science alignment. The design echoes hierarchical memory models (sensory → working → long-term), semantic-network representations of knowledge, and active-reconstruction views of episodic recall.

Limitations

Graph-building cost. Preprocessing is heavy; real-time use (e.g., live streams) needs further optimisation.
Graph quality dependence. Entity-linking and causal-detection errors cascade through the system.
Generalisation. Evaluations focus on narrative content (films, documentaries). Surveillance, scientific, and other non-narrative videos may need different graph schemas.
Interpretability. Retrieval paths are inspectable, but the graph-construction process itself remains automated; better visualisation and human-in-the-loop editing tools are needed.

Future Directions

Real-time, incremental graph construction.
Multimodal memory combining video, audio, text, and sensor streams.
Lifelong learning across tasks and domains.
Collaborative memory shared among multiple agents and humans.
User-editable memory with privacy-aware forgetting controls.

Conclusion

MemDreamer reframes long-video understanding as a memory-organisation problem rather than a perception problem. By replacing flat token streams with a hierarchical relational graph and an agentic retrieval loop, it cuts reasoning context to ~2% of the original while improving accuracy by 6–13 points across four benchmarks. The deeper lesson: intelligence is less about how much information a model ingests and more about how it structures the relationships among that information.

References

Chen, C., et al. (2026). *MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism.* arXiv:2606.07512.
Graesser, A. C., et al. (1994). Memory for metaphorical action in literary comprehension. *Discourse Processes.*
Schank, R. C., & Abelson, R. P. (1977). *Scripts, Plans, Goals, and Understanding.* Lawrence Erlbaum Associates.
Tulving, E. (1972). Episodic and semantic memory. *Organization of Memory.*