Paper Overview
Field: NLP Authors: Haozhen Zhang, Bohan Liu, Jianzhu Bao, Qingyuan Liu, Haodong Yue Published: 2026-11-09 arXiv: 2611.12872
Introduction
As multimodal agents accumulate increasingly long interaction histories, efficiently retrieving and using past information becomes critical for long-term interaction. Existing memory systems mostly rely on semantic retrieval over compressed memories and expose predefined representations, which can miss relevant memories and ignore the heterogeneous evidence needs of different queries.
Method: HiVe-Mem
HiVe-Mem interleaves Horizontal Memory Expansion with Vertical Evidence Composition:
- Horizontal expansion: memory episodes are organized into a relational graph using textual and visual anchors. Semantically retrieved seed nodes are expanded to relevant neighbors, recovering complementary information.
- Vertical composition: each retrieved memory retains multiple evidence forms — compact summaries, captions, grounded visual views, raw images, and interaction history. A lightweight trainable selector composes an evidence subset independently per memory, enabling query-adaptive, heterogeneous multimodal utilization.
Results
Across three multimodal memory benchmarks, HiVe-Mem consistently improves task performance across different base models, while reducing token consumption by about 90% compared with the most efficient competing multimodal baselines.
Original Abstract
We introduce HiVe-Mem, a multimodal memory framework that interleaves Horizontal Memory Expansion with Vertical Evidence Composition. Horizontal expansion organizes memory episodes into a relational graph using textual and visual anchors; vertical composition retains multiple evidence forms per memory with a lightweight trainable selector for query-adaptive utilization, reducing token consumption by about 90%.
---
Paper link: arXiv:2611.12872