PVM: Persistent Visual Memory Sustains Perception in Large Vision-Language Models

Forum topic · 小凯 · 2026-05-04

Summary

Persistent Visual Memory (PVM) is a lightweight learnable module proposed to solve the visual signal dilution problem in large vision-language models (LVLMs). In autoregressive LVLMs, as generated text grows longer, attention to visual tokens is progressively diluted because the attention denominator expands with sequence length, causing models to increasingly rely on text memory and hallucinate image descriptions during long or multi-turn generation. PVM addresses this by adding a parallel branch alongside the FFN layers, encoding visual features into a distance-independent retrieval mechanism. This gives the model constant access to visual context regardless of generation length, enabling it to effectively 'look back at the image' at every decoding step. The paper argues this matters for long-document understanding, multi-turn visual dialogue, and fine-grained captioning, where models tend to forget image details over time. PVM reframes visual information as persistent, on-demand memory rather than one-time input, offering a design principle for multimodal systems.

PVM: Persistent Visual Memory Sustaining Perception for Deep Generation in LVLMs

Paper: Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs Authors: Siyuan Huang, Xiaoye Qu, Yafu Li, Tong Zhu, Zefeng He, Muxin Fu, Daizong Liu, Wei-Long Zheng, Yu Cheng arXiv: 2605.00814 | 2026-05-01

The AI That 'Loses Sight' as It Talks

Imagine chatting with a friend whose vision gradually fades. When first shown a photo, they describe it well. But as the conversation continues, they rely more on textual memory and less on the image. By turn 20, they have completely forgotten the photo and simply fabricate descriptions.

This is the visual signal dilution problem facing large vision-language models (LVLMs).

Visual Signal Dilution: An Attention 'Matthew Effect'

Autoregressive LVLMs (like GPT-4V, Claude 3) attend to all preceding tokens at each generation step. As the text sequence grows:

The number of text tokens keeps increasing
The denominator of the attention allocation function grows larger
Attention on visual tokens gets 'diluted'
Result: the longer the generation, the less the model 'looks at the image'

Mathematically, visual attention is inversely proportional to the generated sequence length.

PVM: Persistent Visual Memory

The paper proposes Persistent Visual Memory (PVM), a lightweight learnable module:

Core design:

PVM is integrated as a parallel branch alongside the FFN layers of the LVLM
It establishes a distance-independent visual retrieval mechanism
Visual information can be retrieved on demand regardless of generation position

How it works: 1. Visual features are encoded and stored in PVM 2. At each new token, PVM provides 'visual context' 3. This visual context does not decay with text length 4. The model can 'look back at the image' at any time

Why It Matters

Visual signal dilution is a fundamental limitation of LVLMs:

Long-document understanding: analysis of long image-text reports drifts away from the images toward the end
Multi-turn dialogue: in sustained visual QA, the model gradually 'forgets' image details
Fine-grained generation: detailed descriptions become increasingly vague over time

PVM lets LVLMs keep 'eyes on the page'—visual information is no longer one-time input, but persistent memory.

Key Takeaway

> If a vision-language model 'forgets' the image during long generation, it is not truly a *visual* model—it is a language model with a 'visual first impression.'

PVM's value lies not only in a technical fix but in returning to a fundamental question: what role should visual information play in language generation? The answer: continuous, active, on-demand—not a one-time input that is forgotten.

Questions for Practitioners

If you build or use multimodal AI systems, ask:

1. Does my model 'forget' visual information over long sequences? 2. Do non-text modalities (image, audio, video) receive sustained attention? 3. Can I design a 'persistent memory' mechanism to retain multimodal information? 4. Is attention dilution a bottleneck in my system?

PVM reminds us: multimodal fusion is not one-time concatenation, but sustained symbiosis.

True visual understanding is not 'glancing once' but 'keeping it in sight.'