PVM: Persistent Visual Memory Stops Vision Decay in Large Vision-Language Models

Forum topic · 小凯 · 2026-05-04

Summary

Large vision-language models (LVLMs) suffer from visual signal dilution: as autoregressive text generation gets longer, attention to visual tokens is progressively diluted because the attention denominator grows with every new text token, so the model effectively stops 'looking' at the image and drifts into hallucination. A zhichai.net post introduces Persistent Visual Memory (PVM), a lightweight learnable module proposed in the paper 'Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs' (arXiv:2605.00814, 2026). PVM runs as a parallel branch alongside each FFN layer and implements a distance-independent visual retrieval mechanism: visual features are encoded and stored once, then supplied as context at every decoding step without decaying with sequence length. This lets the model 'look back' at the image throughout generation, improving long-document understanding, multi-turn visual dialogue, and fine-grained description. The post frames PVM as returning to a first-principles question: visual information should be a persistent, on-demand memory, not a one-time input.

The Problem: Models That 'Forget' What They Saw

Imagine chatting with a friend whose eyesight gradually fades. You show him a photo and he describes it well — but as the conversation continues, he relies more on textual memory and less on actually looking. By turn 20, he has forgotten the image and is just fabricating descriptions.

This is precisely the visual signal dilution problem facing large vision-language models (LVLMs).

Why Attention Dilution Happens

Autoregressive LVLMs (like GPT-4V, Claude 3) attend over all preceding tokens at every generation step. As the text sequence grows:

The number of text tokens keeps increasing
The denominator of the attention function gets larger
Visual tokens' attention share is 'diluted'
Result: the longer the generation, the less the model 'looks' at the image

Mathematically, visual attention is inversely proportional to the generated sequence length.

The Fix: Persistent Visual Memory (PVM)

The paper "Persistent Visual Memory: Sustaining Perception for Deep Generation in LVLMs" (arXiv: 2605.00814, 2026-05-01) by Siyuan Huang, Xiaoye Qu, Yafu Li, Tong Zhu, Zefeng He, Muxin Fu, Daizong Liu, Wei-Long Zheng, and Yu Cheng proposes Persistent Visual Memory (PVM), a lightweight learnable module:

Core design:

PVM is integrated as a parallel branch alongside the LVLM's FFN (feed-forward network)
It establishes a distance-independent visual retrieval mechanism
Visual information can be 'called on demand' regardless of how many tokens have been generated

How it works: 1. Visual features are encoded and stored in PVM 2. At each new token, PVM provides 'visual context' 3. This context does not decay with text length 4. The model can 'look back at the image' at any time

Think of it as giving that fading friend smart glasses that let him re-access his visual memory — no matter how long the chat lasts.

Why It Matters

Visual signal dilution is a fundamental limitation of LVLMs:

Long-document understanding: analysis of long image-text reports drifts away from the images later on
Multi-turn dialogue: models progressively 'forget' image details in sustained visual Q&A
Fine-grained generation: later parts of detailed descriptions become increasingly generic

PVM makes visual information a persistent memory rather than a one-time input.

Takeaways for Multimodal Builders

If you build or use multimodal AI systems, ask:

1. Does my model 'forget' visual information in long sequences? 2. Do non-text modalities (image, audio, video) receive sustained attention? 3. Can I design a 'persistent memory' mechanism to maintain multimodal information? 4. Is attention dilution a bottleneck in my system?

PVM reminds us: multimodal fusion is not one-time stitching, but sustained coexistence. True visual understanding is not 'glancing once' — it is 'keeping it in sight.'