AdaCodec: A Predictive Visual Code for Video MLLMs

Forum topic · 小凯 · 2026-06-03

Summary

AdaCodec is a predictive visual code for video multimodal large language models (video MLLMs), introduced in an arXiv paper (2506.00008) by Haowen Hou, Zhen Huang, and Zheming Liang. The key insight is that video is temporally redundant: adjacent frames usually share most objects, background, and layout, yet existing video MLLMs encode each sampled frame as an independent RGB image, wasting visual tokens on repeated content. AdaCodec instead sends a full reference frame only when the scene cannot be well predicted from prior context; otherwise, it encodes inter-frame changes—including motion and prediction residuals—as compact P-tokens. Across all 11 benchmarks, AdaCodec outperforms the per-frame RGB baseline of Qwen3-VL-8B at matched visual token budgets. Even with one-seventh of the budget (32k tokens), AdaCodec beats the 224k-token baseline on all long-video benchmarks. On five general video benchmarks, it improves average scores while cutting time-to-first-token from 9.26 seconds to 1.62 seconds, offering a more efficient video interface for multimodal models.

Paper Overview

Field: NLP Authors: Haowen Hou, Zhen Huang, Zheming Liang Published: 2026-06-03 arXiv: 2506.00008

English Summary

Video is temporally redundant: adjacent frames usually share most objects, background, and layout. Yet existing video multimodal large language models (video MLLMs) usually encode each sampled frame as an independent RGB image, causing visual tokens to repeat content already present in earlier frames.

This suggests a more direct video interface: send a full reference frame only when the scene cannot be predicted well from prior context, and otherwise transmit a compact description of inter-frame changes. The authors call this interface a predictive visual code, and instantiate it for video MLLMs as AdaCodec.

AdaCodec spends full visual tokens on a reference frame only when its conditional predictive cost is high; otherwise, it encodes inter-frame changes, including motion and prediction residuals, as compact P-tokens.

Key Results

Outperforms the per-frame RGB baseline of Qwen3-VL-8B across all 11 benchmarks at matched visual token budgets.
Even at 1/7 of the budget (32k tokens), AdaCodec surpasses the 224k-token baseline on all long-video benchmarks.
On five general video benchmarks, it improves average scores while reducing time-to-first-token from 9.26s to 1.62s.

Original Abstract (excerpt)

> Video is temporally redundant: adjacent frames usually share most objects, background, and layout. Yet existing video multimodal large language models (video MLLMs) usually encode each sampled frame as an independent RGB image, causing visual tokens to repeat content already present in earlier frames. This suggests a more direct video interface: send a full reference frame only when the scene cannot be predicted well from prior context, and otherwise transmit a compact description of inter-frame changes. We call this interface a predictive visual code, and instantiate it for video MLLMs as AdaCodec...

*Auto-collected on 2026-06-03*