Visual Tokens on Demand: mmVB — Query-Conditioned Visual Budgeting for Scalable Multi-Image Reasoning

Forum topic · 小凯 · 2026-10-15

Summary

A new arXiv paper (2610.18858) introduces mmVB, a lightweight plug-and-play module for efficient multi-image reasoning in multimodal large language models (MLLMs). Most MLLMs are optimized for single-image inputs, and multi-image inference is costly because each added image brings a large number of visual tokens. Existing token reduction methods—pruning, merging, compression, learned resampling, hierarchical compression, retrieval-augmented reasoning, and memory-based architectures—are mostly designed for single images where redundancy is spatial and local; they either require attention-matrix access (impractical with FlashAttention) or model retraining. mmVB addresses scalable multi-image reasoning without retraining the base model or accessing attention matrices at inference. It allocates query-conditioned per-image token budgets based on cross-image context, adding only 1.6M trainable parameters while reducing visual input tokens by 9x. This yields 5x faster time-to-first-token and 7x fewer prefill FLOPs, while preserving 94.4% of full-resolution accuracy across eight multi-image reasoning benchmarks.

Paper Overview

Field: Computer Vision (CV) Authors: Aimon Rahman, Joymallya Chakraborty, Vidit Bansal, Abhishek Dan Published: 2026-10-13 arXiv: 2610.18858

Background

Multimodal large language models (MLLMs) have advanced rapidly, but most remain optimized for single-image inputs. Multi-image reasoning is costly because each image adds many visual tokens, increasing encoding and attention demands. Existing techniques—including visual token pruning, token merging, visual token compression, learned resampling, hierarchical compression, retrieval-augmented visual reasoning, and memory-based architectures—have largely been designed for single-image scenarios, where redundancy is primarily spatial and local.

Challenges in Multi-Image Reasoning

Multi-image reasoning introduces additional difficulties:

Redundancy may occur across images, not just within one
Task-relevant evidence may be sparse or distributed across multiple images
Aggressive per-image compression can remove details needed for comparison, counting, temporal reasoning, retrieval, or evidence anchoring

Limitations of Existing Token Reduction Methods

Current approaches fall into two categories, each with drawbacks:

1. Training-free but inference-inefficient: typically rely on accessing attention matrices to determine token importance, which is impractical with memory-efficient attention implementations such as FlashAttention 2. Training-dependent: require retraining the base MLLM or adapting it to a new compressed-token paradigm

Proposed Method: mmVB

The paper studies scalable multi-image reasoning and understanding without retraining the base model or accessing attention matrices at inference time. The authors propose mmVB, a lightweight plug-and-play module that:

Allocates query-conditioned per-image token budgets based on cross-image context
Adds only 1.6M trainable parameters
Reduces visual input tokens by 9x

Results

The token reduction delivers:

5x faster time-to-first-token
7x fewer prefill FLOPs
94.4% of full-resolution accuracy retained across eight multi-image reasoning benchmarks

---

*Auto-collected on 2026-10-15*