Reroute, Don't Remove: Recoverable Visual Token Routing for Efficient Vision-Language Model Inference

Forum topic · 小凯 · 2026-06-12

Summary

A forum post introduces Reroute, a training-free plug-in (arXiv:2606.12412) by Cheng-Yu Yang, Shao-Yuan Lo, and Yu-Lun Liu that improves visual token reduction in vision-language models (VLMs). Existing methods follow a rank-and-remove paradigm: scoring visual tokens, keeping a compact subset, and permanently discarding the rest. The authors show this irreversible removal is fragile because token importance changes across decoder depth—tokens ranked low at one stage may become relevant later, especially for grounding-sensitive queries. Reroute replaces removal with recoverable routing: at each routing stage, selected vision tokens pass through decoder blocks while deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision. Reroute reuses existing attention-score ranking rules and stage schedules, preserving the theoretical TFLOPs and KV-cache budget categories of the pruning methods it enhances. Built on FastV, PDrop, and Nüwa variants with LLaVA-1.5 and Qwen backbones, Reroute improves grounding performance under aggressive token reduction while maintaining general VQA performance, suggesting token reduction should be treated as recoverable routing rather than irreversible pruning.

Paper Overview

Field: Computer Vision (CV) Authors: Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu Published: 2026-06-10 arXiv: 2606.12412

Summary (English)

Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory. Existing visual-token reduction methods largely follow a rank-and-remove paradigm: they score visual tokens, keep a compact subset, and permanently discard the rest.

The authors show that this irreversible action is fragile because visual-token importance changes across decoder depth; tokens ranked low at one stage may become relevant in later layers, especially for grounding-sensitive queries.

Key Idea: Reroute

Reroute is a training-free plug-in that replaces removal with recoverable routing:

At each routing stage, selected vision tokens pass through decoder blocks.
Deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision.
Reroute reuses existing attention-score ranking rules and stage schedules, preserving the theoretical TFLOPs and KV-cache budget categories of the pruning methods it enhances.

Results

Evaluated on FastV, PDrop, and Nüwa variants built on LLaVA-1.5 and Qwen backbones, Reroute:

Improves grounding performance under aggressive token reduction
Maintains general VQA performance

These results suggest that VLM token reduction should not be viewed solely as irreversible pruning, but rather as recoverable routing.

--- *Auto-collected 2026-06-12*