Reroute, Don't Remove: Recoverable Visual Token Routing for Vision-Language Models

Forum topic · 小凯 · 2026-06-12

Summary

Reroute is a training-free plug-in that improves visual token reduction in vision-language models by replacing irreversible token pruning with recoverable routing. Existing rank-and-remove methods permanently discard low-ranked visual tokens, which is fragile because token importance varies across decoder depth—tokens deemed unimportant early may become relevant later, especially for grounding-sensitive queries. Reroute lets selected tokens pass through decoder blocks while deferred tokens bypass a routing stage and re-enter the candidate pool at the next decision. It reuses existing attention-score ranking rules and stage schedules, preserving the theoretical TFLOPs and KV-cache budget of the base pruning methods. Applied to FastV, PDrop, and Nüwa variants on LLaVA-1.5 and Qwen backbones, Reroute improves grounding performance under aggressive token reduction while maintaining general VQA performance, showing that token reduction should be treated as recoverable routing rather than irreversible pruning. arXiv: 2606.12412.

Paper Overview

Field: Computer Vision (CV) Authors: Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu Published: 2026-06-10 arXiv: 2606.12412

Abstract

Vision-language models (VLMs) project images into hundreds to thousands of visual tokens, making decoder inference expensive in both attention computation and KV-cache memory. Existing visual-token reduction methods largely follow a rank-and-remove paradigm: they score visual tokens, keep a compact subset, and permanently discard the rest. The authors show that this irreversible action is fragile because visual-token importance changes across decoder depth; tokens ranked low at one stage may become relevant in later layers, especially for grounding-sensitive queries.

They propose Reroute, a training-free plug-in that replaces removal with recoverable routing. At each routing stage, selected vision tokens pass through decoder blocks, while deferred tokens bypass the stage and re-enter the candidate pool at the next routing decision. Reroute reuses existing attention-score ranking rules and stage schedules, maintaining the theoretical TFLOPs and KV-cache budget class of the pruning method it enhances.

Key Results

Evaluated on FastV, PDrop, and Nüwa variants with LLaVA-1.5 and Qwen backbones.
Reroute improves grounding performance under aggressive token reduction while preserving general VQA performance.
The findings suggest VLM token reduction should be treated as recoverable routing rather than irreversible pruning.

---

*Auto-collected on 2026-06-12*