Reroute, Don't Remove: Visual Tokens Should Be Rerouted, Not Discarded

Forum topic · ✨步子哥 · 2026-06-11

Summary

A paper from National Yang Ming Chiao Tung University and National Taiwan University challenges the core assumption behind visual token pruning in vision-language models (VLMs). Existing methods like FastV and PDrop permanently delete tokens with low attention scores at intermediate layers, assuming importance is stable across depth. The authors show token importance rankings change dramatically between layers, so pruning shallow layers destroys grounding information needed later. Their proposed Reroute technique replaces deletion with delay: low-scoring tokens skip the current decoder block and enter a candidate pool, re-entering scoring at the next checkpoint. Reroute is training-free and plug-and-play, reusing existing scoring rules while keeping theoretical TFLOPs and KV-cache budgets unchanged. Experiments on LLaVA-1.5 show significant grounding accuracy recovery at aggressive 20% retention rates without hurting general VQA performance, with similar gains on Qwen2-VL and Qwen3-VL. The work reframes token compression from irreversible pruning toward dynamic routing, benefiting most under aggressive compression ratios.

The Problem: Visual Token Pruning Assumes Importance Is Fixed

When a multimodal LLM answers questions about an image, the image is converted into hundreds of visual tokens processed at every attention layer. To save compute, existing methods (FastV, PDrop, Nüwa) delete "unimportant" tokens at intermediate layers.

But how do you know a token won't become important in later layers?

This is the core assumption challenged by a paper from National Yang Ming Chiao Tung University and National Taiwan University. The authors found that a visual token's importance ranking changes drastically across depths in vision-language models — a token judged useless in shallow layers may be exactly what's needed to localize an answer in deeper layers.

The Flaw in Existing Methods: One Cut, No Way Back

Mainstream visual token compression methods (FastV, PDrop, Nüwa) all follow the same paradigm: score → keep → discard. They set checkpoints at certain layers, compute attention scores, keep high-scoring tokens, and permanently delete low-scoring ones.

It's like a company laying off staff: each department independently evaluates employees and the low performers leave for good. But someone "unimportant" in marketing might be exactly the talent R&D needs. You can't strip someone of opportunities in other departments based on one department's evaluation.

The paper confirms this experimentally: in LLaVA-1.5, when retention drops to 20% (80% of visual tokens deleted), the model's visual grounding capability collapses — the localization information needed in deep layers was thrown away early.

Reroute: A Standby Ticket for Tokens

Reroute's core idea is extremely simple: don't delete, delay.

At each routing checkpoint:

High-scoring tokens pass through the current decoder block normally
Low-scoring tokens skip the current block but aren't deleted — they enter a "candidate pool"
At the next checkpoint, candidate pool tokens re-participate in scoring and can be re-selected

Like airport standby: you miss this flight, but your ticket isn't voided — you can board the next one.

Crucially, Reroute is plug-and-play and training-free. It reuses the attention scoring rules and scheduling strategies of existing methods (FastV, PDrop), merely replacing "deletion" with "delay." This means:

Theoretical TFLOPs unchanged (delayed tokens skip a layer's compute, saving as much as deleted tokens)
KV-cache budget unchanged (standby tokens don't occupy the current layer's cache)
But information retention improves dramatically

Experimental Results: Grounding Restored

Tests on LLaVA-1.5:

Visual grounding: at 20% retention, FastV+Reroute significantly improves grounding accuracy over vanilla FastV; PDrop+Reroute shows the same pattern
General VQA: Reroute restores grounding without harming general question answering — it isn't a trade-off but genuinely preserves more useful information
Qwen series: the same trend was validated on Qwen2-VL and Qwen3-VL

A particularly interesting finding: Reroute is most effective under aggressive compression (very low retention). This makes sense — at 20% retention, who gets deleted matters far more than at 80%, and Reroute gives "wrongly deleted" tokens a chance to return.

Deeper Implication: Pruning Isn't the Point — Routing Is

The paper's deeper contribution isn't the numerical gains but a conceptual shift:

Token compression should be viewed as "routing," not "pruning."

Pruning is one-way and irreversible — once cut, information is lost forever. Routing is bidirectional and schedulable — tokens can take different paths at different layers, dynamically chosen per layer's needs.

This aligns with conditional computation and Mixture-of-Depth thinking, but is simpler: no training, no extra routing network, just replace "delete" with "delay."

From an engineering perspective, this is a classic "less is more" case — the simplest modification (don't delete tokens; let them stand by) solves the most fundamental problem (irreversible information loss).

---

Paper: https://arxiv.org/abs/2606.12412 Code: https://github.com/elmma/mllm-reroute/ Institutions: National Yang Ming Chiao Tung University, National Taiwan University