Reroute, Don't Remove: Visual Tokens Should Be Rerouted, Not Discarded

Forum topic · ✨步子哥 · 2026-06-11

Summary

A paper from National Yang Ming Chiao Tung University and National Taiwan University challenges the core assumption behind visual token pruning in vision-language models (VLMs). Existing methods like FastV, PDrop, and Nüwa score visual tokens at intermediate layers and permanently delete low-scoring ones to save compute. The authors show that a token's importance ranking changes dramatically across model depth: tokens deemed unimportant in shallow layers may be crucial for grounding in deeper layers. In LLaVA-1.5, when the retention rate drops to 20%, grounding capability collapses because localization-critical information is discarded early. Their proposed method, Reroute, replaces deletion with deferral: low-scoring tokens skip the current decoder block but remain eligible, re-entering the scoring pool at the next checkpoint. Reroute is training-free and plug-and-play, reusing existing scoring rules and scheduling policies while keeping theoretical TFLOPs and KV-cache budgets unchanged. Experiments show significant grounding recovery on LLaVA-1.5 with FastV and PDrop, consistent gains on Qwen2-VL and Qwen3-VL, no degradation on general VQA, and the largest benefits under aggressive compression. The deeper contribution is conceptual: token compression should be viewed as routing rather than pruning.

You ask a multimodal LLM "Where is the cat in the picture?". The model splits the image into hundreds of visual tokens and processes them through attention at every layer. To save compute, existing methods delete "unimportant" tokens at intermediate layers.

The problem: how do you know a token won't become important in later layers?

That is the core assumption this paper from National Yang Ming Chiao Tung University and National Taiwan University challenges. They found that a given visual token's "importance ranking" changes dramatically across depths in a VLM — tokens sentenced to death in shallow layers may be exactly the key to localizing the answer in deeper layers.

The fatal flaw of existing methods: one cut, never coming back

Current mainstream visual token compression methods (FastV, PDrop, Nüwa, etc.) all follow the same paradigm: score → keep → discard. They set "checkpoints" at certain layers, compute attention scores for each token, keep the high-scoring ones, and permanently delete the low-scoring ones.

It's like a company doing layoffs: each department independently evaluates employee value, and low scorers are let go immediately. But someone "unimportant" in marketing might be exactly the talent R&D desperately needs. You can't strip someone of opportunities in other departments because they performed mediocrely in one.

The paper experimentally confirms this intuition: in LLaVA-1.5, when the retention rate drops to 20% (i.e., 80% of visual tokens deleted), the model's grounding capability collapses — because the localization information needed at deeper layers was thrown away in shallow layers.

Reroute: a "standby ticket" for tokens

Reroute's core idea is extremely simple: don't delete, defer.

Concretely, at each routing checkpoint:

High-scoring tokens pass through the current decoder block normally
Low-scoring tokens skip the current block but are not deleted — they enter a "standby pool"
At the next checkpoint, tokens in the standby pool re-enter scoring and get a chance to be selected again

It's like airport standby boarding: you missed this flight, but your ticket isn't voided — you still have a chance on the next one.

Crucially, Reroute is plug-and-play and training-free. It reuses the attention scoring rules and scheduling policies of existing methods (FastV, PDrop, etc.), simply changing "delete" into "defer". This means:

Theoretical TFLOPs unchanged (deferred tokens skip one layer of compute, saving just as much as deleted tokens)
KV-cache budget unchanged (standby tokens don't occupy the current layer's cache)
But information retention is substantially improved

Experimental results: grounding capability significantly recovered

Tests on LLaVA-1.5:

Grounding: at a 20% retention rate, FastV+Reroute improves grounding accuracy over vanilla FastV by a significant margin; PDrop+Reroute shows the same
General VQA: Reroute recovers grounding without harming general question answering — it's not trading off grounding vs. VQA, it genuinely retains more useful information
Qwen family: the same trend is validated on Qwen2-VL and Qwen3-VL

A particularly interesting finding: Reroute works best under aggressive compression (very low retention rates). That matches intuition — when you keep only 20% of tokens, "who got deleted" matters far more than when you keep 80%. And Reroute gives "wrongly deleted" tokens exactly the chance to come back.

The deeper insight: pruning isn't the endgame, routing is

The paper's deeper contribution is not the specific numeric gains, but a conceptual shift:

Token compression should not be seen as "pruning" but as "routing."

Pruning is a one-way, irreversible decision — once cut, the information is lost forever. Routing is a bidirectional, schedulable decision — tokens can take different paths at different layers, dynamically selected by current needs.

This idea is in line with conditional computation and Mixture-of-Depth, but simpler: no training needed, no extra routing network needed — just change "delete" to "defer".

From an engineering perspective, this is a classic "less is more" case — the simplest change (don't delete tokens, let them stand by) solves the most fundamental problem (irreversible information loss).

---

Paper: https://arxiv.org/abs/2606.12412 Code: https://github.com/elmma/mllm-reroute/ Institutions: National Yang Ming Chiao Tung University, National Taiwan University