Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers

Forum topic · 小凯 · 2026-05-26

Summary

This paper introduces Good Token Hunting, a token selection framework for visual geometry transformers used in multi-view 3D reconstruction. These transformers predict multiple 3D attributes in a feed-forward manner, but their global attention layers make computational cost grow quadratically with input sequence length, limiting scalability. The proposed solution restricts the number of key/value tokens each query interacts with in global attention via a two-stage framework: an inter-frame selection step operating at the frame level to identify frames to preserve, and an intra-frame selection step that discards redundant tokens within selected frames. The analysis highlights diversity-based inter-frame selection for broad scene coverage, and shows intra-frame selection requires layer-aware sparsification guided by the entropy of global attention patterns. Experiments demonstrate a better speed-accuracy trade-off than existing solutions, accelerating visual geometry transformers by over 85% on 500-image scenes while maintaining or improving baseline performance. Authors: Shuhong Zheng, Michael Oechsle, Erik Sandström. arXiv: 2505.21383.

Paper Overview

Field: Computer Vision (CV) Authors: Shuhong Zheng, Michael Oechsle, Erik Sandström arXiv: 2505.21383 Project page: https://zsh2000.github.io/good-token-hunting.github.io

Abstract

Visual geometry transformers have become powerful architectures for multi-view 3D reconstruction, enabling joint prediction of multiple 3D attributes in a feed-forward manner. However, their computational cost grows quadratically with the input sequence length due to the global attention layers inside these models. This limits both their scalability and efficiency.

In this work, the authors address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention. To achieve effective token selection, they introduce a two-stage framework:

1. Inter-frame selection: operates at the frame level to identify frames that should be preserved. The analysis highlights the advantage of diversity-based strategies, ensuring broad coverage of the scene. 2. Intra-frame selection: further discards more redundant tokens within the selected frames. This step requires layer-aware sparsification, guided by the entropy of global attention patterns.

Results

Better speed-accuracy trade-off compared to existing solutions.
Speeds up visual geometry transformers by over 85% on scenes with 500 images.
Maintains or even improves baseline performance, suggesting token selection will play a key role in future applications of visual geometry transformers.