Good Token Hunting: A Hitchhiker's Guide to Token Selection for Visual Geometry Transformers

Forum topic · 小凯 · 2026-05-26

Summary

This paper introduces a token selection framework that accelerates visual geometry transformers used for multi-view 3D reconstruction. These models predict multiple 3D attributes in a feed-forward manner, but their global attention layers cause computational cost to grow quadratically with input sequence length. The proposed solution restricts the number of key/value tokens each query interacts with during global attention, using a two-stage framework: an inter-frame selection step identifies which frames to preserve at the frame level, and an intra-frame selection step discards redundant tokens within selected frames. Analysis highlights the advantage of diversity-based inter-frame selection for broad scene coverage, and shows intra-frame selection requires layer-aware sparsification guided by the entropy of global attention patterns. Experiments demonstrate superior speed-accuracy trade-offs over existing solutions: on 500-image scenes the method accelerates visual geometry transformers by over 85% while maintaining or even improving baseline performance, suggesting token selection will play a key role in future applications. Paper by Shuhong Zheng, Michael Oechsle, and Erik Sandström, arXiv:2505.21383.

Paper Overview

Field: Computer Vision Authors: Shuhong Zheng, Michael Oechsle, Erik Sandström Published: 2026-05-26 arXiv: 2505.21383

Abstract

Visual geometry transformers have become powerful architectures for multi-view 3D reconstruction, enabling joint prediction of multiple 3D attributes in a feed-forward manner. However, their computational cost grows quadratically with the input sequence length due to the global attention layers inside these models. This limits both their scalability and efficiency.

In this work, the authors address this challenge with a simple yet general strategy: restricting the number of key/value tokens that each query interacts with during global attention. To achieve effective token selection, they introduce a two-stage framework:

1. Inter-frame selection — operates at the frame level to identify frames that should be preserved. A diversity-based selection strategy is highlighted as advantageous, ensuring broad coverage of the scene. 2. Intra-frame selection — further discards more redundant tokens within the selected frames. The analysis shows layer-aware sparsification is needed, with the selection process guided by the entropy of global attention patterns.

Results

The method offers a superior speed-accuracy trade-off compared to existing solutions. Extensive experiments show it accelerates visual geometry transformers by over 85% in scenes with 500 images, while maintaining or even improving baseline performance — suggesting the token selection strategy can play a key role in future applications of visual geometry transformers.