LocateAnything: Parallel Box Decoding Makes VLM Visual Grounding 10x Faster

Forum topic · 小凯 · 2026-06-07

Summary

LocateAnything (NVIDIA et al., arXiv:2605.27365) replaces token-by-token bounding box serialization in vision-language models with Parallel Box Decoding (PBD), generating each 6-token box block (<box>, four coordinates, </box>) in a single forward pass. Training jointly optimizes next-token-prediction and multi-token-prediction streams with a three-tier attention mask (causal, block-causal, bidirectional intra-block), preserving language ability while learning box-level parallelism. A hybrid inference mode runs fast parallel decoding by default and falls back to sequential decoding on format anomalies or spatial ambiguity (top-1 probability < 0.7). The model is trained on LocateAnything-Data: 138M queries, 12M images, 785M boxes covering detection, grounding, UI elements, documents, OCR, and pointing. A 3B model achieves 12.7 boxes-per-second on H100 (vs. 1.1 for Qwen3-VL-4B, ~10x faster), while improving LVIS mean F1 by +3.8 to 50.7 and reaching 60.3 F1 on ScreenSpot-Pro GUI grounding, surpassing 30B+ models. Limitations include no RL training yet and validation limited to boxes and points.

LocateAnything: Parallel Box Decoding for Fast, High-Quality VLM Grounding

> Paper: *LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding* > Authors: Shihao Wang, Shilong Liu, et al. (NVIDIA and collaborators) > Link: https://arxiv.org/abs/2605.27365

Key points

2.5x decoding speedup over prior VLM grounding models, LVIS mean F1 +3.8
138M training samples unifying detection, grounding, UI, documents, and OCR
ThreeB-scale model outperforms 30B+ models on GUI grounding

The problem

VLMs typically serialize 2D bounding boxes into 1D token streams (textual digit sequences or quantized coordinates), causing:

Inference bottleneck: strict serial decoding (300 boxes = 1200 serial steps)
Structural mismatch: box coordinates are geometrically coupled, but tokens are trained largely independently
Error propagation: one wrong coordinate distorts the whole box
Low throughput in dense scenes

Core innovation: Parallel Box Decoding (PBD)

LocateAnything treats the bounding box as an atomic generation unit. Four block types, each exactly 6 tokens: Semantic, Box (<box> x1 y1 x2 y2 </box>), Negative, and End. All 6 tokens of a box are generated in parallel in one forward pass.

Three inference modes

| Mode | Mechanism | Throughput (BPS) | |---|---|---| | Fast | pure MTP, parallel per box | 16.9 | | Slow | pure NTP, token-by-token | 3.9 | | Hybrid | Fast by default, fallback to Slow | 12.7 |

Hybrid mode falls back when: (1) token order violates the box format, or (2) spatial ambiguity (top-1 coordinate probability < 0.7 with top-5 max-min spread > 80 in normalized 0–1000 space).

Training: dual-sequence joint supervision

Input concatenates visual tokens, query tokens, an NTP sequence, and an MTP (block-wise) sequence. A three-tier attention mask isolates the streams:

1. Causal attention for the NTP stream + shared context 2. Block-causal between MTP blocks 3. Bidirectional within each 6-token MTP block

Loss: L = L_NTP + L_MTP. Ablations show pure MTP training collapses (47.2 F1); joint training is essential (52.1 Slow / 51.6 Hybrid).

Data engine: LocateAnything-Data

12M unique images, 138M natural language queries, 785M boxes
Task mix: 66.9% detection, 16.5% UI grounding, 7.3% referring, 3.6% text, 3.5% document layout, 2.2% pointing
A synthesis pipeline prompts Qwen3-VL to generate queries from detection datasets, verifies points via Molmo, and generates boxes with SAM 3
Explicit negative samples teach the model not to hallucinate absent objects

Results

| Model | Throughput (BPS) | LVIS F1 | ScreenSpot-Pro F1 | |---|---|---|---| | Qwen3-VL-4B | 1.1 | – | 53.7 (30B-A3B) | | Rex-Omni-3B | 5.0 | 46.9 | – | | LocateAnything-3B | 12.7 | 50.7 | 60.3 |

Dense detection: VisDrone 39.9 F1 (+4.1 over Rex-Omni)
Documents: M6Doc 70.1 F1 (+14.5), DocLayNet 76.8 (+6.1), TotalText 43.3 (+2.7)
Ablations confirm box-aligned PBD beats generic multi-token methods (SDLM, Block Diff)

Why it matters

Demonstrates structured outputs can be generated atomically in parallel, suggesting applications to 3D boxes, keypoints, masks, and video trajectories
Engineering highlights: Stream Packing (>95% batch utilization) and MagiAttention for heterogeneous attention masks
Limitations: no RL yet, only boxes/points validated, and training cost (4 stages, 256x H100)

Conclusion

LocateAnything lifts the bounding box from a 1D token sequence to an atomic parallel unit, solving VLM grounding's speed, accuracy, and robustness pain points in one framework: ~10x throughput, higher detection F1, and hybrid reliability fallback. The takeaway: the future of multimodal generation lies in smarter structures, not longer sequences.

References

Wang S, Liu S, Kuang Y, et al. LocateAnything. arXiv:2605.27365, 2026.
Bai K, et al. Qwen3-VL Technical Report, 2025.
Jiang Y, et al. Rex-Omni, 2025.
Chen J, et al. Pix2Seq, ICLR 2022.