Overview
Paper: *LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding* Authors: Shihao Wang, Shilong Liu, Yuanguo Kuang, Xinyu Wei, Yangzhou Liu, Zhiqi Li, Yunze Man, Guo Chen, Andrew Tao, Guilin Liu, Jan Kautz, Lei Zhang, Zhiding Yu (NVIDIA et al.) Link: https://arxiv.org/abs/2605.27365
Headline results: 2.5× decoding speedup, LVIS mean F1 +3.8%, and a unified 138M-sample training set covering detection, grounding, UI, documents, and OCR.
The Problem: Sequential "Toothpaste" Decoding
Mainstream VLMs serialize 2D bounding boxes into 1D token streams—either as digit characters ("1" "0" "2" "4" ...) or quantized coordinates predicted one at a time. This causes:
- Inference bottleneck: each coordinate waits for the previous one (strictly serial)
- Structural mismatch: box coordinates are inherently coupled, but tokens are trained largely independently
- Error propagation: one wrong coordinate skews the whole box
- Low throughput: 300 boxes = 1200 serial steps in dense scenes
- 12M unique images, 138M natural-language queries, 785M boxes
- Task mix: general detection 66.9%, UI grounding 16.5%, referring 7.3%, text localization 3.6%, document/layout 3.5%, point localization 2.2%
- Synthesis pipeline: detection datasets (OpenImages, Objects365) are augmented by prompting Qwen3-VL with class labels to generate rich queries; Molmo predicts points, and points inside ground-truth boxes are kept as supervision. For unlabeled images, Qwen3-VL generates queries, Molmo points are converted to boxes with SAM 3, then Qwen3-VL verifies.
- Explicit negative samples teach the model to say "not found" instead of hallucinating.
- Coordinate representation: box-aligned PBD beats textual (1.3 BPS, 49.1 F1) and quantized (3.9 BPS, 50.1 F1) representations; PBD (Slow) reaches 52.1 F1 at the same throughput.
- MTP formulation: structure-agnostic multi-token prediction (SDLM, Block Diff) is harmful (44.8–46.5 F1); box-aligned PBD (Fast) reaches 49.6 F1 at 16.9 BPS.
- Loss: joint NTP+MTP is required; MTP-only training underperforms.
- Breaks the "generation = serial" assumption: structured outputs (3D boxes, keypoints, masks, trajectories) could likewise be generated as atomic parallel units.
- Data-scale effect: 3B parameters beat 30B+ models on several benchmarks.
- Embodied AI: Fast Mode (16.9 BPS) enables real-time perception; Hybrid Mode preserves reliability.
- Engineering highlights: Stream Packing (>95% batch utilization for variable-length sequences) and MagiAttention (distributed training with heterogeneous attention masks).
- SFT only; RL-based optimization of block-level policies is future work.
- Validated on boxes and points only; polygons, 3D cuboids, and poses remain untested.
- Training cost: 4 stages, 256×H100, tens of thousands of steps.
- Wang S, Liu S, Kuang Y, et al. LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding. arXiv:2605.27365, 2026.
- Bai K, et al. Qwen3-VL Technical Report. 2025.
- Jiang Y, et al. Rex-Omni: Unified Detection and Grounding in a VLM Framework. 2025.
- Chen J, et al. Pix2Seq: A Language Modeling Framework for Object Detection. ICLR, 2022.
Core Idea: Parallel Box Decoding (PBD)
Since a bounding box is a geometric whole, LocateAnything generates it as an atomic unit. Four fixed-length (6-token) block types are defined:
| Block | Content | Length |
|---|---|---|
| Semantic | language description (long text split into blocks) | 6 |
| Box | <box> x1 y1 x2 y2 </box> | 6 |
| Negative | target absent | 6 |
| End | termination | 6 |
All 6 tokens of a box block are generated in parallel in one forward pass.
Three Inference Modes
| Mode | Mechanism | Throughput | Use case | |---|---|---|---| | Fast | pure MTP, boxes in parallel | 16.9 BPS | latency-sensitive, robotics, embedded | | Slow | pure NTP, token-by-token | 3.9 BPS | high-precision annotation | | Hybrid | Fast by default, fallback to Slow | 12.7 BPS | production, balanced |
Hybrid mode falls back on: (1) format anomalies (token order deviates from the box structure), or (2) spatial ambiguity (top-1 coordinate probability < 0.7 and top-5 max-min spread > 80 in the 0–1000 normalized space).
Training: Dual-Sequence Joint Supervision
Input: x_all = [visual_tokens] + [query_tokens] + [NTP_sequence] + [MTP_sequence]
The attention mask has three regimes: 1. Causal attention over the NTP stream + shared context 2. Block-causal between MTP blocks 3. Bidirectional intra-block within each 6-token MTP block
NTP and MTP streams cannot see each other (no data leakage), but both see the shared visual/text context. Loss: L = L_NTP + L_MTP. Ablations show joint training is essential—pure MTP training collapses to 47.2 F1 vs 52.1 for joint (Slow).
Data Engine: LocateAnything-Data
Results
Throughput (H100, batch size 1)
| Model | BPS | Relative | |---|---|---| | Qwen3-VL-4B | 1.1 | 1× | | Rex-Omni-3B | 5.0 | 4.5× | | LocateAnything-3B | 12.7 | 11.5× |
Accuracy
| Benchmark | Rex-Omni-3B | LocateAnything-3B | |---|---|---| | LVIS mean F1 | 46.9 | 50.7 (+3.8) | | COCO mean F1 | 52.9 | 54.7 (+1.8) | | Dense200 mean F1 | 58.3 | 58.7 | | VisDrone mean F1 | 35.8 | 39.9 (+4.1) | | ScreenSpot-Pro F1 | (GUI-Owl-32B: 58.0) | 60.3 | | DocLayNet mean F1 | 70.7 | 76.8 (+6.1) | | M6Doc mean F1 | 55.6 | 70.1 (+14.5) | | TotalText mean F1 | 40.6 | 43.3 (+2.7) | | RefCOCOg val mean F1 | 73.6 | 76.7 (+3.1) |
Ablations
Why It Matters
Limitations
Conclusion
LocateAnything elevates the bounding box from a 1D token sequence to an atomic parallel unit, addressing speed (10× throughput), precision (box-aligned supervision), and robustness (Hybrid fallback). Combined with the 138M-sample data engine, it offers a complete unified visual grounding solution—a statement that multimodal generation's future lies in smarter structures, not longer sequences.
References