NVIDIA LocateAnything: Parallel Box Decoding Makes Visual Grounding 10x Faster and More Accurate

Forum topic · 小凯 · 2026-05-30

Summary

NVIDIA, together with Hong Kong Polytechnic University and Nanjing University, introduces LocateAnything, a vision-language model that replaces token-by-token bounding box generation with Parallel Box Decoding (PBD). Instead of serially emitting coordinates like a typewriter, PBD treats each box as a fixed-length atomic unit predicted in a single parallel step, using block-level causal masking so coordinates inside a box attend to each other while boxes remain autoregressive. Built on Moon-ViT, an MLP projector, and Qwen2.5, the model offers three inference modes: Fast (multi-token prediction, ~15-17 boxes/sec), Slow (next-token prediction, highest accuracy), and Hybrid, which falls back from Fast to Slow only when format errors or spatial ambiguity are detected. On a single NVIDIA H100, LocateAnything-3B reaches 12.7 boxes per second in Hybrid mode—about 10x faster than Qwen3-VL-30B-A3B and 2.5x faster than Rex-Omni-3B—while setting new SOTA results on LVIS, COCO, DocLayNet, M6Doc (+14.5 F1), ScreenSpot-Pro, HumanRef, and pointing benchmarks. Trained on 12M images with 138M queries and 785M boxes spanning detection, GUI grounding, referring expression, OCR, layout, and pointing tasks, it delivers one unified model for six grounding capabilities.

Overview

NVIDIA, with Hong Kong Polytechnic University, Nanjing University, and collaborators, released LocateAnything, a vision-language model for visual grounding whose core innovation is Parallel Box Decoding (PBD). Instead of generating bounding box coordinates as a sequence of discrete tokens, PBD treats each box (or point) as a fixed-length atomic unit and predicts all four coordinates in one parallel step—turning "typewriter mode" into "stamp mode." Result: roughly 10x faster decoding with *higher* accuracy.

Key points

The problem: Conventional VLMs serialize bounding boxes token by token (<box> <x1> <y1> <x2> <y2> </box>), which breaks geometric coupling between coordinates, forces strict serial decoding, and risks malformed outputs. A dense scene with 300 boxes would require ~1200 generation steps.
PBD design:
Each box is an atomic token block; coordinates inside a block use bidirectional attention (they know each other), while blocks remain causally ordered (block-level causal masking).
Joint training with NTP (next-token prediction) and MTP (multi-token prediction) teaches the model both precision and speed.
Three inference modes:
Fast: pure MTP parallel decoding—fastest, slightly lower accuracy; suits on-device robots and real-time agents.
Slow: pure NTP—highest accuracy (COCO F1 52.1, 3.9 BPS); suits offline annotation.
Hybrid (default): runs Fast, but falls back to NTP re-decoding when it detects (1) format irregularities or (2) spatial ambiguity—top-1 coordinate token probability < 0.7 AND top-5 coordinate spread > 80 in the [0,1000] normalized space. Hybrid reaches 12.7 BPS with COCO F1 51.6 (97% of Slow accuracy at 78% of Fast speed).

Benchmarks (LocateAnything-3B, single NVIDIA H100)

| Benchmark | Task | Result | Gain vs best baseline | |---|---|---|---| | LVIS | open-world detection | F1 50.7 | +3.8 vs Rex-Omni | | COCO | generic detection | F1 54.7 | +1.8 vs Rex-Omni | | DocLayNet | document layout | F1 76.8 | +6.1 | | M6Doc | Chinese document layout | F1 70.1 | +14.5 | | ScreenSpot-Pro | GUI grounding | Avg 60.3 | +2.3 vs GUI-Owl-32B | | HumanRef | referring expression | F1@0.95 68.8 | +3.4 | | Pointing (7 tasks) | point localization | first on all | — |

Speed: 12.7 boxes/sec (Hybrid) vs Qwen3-VL-30B-A3B at 1.1 BPS (0.09x) and Rex-Omni-3B at 5.0 BPS (0.39x). The advantage widens with box count: as scenes go from 20 to 300 boxes, NTP latency explodes linearly while PBD throughput rises from ~12 to ~25 BPS.

At IoU=0.95 on LVIS, LocateAnything scores 31.1 vs Rex-Omni's 20.7, showing that block-level parallelism improves fine-grained geometric consistency.

Architecture and training

Architecture: Moon-ViT (native-resolution vision encoder) → MLP projector → Qwen2.5 decoder → PBD output head. Main model size: 3B parameters.
Training data (LocateAnything-Data): 12M images, 138M natural-language queries, 785M boxes across six domains—generic detection, GUI element grounding, referring expressions, OCR text localization, layout analysis, and pointing. Annotation combines public datasets, GUI collections, and a synthetic engine (Qwen3-VL for queries, Molmo for points, SAM 3 for masks/boxes, Rex-Omni + Qwen3-VL verification), plus 22M negative samples to curb hallucination.
Four-stage training: world knowledge → detection/grounding basics → full 138M-query joint training (lr 4e-5) → dense-scene enhancement with 80% multi-object data (lr 1e-5).

Ablations confirm each choice: PBD (Slow) beats quantized coordinates by +2.0 F1 at equal throughput; PBD (Fast) is 4.3x faster than quantized MTP with only -0.5 F1; X-Y corner ordering beats other box orderings (52.1 F1).

Limitations and future work

Currently supervised fine-tuning only—reinforcement learning could reduce fallback rates.
Hybrid worst-case speed in dense scenes approaches Slow mode.
Results mostly tied to the Moon-ViT + Qwen2.5 stack (though PBD also validated on Qwen3-VL-4B).

Future directions include RL-optimized decoding, video temporal grounding, and integration with NVIDIA GR00T, Cosmos, and Metropolis.

Why it matters

The deeper lesson: geometric structure should not be hijacked by 1D token sequences. Outputs with internal structural coupling—polygon segmentation, 3D boxes, pose keypoints, temporal action spans—could all benefit from the same atomic-block parallel prediction template.

Reference

Wang, S., Liu, S., Kuang, Y., et al. (2026). *LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding*. arXiv:2605.27365.