DIRECT: Routing Test-Time Compute for Embodied VLM Planners

Forum topic · 小凯 · 2026-06-12

Summary

DIRECT is a routing framework that decides when and where to spend test-time compute for vision-language models (VLMs) acting as high-level planners for embodied agents. The authors, from Stanford and collaborators (arXiv:2606.12402), observe that naively scaling test-time compute — via chain-of-thought depth, larger models, or longer memory history — raises latency, token usage, and FLOPs while producing uneven, often diminishing gains in downstream task success. DIRECT instead uses multimodal scene context to allocate compute per prompt, improving the success-cost Pareto frontier over fixed model selection. Experiments on VLABench and RoboMME show that the three scaling axes yield qualitatively different capability benefits, meaning test-time compute is not a uniform lever. The approach was validated on a physical Franka arm in DROID setups covering zero-shot manipulation and long-horizon chained tasks, where DIRECT matched or exceeded the success rates of stronger models while reducing average latency by up to 65%. The results suggest that blindly scaling test-time compute is wasteful and that per-prompt routing enables frontier-level embodied planning at a fraction of the cost.

Overview

Field: Computer Vision (CV) Authors: Jadelynn Dao, Milan Ganai, Yasmina Abukhadra, Ajay Sridhar, Mozhgan Nasr Azadani, Katie Luo, Clark Barrett, Jiajun Wu, Chelsea Finn, Marco Pavone Released: 2026-06-10 arXiv: 2606.12402

Abstract

Vision-Language Models (VLMs) are increasingly deployed as high-level planners for embodied agents, with an emerging strategy of scaling test-time compute to improve capability. However, the authors observe that doing so increases latency, token usage, and FLOPs while yielding uneven, often diminishing gains in downstream success, limiting where embodied agents can be deployed. They argue that choosing when and where to spend test-time compute is central to bringing frontier performance to the real world.

The paper introduces DIRECT, a routing framework that uses multimodal scene context to allocate compute per prompt, improving the success-cost Pareto frontier over fixed model selection. Across three dominant scaling axes — chain-of-thought depth, model size, and memory history — experiments on VLABench and RoboMME show that test-time compute is not a uniform lever: different axes produce qualitatively different capability gains.

These insights were validated on a physical Franka arm in DROID setups, covering zero-shot manipulation and long-horizon chained tasks. The router matches or exceeds the success rates of stronger models while reducing average latency by up to 65%.

Conclusion

Naively scaling test-time compute is wasteful. DIRECT enables frontier-level embodied planning in machine systems at a small fraction of the cost.

--- *Auto-collected on 2026-06-12*