DIRECT: Stanford-Led Team Reveals the 'Compute Economics' of Embodied AI Planning

Forum topic · 小凯 · 2026-06-11

Summary

A Stanford, Waterloo, and NVIDIA collaboration introduces DIRECT (Dynamic Inference Router for Embodied Compute Tradeoffs), a lightweight task-level router that decides when and where to allocate test-time compute in embodied AI planners. The team's systematic diagnosis across three axes — chain-of-thought depth, model size, and memory history — shows that scaling test-time compute yields uneven, task-dependent returns: in 44% of VLABench tasks, a non-thinking Qwen3-VL 8B matched the thinking variant at over 50x lower latency; larger models mainly expand skill coverage rather than per-task quality; and simple memory architectures outperform heavy ones on short-horizon tasks. DIRECT learns to predict quality-cost tradeoffs from a frozen SigLIP and BGE-M3 embedding, then selects the best planner from a fixed pool with only 20-50ms routing overhead. On a physical Franka arm (DROID setup), DIRECT matched stronger planners' success rates while cutting average latency by up to 65%, validated by over 270,000 simulated routing decisions and 245 real-world trajectories. The paper argues test-time compute is not a uniform lever, and that smart allocation — not brute-force scaling — is the path to deployable embodied intelligence. arXiv:2606.12402.

DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?

> Paper: DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners? > Authors: Jadelynn Dao, Milan Ganai, Yasmina Abukhadra, et al. (Stanford / Waterloo / NVIDIA) > arXiv: 2606.12402

The Problem: Robots Waste Compute on Easy Tasks

Modern embodied AI systems use a hierarchical structure: a high-level VLM planner decomposes language instructions into sub-skills, and a low-level VLA policy executes motor control. The community's default strategy is to scale test-time compute uniformly — deeper chain-of-thought (CoT), bigger models, longer memory — regardless of task difficulty. This is like deep-thinking about fluid dynamics just to pour a glass of milk.

The DIRECT team (Dynamic Inference Router for Embodied Compute Tradeoffs) systematically diagnoses this waste across three axes.

Key Findings

Axis 1: Chain-of-Thought Depth

Comparing Qwen3-VL 8B Instruct vs. Thinking on VLABench: the Thinking model wins on average, but in 44% of tasks the non-thinking model performs as well or better — with over 50x lower latency (under 2% of the delay).
CoT benefits are task-dependent: deep reasoning pays off only when tasks involve implicit semantic, physical, or spatial constraints.

Axis 2: Model Size

Sweeping Qwen3-VL from 2B to 235B parameters: both performance and latency follow non-monotonic curves. A 32B model can be slower than 235B because token generation doesn't scale with parameters — smaller models sometimes produce verbose outputs.
Model size mainly expands skill coverage, not per-skill quality. On skills small models already handle, larger models offer no advantage.

Axis 3: Memory History

On RoboMME, lightweight FrameSamp outperforms MemER by orders of magnitude fewer FLOPs on simple tasks, while MemER and GroundSG only show advantages on complex long-horizon tasks. No single memory architecture dominates all difficulty levels.

The DIRECT Framework

DIRECT formalizes routing as a selection problem: given a fixed pool of K VLM planners with different cost/capability profiles, pick the best one per task input (scene image + instruction).

Data collection: quality matrix Q (success rates) and cost matrix C (latency or FLOPs) over N tasks × K planners. On physical hardware, the team synthesizes tasks with VLMs and scores them with an LLM judge, avoiding exhaustive real-robot rollouts.

Router: a lightweight model over frozen SigLIP (vision) and BGE-M3 (text) embeddings — candidates include linear models, KNN, K-means, and a two-layer MLP. Routing costs only 20-50ms, negligible versus any VLM planner (>1s).

Utility function:

k_i^* = \arg\max_k U(q_{i,k}, c_{i,k}), \quad U_{\text{reg}} = \frac{1}{\hat{c}_{i,k}} \cdot \mathbb{1}\left[\max_{k'} \hat{q}_{i,k'} - \hat{q}_{i,k} < \alpha\right]

Minimize cost subject to quality being within a tolerance α of the best.

Results

CoT routing (Qwen3-VL 8B Instruct vs. Thinking): ~30% latency reduction while recovering near-Thinking quality (routing efficiency η ≈ 75%). Works across closed-source and cross-family heterogeneous models too.
Size routing: pairwise routing beats either model alone; cumulative routing converts non-monotonic scaling curves into monotonic ones. The cumulative router (2B+4B+8B+32B) improves performance by 5.1 percentage points while cutting average latency by 32.4 seconds.
Physical Franka arm (DROID setup): matched or exceeded stronger models' success rates with up to 65% lower average latency — e.g., compressing a 21.9-second reasoning step to 0.8-0.9 seconds on simple scenes. Validated with 270,000+ simulated routing decisions and 245 real trajectories.

Why It Matters

Test-time compute is not a uniform lever. CoT depth buys reasoning depth, model size buys skill breadth, and memory buys long-horizon capability — orthogonal gains that shouldn't be maxed simultaneously. Task difficulty can be inferred from the initial scene and instruction alone, enabling ultra-cheap upfront triage to avoid indiscriminate expensive computation. Unlike MoE (token-level routing to expand capacity), DIRECT routes at the task level to optimize the quality-cost Pareto frontier.

Limitations

1. Fixed pool assumption: adding planners requires recollecting data and retraining. 2. Single-step decision: routing happens once per task; multi-stage tasks could benefit from dynamic re-routing. 3. Cost proxies: latency/FLOPs are deployment-specific, not universal constants.

> "Naively scaling test-time compute is wasteful, and DIRECT can provide frontier-level embodied planning in robotic systems at a fraction of the cost."

The shift from "how much compute can we stack?" to "how smartly can we allocate what we have?" may be the key to moving embodied AI from labs into factories, hospitals, and homes.

References

1. Dao, J., et al. (2026). DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners? arXiv:2606.12402. https://arxiv.org/abs/2606.12402 2. Brohan, A., et al. (2023). RT-2: Vision-Language-Action Models. arXiv:2307.15818. 3. Team, Q. (2025). Qwen3-VL Technical Report. arXiv:2505.18201. 4. Snell, C., et al. (2024). Scaling LLM Test-Time Compute Optimally. arXiv:2408.03314. 5. Ong, I., et al. (2024). RouteLLM: Learning to Route LLMs with Preference Data. arXiv:2406.18665. 6. Ganai, M., et al. (2025). VLABench. arXiv:2502.03258. 7. Ding, Y., et al. (2024). RoboMME. arXiv:2406.12424. 8. Wu, J., et al. (2025). DROID. arXiv:2403.12945.