DIRECT: When and Where to Allocate Test-Time Compute in Embodied Robot Planners

Forum topic · 小凯 · 2026-06-11

Summary

A Chinese tech forum post explains DIRECT (Dynamic Inference Router for Embodied Compute Tradeoffs), a framework from Stanford, Waterloo, and NVIDIA researchers that addresses how embodied AI systems should allocate test-time compute. The paper diagnoses three axes of compute scaling: chain-of-thought depth, model size, and memory history. Key findings: 44% of tasks perform equally well with non-thinking VLMs at 50x lower latency; larger models mainly expand skill coverage rather than per-task performance; no single memory architecture dominates all difficulty levels. DIRECT trains a lightweight router (20-50ms inference) using frozen SigLIP and BGE-M3 embeddings to select the cheapest planner that meets a quality tolerance. Experiments on VLABench, RoboMME, and physical Franka hardware show the router recovers near-frontier success rates while cutting latency by up to 65%, converting non-monotonic scaling curves into monotones across a 2B-235B model pool. The post argues test-time compute is not a uniform lever, and smart allocation beats brute-force scaling for real-world robot deployment. Source: arXiv:2606.12402.

DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?

> Paper: DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners? > Authors: Jadelynn Dao, Milan Ganai, Yasmina Abukhadra, et al. (Stanford / Waterloo / NVIDIA) > arXiv: arXiv:2606.12402

The Problem: Blind Compute Scaling in Embodied AI

Modern embodied AI systems use a hierarchical structure: a high-level VLM planner decomposes language instructions into sub-skills, and a low-level VLA policy executes them. When robots use VLM planners, they typically apply the most expensive reasoning mode to *every* task — deep chain-of-thought for frying eggs and for pouring milk alike. The result: exploding latency and cost, with uneven and sometimes diminishing returns.

DIRECT asks a simple but deep question: when and where should test-time compute be invested in embodied agents?

Three Systematic Diagnoses

1. Chain-of-Thought Depth

Comparing Qwen3-VL 8B Instruct vs. Thinking modes on VLABench:

Thinking models do better on average, but in 44% of tasks the non-thinking model performs comparably or better, at over 50x lower latency (under 2% of the thinking latency).
CoT benefits are task-dependent — deep reasoning only pays off when tasks involve implicit semantic, physical, or spatial constraints.

2. Model Size (2B → 235B)

Performance and latency are both non-monotonic. A 32B model can be slower than 235B because token generation does not scale directly with parameters — smaller models can be verbose.
Model size mainly expands skill coverage, not per-skill quality. On skills small models already handle well, larger models add nothing — like a bigger toolbox that doesn't tighten screws any faster.

3. Memory History (RoboMME benchmark)

On simple tasks, FrameSamp outperforms MemER with orders of magnitude fewer FLOPs.
On complex long-horizon tasks, MemER and GroundSG show advantages.
No single architecture dominates across all difficulty levels.

The DIRECT Framework

DIRECT formulates planning as a selection problem: given a fixed pool of K VLM planners, choose one per task input (scene image + instruction) to maximize a utility function trading off quality and cost:

k_i^* = \arg\max_k U(q_{i,k}, c_{i,k})

A concrete utility uses predicted quality and cost:

U_{\text{reg}}(\hat{q}_{i,k}, \hat{c}_{i,k}) = \frac{1}{\hat{c}_{i,k}} \cdot \mathbb{1}\left[\max_{k'} \hat{q}_{i,k'} - \hat{q}_{i,k} < \alpha\right]

Key design choices:

Lightweight router: frozen SigLIP for vision + frozen BGE-M3 for text, followed by a simple model (linear, KNN, MLP, etc.). Inference costs only 20–50 ms — negligible versus >1s VLM planners.
Synthetic data collection: on physical hardware (Franka), tasks are synthesized with VLMs and scored by an LLM judge, avoiding exhaustive real-robot rollouts.

Results

CoT routing: ~30% latency reduction while recovering near-Thinking quality (routing efficiency η ≈ 75%); works across open-source, closed-source, and cross-family model pairs.
Size routing: cumulative routing across a 2B+4B+8B+32B pool turns non-monotonic scaling curves into monotonic ones, gaining 5.1 percentage points in performance while cutting average latency by 32.4 seconds versus the largest model.
Physical validation (DROID setup, Franka arm): matched or exceeded stronger models' success rates with up to 65% lower average latency, validated with 270,000+ simulated routing decisions and 245 physical trajectories.
A step requiring 21.9 s of deep thinking can be compressed to 0.8–0.9 s in simple scenes — not by cutting corners, but because the router knows the scene doesn't require deep thought.

Why It Matters

Test-time compute is not a uniform lever: CoT depth buys reasoning depth, model size buys skill breadth, and memory buys long-horizon capability — three orthogonal, complementary axes. Task difficulty can be inferred cheaply from the initial scene and instruction, enabling low-cost upfront routing instead of indiscriminate expensive computation.

DIRECT differs from Mixture-of-Experts: MoE routes at token level to expand model capacity; DIRECT routes at task level to optimize the quality-cost Pareto frontier.

Limitations

1. Fixed pool assumption: adding a new planner requires re-collecting data and retraining the router. 2. Single-step routing: difficulty may vary across stages of long-horizon tasks; dynamic re-routing is future work. 3. Cost proxies: latency and FLOPs are deployment-specific, not universal constants.

As the paper concludes: "Naively scaling test-time compute is wasteful, and that DIRECT can provide frontier-level embodied planning in robotic systems at a fraction of the cost."