RoboArchitect: Automating Agent Harness Design for Robotic Policies

Forum topic · 小凯 · 2026-11-03

Summary

RoboArchitect (arXiv 2610.42141) automates the design of agent harnesses around frozen robot policies. Recent work shows frozen robot policies perform far better inside an agent harness — control code that decides when to query the policy, consult vision tools, or directly command the arm — but such harnesses are typically hand-built per robot or benchmark. RoboArchitect uses a coding model to observe robot rollouts, diagnose failures, and iteratively revise the harness. Without updating any model weights, it raises average success rates from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x) across multiple benchmarks and policies. Gains are largest when benchmarks diverge from the policy's training data, where evolved harnesses sometimes outperform finetuned versions. The system also automatically discovers intuitive control strategies and evolves persistent memory carrying learned experience into future decisions.

Paper Overview

Field: CV
Authors: Shengguang Wu, Suning Huang, Cherie Ho, Ria Doshi, Hao Zhu, Serena Yeung-Levy
Released: 2026-10-30
arXiv: 2610.42141

Abstract (Translation)

Recent work shows that a frozen robot policy performs far better inside an agent harness, where control code decides when to query the policy, consult vision tools, or command the arm directly. Yet today these harnesses are built by hand and redesigned for each new robot or benchmark. RoboArchitect shows that harness design can be automated, offering a strong alternative to finetuning on novel setups.

In RoboArchitect, a coding model observes a robot's rollouts, diagnoses failures, and iteratively revises the harness around the frozen policy. The harness evolves on a set of optimization episodes and is evaluated on held-out scenes and tasks.

Key Results

Without updating any model weights, RoboArchitect raises average success from 9.7% to 43.3% on unseen scenes (4.5x) and from 12.7% to 40.2% on unseen tasks (3.2x), across multiple benchmarks and policies.
Gains are largest when the benchmark is far from the policy's training data: wrapping a general policy with the evolved harness produces an expert that often even outperforms its finetuned counterpart.
Intuitive control strategies emerge automatically, such as lifting a stuck arm to clear it, or reaching horizontally when obstacles block from above.
The harness also evolves its own persistent memory, carrying learned experience (e.g., effective grasp depths) into subsequent decisions.

Takeaway

A large share of potential improvement in robotic manipulation lives in the agent harness, and its design can be effectively automated and optimized without any weight updates.

---

*Auto-collected on 2026-11-03*