Profiling a Frontier Agent with EBench: From Direct Control to Harness with Policy

Forum topic · 小凯 · 2026-10-27

Summary

This technical report evaluates GPT-6-Astra, a frontier multimodal large language model (MLLM) agent, on the EBench robotics benchmark, comparing its capabilities against post-trained vision-language-action (VLA) and world action model (WAM) policies under an identical visual and proprioceptive observation protocol. Two configurations are studied: direct control, where the agent generates waypoint commands itself, and harness with policy, where it can invoke a post-trained policy for action execution. Under direct control, the agent performs competitively overall, leading on short-horizon tasks and showing the least variation across generalization dimensions, but lagging on long-horizon tasks, tabletop manipulation, and high-precision contact-rich tasks. With harness invocation of post-trained policies, combined strengths raise overall success to the level of the strongest post-trained policies while substantially reducing token cost. Performance still falls below per-task upper bounds due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control. The study characterizes frontier MLLM agent capability boundaries and points to improvements in harness design. Paper: arXiv 2610.35832.

Paper Overview

Field: ML
Authors: Yifei Yao, Yude Zou, Guanbo Jiao, Xing Gao, Ning Gao et al.
arXiv: 2610.35832
Project page: https://internrobotics.github.io/EBench-Astra-Tech-Report/

Abstract

Frontier multimodal large language models (MLLMs) can control robots without robot-specific training, but how their capabilities compare and interact with post-trained vision-language-action (VLA) and world action model (WAM) policies across manipulation demands remains to be thoroughly evaluated.

In this report, the authors evaluate GPT-6-Astra on EBench as the core of an agent that controls a robot through an agent harness under the same visual and proprioceptive observation protocol as post-trained policies. Two agent configurations are studied:

1. Direct control — the agent generates waypoint commands directly. 2. Harness with policy — the agent can also invoke a post-trained VLA or WAM policy for action execution.

Key Findings

Direct control: The agent achieves competitive overall performance relative to post-trained policies. It leads on short-horizon tasks and exhibits the least performance variation across generalization dimensions.
Weaknesses of direct control: It underperforms post-trained policies on long-horizon tasks, tabletop manipulation, and high-precision contact-rich tasks.
Harness with policy: Invoking a post-trained VLA or WAM combines the complementary strengths of both components, raising overall task success to the level of the strongest compared post-trained policies while substantially reducing token cost.
Remaining gaps: Performance remains below the per-task upper bound of the standalone components due to suboptimal policy invocation, fixed-instruction execution without subgoals, and grasp disruption when the agent resumes control.

Conclusion

The study characterizes the capability boundary of a frontier MLLM agent and suggests directions for improving future agent harness design.