Affordance Agent Harness: Verification-Gated Skill Orchestration for Open-World Robots

Forum topic · 小凯 · 2026-05-04

Summary

This forum post introduces the paper 'Affordance Agent Harness: Verification-Gated Skill Orchestration' (arXiv: 2605.00663) by Haojian Huang, Jiahao Shi, Yinchuan Li, and Yingcong Chen. The paper tackles the open-world affordance grounding problem: how AI agents and robots determine what actions objects afford (opening an oven door, pulling a cabinet handle). Open-world grounding faces three challenges: small and occluded interactive regions, visual ambiguity from reflective or transparent surfaces, and the complexity of composing multiple skills. Existing systems rely on fixed pipelines that either over-process easy cases or fail on hard ones, with no error recovery or experience reuse. The proposed framework introduces three mechanisms: difficulty-aware skill selection that dynamically matches skills (detection, segmentation, interaction imagination) to instance complexity; verification gating that validates each skill's output and triggers recovery when untrustworthy; and experience reuse for recurring objects. The author argues this mirrors human cognitive strategy—applying minimal effort to familiar cases and deeper analysis to unfamiliar ones—citing Feynman's insight that knowing when not to solve a problem matters. The post closes with design questions for agent builders: Does the system adapt to difficulty, verify each step, recover from errors, and reuse experience? Dynamic orchestration, not fixed pipelines, defines truly intelligent agents.

Paper: Affordance Agent Harness: Verification-Gated Skill Orchestration Authors: Haojian Huang, Jiahao Shi, Yinchuan Li, Yingcong Chen arXiv: 2605.00663 | 2026-04-30

The AI That Stares at an Oven but Doesn't Know It Can Open It

Imagine a robot walking into a kitchen. It sees an oven, a refrigerator, a faucet, and cabinets—but doesn't know that the oven door can be opened, the fridge can be raided, the faucet dispenses water, or the cabinet handle can be pulled.

This is the affordance problem: what action possibilities does an object provide?

Three Challenges of Open-World Affordance Grounding

1. Small, hidden actionable regions — drawer handles are only centimeters wide; buttons may be occluded; switches may hide on the back. 2. Visual ambiguity — reflective surfaces, transparent objects like glass doors, and visually similar objects with different functions. 3. Skill composition complexity — no single skill (detection, segmentation, interaction imagination) suffices; multiple skills must combine, but fixed pipelines can't adapt to varying difficulty.

Blind spots of existing systems:

Fixed pipelines apply the same process regardless of difficulty
No error recovery: a single mid-step failure dooms the run
No experience reuse: similar objects are explored from scratch every time

Verification-Gated Skill Orchestration

The paper proposes the Affordance Agent Harness, built on a core insight:

> Test-time grounding must acquire the right evidence. Rather than executing a fixed skill sequence, the agent should dynamically select, verify, and adjust based on the current state.

Three key mechanisms:

1. Difficulty-aware skill selection — easy instances need only detection; hard ones add segmentation and interaction imagination. Skills are chosen dynamically based on the sufficiency of current evidence. 2. Verification gating — after each skill executes, the result is validated. Untrustworthy results trigger recovery mechanisms; trustworthy ones proceed. Every step is checked, not blindly executed. 3. Experience reuse — recurring objects are recognized and previously successful strategies are reused, avoiding repeated exploration.

It's like an experienced repair technician: not disassembling the whole machine every time, but diagnosing first, choosing the right tool, verifying each step, and applying known solutions to familiar faults.

Why Dynamic Orchestration Beats Fixed Pipelines

Fixed pipelines suffer from:

Over-processing: complex flows for simple tasks waste compute and may introduce unnecessary errors
Under-processing: simple flows can't solve hard tasks and can't recover from errors

Dynamic orchestration offers:

Right-sized effort: skills matched to difficulty
Fault tolerance: verification gating ensures quality
Learning: experience reuse improves efficiency

This mirrors human cognitive strategy: pull a familiar door handle directly; carefully observe, probe, and confirm with an unfamiliar device. Not everything gets a full analysis.

A Feynman-Style Judgment: Good Systems Know When to Stop Thinking

Feynman observed that knowing how to solve a problem matters, but knowing when it *doesn't* need solving matters more. In affordance grounding: not every instance needs the full skill suite. Simple tasks should resolve quickly; hard ones deserve deep analysis. Good systems know "when enough is enough."

The verification-gated philosophy: intelligence lies not only in *what an agent can do*, but in *knowing what to do when*.

Fixed pipeline = mechanical executor
Dynamic orchestration = agent with judgment

Takeaways for Agent and Robotics Builders

Ask yourself:

1. Does my system have difficulty awareness—adjusting strategy to task complexity? 2. Is there a verification mechanism after every step? 3. Can errors be recovered, or does one mistake collapse everything? 4. Can experience be reused?

The Affordance Agent Harness reminds us: an AI agent's intelligence lies not in how many skills it has, but in how it orchestrates them. In the open world, fixed pipelines are brittle. Dynamic orchestration that senses, verifies, adjusts, and learns—that's a truly intelligent agent.

When AI learns *what to look at, what to touch, and what to use*, it transforms from a passive observer into an active actor.