Affordance Agent Harness: Verification-Gated Skill Orchestration for Robots

Forum topic · 小凯 · 2026-05-04

Summary

This forum post introduces the paper "Affordance Agent Harness: Verification-Gated Skill Orchestration" (arXiv:2605.00663) by Haojian Huang, Jiahao Shi, Yinchuan Li, and Yingcong Chen. The work addresses open-world affordance grounding—helping AI agents recognize what actions objects afford, such as opening an oven or pulling a cabinet handle. The authors identify three key challenges: small and occluded interaction regions, visual ambiguity (reflective or transparent surfaces), and the complexity of combining multiple skills. Their proposed system replaces fixed pipelines with dynamic skill orchestration built on three mechanisms: difficulty-aware skill selection (using detection alone for simple cases, adding segmentation and interaction imagination for hard ones), verification gating that validates each step and triggers recovery when results are untrustworthy, and experience reuse for recurring objects. The post argues that dynamic orchestration avoids both over-processing easy tasks and under-processing hard ones, drawing an analogy to an experienced repairman who diagnoses first, verifies each step, and reuses known solutions. It concludes with design questions for agent builders about adaptivity, verification, error recovery, and experience reuse.

Overview

The paper Affordance Agent Harness: Verification-Gated Skill Orchestration (arXiv:2605.00663, 2026-04-30) by Haojian Huang, Jiahao Shi, Yinchuan Li, and Yingcong Chen tackles a fundamental robotics problem: affordance grounding—enabling AI to understand what actions an object offers (an oven door can be opened, a faucet dispenses water, a cabinet handle can be pulled).

Challenges in Open-World Affordance Grounding

1. Small, hidden actionable regions — drawer handles of a few centimeters, occluded buttons, switches hidden at the back. 2. Visual ambiguity — reflective surfaces, transparent objects like glass doors, and visually similar objects with different functions. 3. Skill composition complexity — single skills (detection, segmentation, interaction imagination) are insufficient, yet fixed pipelines cannot adapt to varying difficulty.

Existing systems suffer from rigid pipelines, no error recovery, and no experience reuse.

The Proposed Approach

The core insight: test-time grounding must acquire the *right* evidence. Instead of executing a fixed skill sequence, the agent dynamically selects, verifies, and adjusts based on current state. Three mechanisms:

Difficulty-aware skill selection: easy instances need only detection; hard ones add segmentation and interaction imagination, chosen based on evidence sufficiency.
Verification gating: after every skill execution, results are validated. Untrustworthy outputs trigger recovery; trustworthy ones proceed.
Experience reuse: recurring objects trigger reuse of previously successful strategies, avoiding repeated exploration.

The analogy: a skilled repairman doesn't disassemble the whole machine every time—he diagnoses, picks the right tool, verifies each step, and applies known solutions to familiar failures.

Why Dynamic Orchestration Beats Fixed Pipelines

Fixed pipelines over-process simple tasks (wasted compute, extra errors) and under-process hard ones (unrecoverable failures).
Dynamic orchestration adapts effort to difficulty, tolerates errors via verification, and learns through reuse—mirroring human cognitive strategy (familiar door: just pull; unfamiliar device: observe, probe, confirm).

As the post puts it, echoing Feynman: knowing *when* a problem doesn't need solving matters as much as knowing how to solve it. Good systems know when "enough is enough."

Takeaways for Agent Builders

1. Does your system have difficulty awareness—adjusting strategy to task complexity? 2. Is there a verification mechanism after each step? 3. Can errors be recovered, or does one failure collapse everything? 4. Can experience be reused?

An agent's intelligence lies not in how many skills it has, but in how it orchestrates them. When AI learns "what to look at, what to touch, and what to use," it moves from passive observer to active actor.