[论文] Can LLMs Improve Their Own Harness? A Controlled Study on ARC-AGI-3

论文概要

研究领域: ML
作者: Jens Lehmann, Sahar Vahdati
发布时间: 2026-10-24
arXiv: 2610.30499

中文摘要

通用智能系统的标志，是无需人类为每个新任务重新设计其软件脚手架就能高效获得新技能。但语言模型很少独自行动：harness 是一个编排模型调用的符号程序，代管感知、记忆、工具、恢复与推理分配。ARC-AGI-3 是一个交互式基准，要求智能体探索陌生环境、即时弄清目标、持续学习——人工引导的系统已在公开游戏上报告了 ceiling 分数，说明有效设计存在。问题随之出现：前沿模型能否自己构建并改进这样的 harness？通过固定的编码 bootstrap，每个模型构建一个完整的任务 harness，在五个开发游戏上经该 harness 行动，并两次整体修订代码库；每个版本都在 20 个从不参与适应的游戏上评估。用相对人类动作效率（RHAE）结合过关率与相对人类基线的动作效率衡量任务表现，并以美元计推理成本作为实用资源度量。27 个谱系的 harness 表现从 0 到接近 ceiling 不等；两轮修订把 held-out 平均 RHAE 仅提高 2.80 分（95% 区间 [−1.44, 7.35]）。开发集上的增益高估了 held-out 改进；推理成本的下降可能以任务表现为代价。人工引导的系统显示出更多余量。这些发现揭示了自主 harness 构建与「两修订」协议下可靠适应之间的鸿沟：即便任务表现很强，也不能保证后续修订能泛化或保住已有能力。

原文摘要

A generally intelligent system acquires new skills efficiently without requiring humans to redesign its software scaffold for every new task. Yet language models rarely act alone. A harness is a symbolic program that orchestrates calls to the model, handling perception, memory, tool use, recovery, and inference allocation on the model's behalf. ARC-AGI-3 is an interactive benchmark in which agents must explore unfamiliar environments, figure out their goals on the fly, and keep learning as they go. On it, human-steered systems already report ceiling scores across the public games, so we know effective designs exist. We ask whether frontier models can build and improve such a harness on their own. Through a fixed coding bootstrap, each model constructs a complete task harness, acts through that harness on five development games, and revises the entire codebase twice. Every version is evaluated on twenty games that never influence adaptation. We measure task performance with Relative Human Action Efficiency (RHAE), which combines level completion with action efficiency relative to human baselines, and record dollar-denominated inference cost as a practical resource measure. Across 27 lineages, harness performance ranges from zero to near-ceiling scores. Two revisions raise mean held-out RHAE by only 2.80 points (95% interval [-1.44,7.35]). Development gains overstate held-out improvement, and reductions in inference cost can come at the expense of task performance. Human-steered systems demonstrate further headroom. These findings reveal a gap between autonomous harness construction and reliable adaptation under this two-revision protocol: even strong task performance does not ensure that subsequent revisions generalize or preserve existing capabilities.

自动采集于 2026-10-24

#论文 #arXiv #ML #小凯