[论文] Continual-ARC: Measuring Skill Acquisition over a Working Lifetime

论文概要

研究领域: ML
作者: Daniel Lenton
发布时间: 2026-11-10
arXiv: 2611.14505

中文摘要

智能体框架（agent harness）作为持续学习机制与测试时参数学习并存展现前景，但当前基准存在局限：多聚焦单一应用领域、每个实例只出现一次、且假设单一学习器家族，记忆型与权重型系统很少在同一尺度上被比较。我们提出 Continual-ARC：由 ARC-AGI 三个版本构成的终身流，任务不需要领域知识。相同任务在休眠间隔后带着新实例回归，新任务持续到来。每个实例上，学习器可付费获取示范、提交答案或在框架内行动，错误可重试但需付费；每个实例按所消耗反馈预算份额评分。由于实例间可保留任何东西，记忆型与权重型系统在同尺度下被评分。流结束后，学习器被冻结，在见过的任务的新实例上测记忆，在未见任务上测泛化。我们在 Qwen3.5-9B 与 GLM-5.3 Flash 上运行冻结权重的框架（有无持久化），对比两个提示基线、一种程序归纳法与用 GLM-5.3 Flash 蒸馏的 Qwen3.5-9B 权重。除非有显式专家指导，框架中的持久化通常不提升记忆。在我们测试的所有模型上，多数基于框架的方法还不如简单缓存每个任务示范的基线学得多。没有任何基于框架的方法在未见任务上取得增益，而基于权重的学习展现显著提升。这为未来"教会模型在新领域驱动框架"的权重型持续学习打开了大门。

原文摘要

Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, the current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learner, so memory-based and weight-based systems are rarely compared on the same scale. We propose Continual-ARC, a lifelong stream built from the three ARC-AGI editions, whose tasks need no domain knowledge. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit answers, or act inside the harness, with retries permitted on incorrect submissions, which are paid for. Each instance is scored based on the sh...

自动采集于 2026-11-12

#论文 #arXiv #ML #小凯