English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Continual-ARC: Measuring Skill Acquisition over a Working Lifetime

Forum topic · 小凯 · 2026-11-12

Summary

Continual-ARC (arXiv:2611.14505, by Daniel Lenton) is a new lifelong-learning benchmark built from the three ARC-AGI editions, designed to compare memory-based and weight-based continual learning systems on the same scale. Tasks require no domain knowledge and recur after dormant gaps with fresh instances while new tasks keep arriving. Learners can pay for demonstrations, submit answers, or act within an agent harness; incorrect submissions can be retried at a cost, and each instance is scored by the share of the feedback budget consumed. After the stream ends, learners are frozen and evaluated for memorization on seen tasks and generalization on unseen tasks. Experiments compared frozen-weight harnesses (with and without persistence) on Qwen3.5-9B and GLM-5.3 Flash against prompting baselines, a program-induction method, and GLM-5.3 Flash-distilled Qwen3.5-9B weights. Results show harness persistence rarely improves memorization without expert guidance, most harness-based methods learn less than simply caching task demonstrations, and no harness-based method gains on unseen tasks—while weight-based learning shows significant improvement, motivating future weight-based continual learning that teaches models to drive harnesses in new domains.

Overview

Research area: Machine Learning Author: Daniel Lenton Published: 2026-11-10 arXiv: 2611.14505

Abstract

Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learners, so memory-based and weight-based systems are rarely compared on the same scale.

Continual-ARC addresses this with a lifelong stream built from the three ARC-AGI editions, whose tasks require no domain knowledge. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit answers, or act inside the harness, with paid retries permitted on incorrect submissions. Each instance is scored based on the share of the feedback budget consumed. Because anything can be retained between instances, memory-based and weight-based systems are scored on the same scale.

At the end of the stream, learners are frozen: memorization is measured on fresh instances of seen tasks, and generalization on unseen tasks.

Experiments and Findings

The authors ran frozen-weight harnesses (with and without persistence) on Qwen3.5-9B and GLM-5.3 Flash, compared against two prompting baselines, a program-induction method, and Qwen3.5-9B weights distilled from GLM-5.3 Flash.

Key results:

  • Persistence in the harness generally does not improve memorization unless there is explicit expert guidance.
  • Across all tested models, most harness-based methods learn less than a simple baseline that caches demonstrations for each task.
  • No harness-based method achieved gains on unseen tasks, while weight-based learning showed significant improvements.
These findings open the door for future weight-based continual learning approaches that teach models to drive harnesses in new domains.

Tags

#continual-learning#benchmark#arc-agi#agent-harness#machine-learning#arxiv#llm

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637067