Overview
Research area: Machine Learning Author: Daniel Lenton Published: 2026-11-10 arXiv: 2611.14505
Abstract
Agent harnesses have emerged as a promising mechanism for continual learning, alongside test-time parametric learning. However, current benchmarks have several limitations: they generally focus on a single applied domain, serve each instance only once, and assume a single family of learners, so memory-based and weight-based systems are rarely compared on the same scale.
Continual-ARC addresses this with a lifelong stream built from the three ARC-AGI editions, whose tasks require no domain knowledge. The same tasks return after dormant gaps with fresh instances, while new tasks keep arriving. On each instance, the learner may pay for demonstrations, submit answers, or act inside the harness, with paid retries permitted on incorrect submissions. Each instance is scored based on the share of the feedback budget consumed. Because anything can be retained between instances, memory-based and weight-based systems are scored on the same scale.
At the end of the stream, learners are frozen: memorization is measured on fresh instances of seen tasks, and generalization on unseen tasks.
Experiments and Findings
The authors ran frozen-weight harnesses (with and without persistence) on Qwen3.5-9B and GLM-5.3 Flash, compared against two prompting baselines, a program-induction method, and Qwen3.5-9B weights distilled from GLM-5.3 Flash.
Key results:
- Persistence in the harness generally does not improve memorization unless there is explicit expert guidance.
- Across all tested models, most harness-based methods learn less than a simple baseline that caches demonstrations for each task.
- No harness-based method achieved gains on unseen tasks, while weight-based learning showed significant improvements.