Artifact-to-Capability: Three Self-Evolving Agents Fail on a Gomoku Board Despite Green Signals (arXiv 2611.04952)

Forum topic · 小凯 · 2026-11-04

Summary

A paper from Tianjin University and IDEA (arXiv 2611.04952) asks a foundational question about self-evolving agents: what do the standard proxy signals of progress actually prove? Most evaluations rely on three signals—artifacts accumulate, behavior changes when loaded, and metrics rise. The authors define capability formation as five sequentially dependent properties, with task-relation realization and boundary-safe reuse forming the checkpoint, and introduce a three-scale diagnosis (state, game, and lineage levels). In freestyle Gomoku, a controlled environment where board states are replayable and move quality is judged by rules, they audit Reflexion, Voyager, and DGM. All three systems show all three green signals, yet each breaks at a different link in the capability chain: Reflexion exhibits legalization gain (only more moves become legal, move selection does not improve); Voyager exhibits compositional override (individually sound components combine to miss all 20 available one-move wins, losing 0/20 games); DGM exhibits loss-only drift (after passing the checkpoint, later versions only lose passing states while the evolution score keeps rising). The paper concludes that signal-based evaluation cannot determine whether a capability has formed, and offers a practical checklist: decompose metrics, run adversarial combination tests, and track lineage-level gains versus losses.

Overview

Field: cs.AI / self-evolving agents / evaluation methodology Authors: Ziqian Ban, Haoxiang Yang, Yanfei Zhang, Shixin Liang, Haoran Yu, Xu Lin, Shaobin Zhao, Xuran Li, Zhiyi Cheng (Tianjin University, IDEA) arXiv: 2611.04952

Key points

Self-evolving agents write interaction experience into "artifacts" (textual reflections, executable skills, code) and reuse them in later tasks. Almost all work in this area supports effectiveness with three proxy signals:

Artifacts accumulate
Behavior changes once artifacts are loaded
Metrics rise

The paper asks: what are these signals meant to indicate? The authors formalize capability formation from artifacts as five sequentially dependent properties:

1. Artifact formation — a persistent structure exists and is loaded into decisions. 2. Behavioral activation — with vs. without the artifact, moves differ beyond sampling noise. 3. Task-relation realization — the change produces *positive* task effects among legal moves (legality itself is an environment rule, not a task gain). 4. Boundary-safe reuse — no other part of the system overturns an established positive relation. 5. Capability extension — later versions add positive effects while retaining existing ones.

Properties 3 and 4 together form the checkpoint for capability formation. Diagnosis proceeds in order; the first failed property is the break point.

The three-scale diagnosis in freestyle Gomoku

Gomoku is chosen because it offers state replayability, rule-based independent judging of move quality, and full logging of games and versions. The diagnosis runs at three scales:

State level: 1,876 fixed positions (120 handcrafted tactical positions plus 1,756 real positions from DGM's evolution), comparing with-artifact vs. without-artifact moves against a rule-defined acceptable set $$G(s)$$ .
Game level: the frozen full system plays 20 games against a strong programmatic opponent (search_2), checking whether all immediate-win positions are cashed in.
Lineage level: adjacent versions are paired on the same positions, counting newly added vs. lost passing states.

All three systems share the same base model (DeepSeek-V4-Flash), environment, and opponents.

Three findings, three distinct breaks

Reflexion — break at property 3: legalization gain

Pass rate rose from 8.0% to 9.2%–10.3%, but decomposition shows:

P(\text{pass}) = P(\text{legal}) \times P(\text{pass} \mid \text{legal})

The gain comes entirely from a higher legality rate. Conditional pass rate among legal moves did not improve in any of the 12 memory×position-set combinations, and declined 1.6–4.4 points on the larger set. Memory taught the model to make legal moves, not better moves.

Voyager — break at property 4: compositional override

State-level tests were green; the live-four threat rose from 0/20 to 20/20. Yet in all 20 games, Voyager reached an immediate-win position and cashed in zero of them, losing every game. The culprit is the auto_player script's fixed-order, defense-first early return: in conflict positions (both sides have immediate wins), the defensive branch fires first and returns, so the win-when-possible logic (correct on its own) is never executed. Each component passes individual testing; the combination systematically forfeits wins.

DGM — break at property 5: loss-only drift

Version OB-3 genuinely passed the full checkpoint: net +383 passing states, 19/20 immediate wins cashed in, 19/20 game wins. But subsequent versions only lost capability: OB-5 added 0 / lost 103 (9/20 wins); OB-6 added 0 / lost 29 (0/20 wins); OB-7 barely changed behavior at all — while the evolution score rose from 150.4 to 312.6. The final metric measures internal activity, not capability.

Implications

Signal-based evaluation cannot determine whether a capability has formed. None of the three signals can substitute for the capability chain; each break type is invisible to a different signal.
Compositional override is a warning for agent engineering: as skill libraries, tool chains, and sub-agents multiply, "every part correct, system broken" will become the norm. Diagnosis should target the frozen system as a whole, not individual components.
The methodology is transferable: any environment with replayable states and rule-adjudicable effects (theorem proving, database operations, GUI agents) can host this microscope. The paper sketches a GUI-agent scenario where a popup-closing skill overrides a navigation skill — another compositional override.

Practitioner checklist

Decompose your metrics (Reflexion's lesson): split success into $P(\text{valid}) \times P(\text{good} \mid \text{valid})$ ; formatting and validity gains can mask zero or negative choice quality.
Run adversarial combination tests (Voyager's lesson): construct conflict states where two components trigger with contradictory outputs and check the arbitration logic.
Track lineage gains and losses (DGM's lesson): for every version, count newly added vs. lost passing states on a fixed position set; a version that only loses capability is regressing regardless of its aggregate score.

Limitations

Only three representative mechanisms were tested; multi-agent co-evolution and direct parameter updates may break elsewhere.
Only Gomoku was used; operationalizing "task relation" in open-ended tasks remains open (the authors flag this as conjecture).
A single base model; break locations may shift across models and longer evolution runs.
Only DGM reached property 5, so loss-only drift cannot yet be generalized to code-evolution mechanisms.

Reference

Ban Z., Yang H., Zhang Y., et al. "Artifact-to-Capability: Diagnosing Breaks in Capability Formation of Self-Evolving Agents in a Controlled Environment." arXiv:2611.04952