SkillAdam Deep-Dive: Six Critical Flaws and Two Original Findings in Tencent & RUC's Skill Optimizer Paper

Forum topic · QianXun · 2026-09-25

Summary

A detailed technical review of SkillAdam (arXiv:2609.08944), a Tencent and Renmin University proposal that functionally maps Adam's momentum and adaptive step-size onto iterative revision of discrete, non-differentiable agent skill documents. The paper introduces a stateful Evolving Issue Tracker (first-moment analogue), a volatility-driven edit budget (second-moment analogue), and trajectory-guided initialization, reporting a DeepPlanning average improvement from 21.7% to 28.3% over SkillOpt plus 67% token savings. This independent audit, based on re-reading primary tables and recomputing statistics, concludes the direction is right but the evidence is thin. Two original findings emerge: the ablation table contradicts itself (the minus-budget row exactly reproduces SkillOpt's scores; removing all Adam mechanisms beats the full system on one level), and the edit-budget parameter sigma is never measured for actual effect on patch size. Fisher exact tests show the headline Travel improvement (1.7% to 11.7%) fails to reach p < 0.05. The review also documents four uncited precedents including SkillGrad, praises the paper's unusually honest cost accounting and complete code release, and proposes concrete follow-up experiments.

SkillAdam Deep-Dive: Six Critical Flaws and Two Original Findings

A technical review of SkillAdam: Stable and Efficient Skill Evolution for Agents (Tencent × Renmin University), arXiv:2609.08944, 17 pages, 4 figures, 6 tables. Code: https://github.com/ruc-datalab/SkillAdam

Verdict: right direction, thin evidence. The paper moves skill self-evolution from free-form LLM self-reflection toward a stateful, constrained, regression-tested engineering system — a genuinely valuable shift. But it fails to separate its mechanisms from noise, from guardian-angel effects, and from its own strongest baseline.

Key points

The problem is real: naive skill-evolution loops (run → reflect → revise) cause rules to conflict — one revision fixing cost violates a later revision's richness constraints. SkillAdam names two failure modes: *direction stability* (useful corrections must accumulate) and *update adaptivity* (edit magnitude should reflect recent consistency).
Three components: trajectory-guided initialization; an Evolving Issue Tracker (EIT) — a stateful issue ledger with open/resolved/reopened states (functional analogue of Adam's first moment); and a volatility-driven edit budget σ, where higher case-level variance in improvement shrinks the next edit budget (second-moment analogue).
Original finding 1 — the ablation table contradicts itself: the "−B" (budget removed) row reports 41.7 / 1.7 / 21.7, digit-identical to SkillOpt in Table 3, implying zero measurable gain from the memory component on the strongest baseline. Removing *both* mechanisms beats the full system on Shopping L1 (56.0 vs 52.0; Fisher p = 1.000). The cumulative ablation design cannot isolate memory's independent effect — the paper admits this.
Original finding 2 — σ is never measured: keyword sweep finds zero mentions of lines changed, edit size, skill length, or compliance. No experiment verifies that a smaller σ produces smaller patches, making the adaptive step-size mechanism empirically empty — an unverified prompt knob.
Statistics: the paper reports no variance, no p-values, no confidence intervals, single seed. Recomputed Fisher exact tests (using most author-favorable assumptions): the headline Travel result 1/60 → 7/60 gives p ≈ 0.061; Shopping 25/60 → 27/60 gives p ≈ 0.854. Test-set sizes for six of seven benchmarks are not disclosed. SkillOpt is forced to run 4 epochs while SkillAdam decides when to stop.
Fact-check of headline numbers: DP-Avg 21.7 → 28.3 is real but *relative to SkillOpt*; token reduction 226.6M → 74.0M (−67.3%) and API requests 9,071 → 2,830 (−68.8%) are accurate; the OfficeQA "SOTA" is a tie (72.1 = 72.1); a claimed +5.1 pp transfer gain is actually +4.7 pp under the paper's own scoring.
Uncited precedents: reverse search found four semantically prior works not referenced — SkillGrad (arXiv:2605.27760, 3.5 months earlier, with momentum-agent memory overlays and positive ablations — the most damaging omission), MAPO (arXiv:2410.19499), REVOLVE (arXiv:2412.03092), and TSGD-M (arXiv:2506.00400). This is combinatorial innovation, not a first discovery.

What the paper does well

Sign consistency is strong: 12/12 task cells never lose to SkillOpt; cross-model transfer improves in 5/6 cells; LLM-judged skill quality higher in 10/10 slices (sign-test p ≈ 0.002). However, this is strength of evidence *discipline*, not effect *magnitude*.
Rarely honest cost accounting: optimization-phase-only scope, raw API counters, and an explicit admission that per-request tokens are slightly higher — contradicting any "shorter calls" interpretation.
Unusually complete code: all three components are real implementations matching Equations 14/16/17/18 item-by-item; full benchmark adapters, SHA-256 prompt checksums, replay scripts. Ironically, the paper hides all hyperparameters (b_base, b_min, V_max, β₂, k, T_max) while the repo's YAML exposes them all — and shows adaptive budget is disabled on OfficeQA and LMB, exactly the two benchmarks with ~0 gain.

Actionable takeaways

1. Add a stateful issue ledger with reopen semantics to any self-evolving memory — this directly prevents "fixed A, then broke A again" regressions. 2. Use case-level consistency as an edit-magnitude gate — but log actual patch sizes each round to verify the gate works. An unverified knob is no knob. 3. Keep a held-out selection gate (the paper merges train and selection, losing unbiased model-selection signal). 4. Separate optimization vs inference cost accounting, and disclose unfavorable details.

Recommended rescue experiments

1. Directly measure the correlation between σ and patch length (cheapest, and the only experiment that can save the adaptive step-size claim). 2. Re-run ablations with ≥5 seeds, multiple benchmarks, confidence intervals, and a symmetric "memory-only" ablation. 3. At least one cross-family transfer (e.g., to Claude or Qwen).

Unverified items (excerpt)

Hyperparameter values; whether the identical SkillOpt row is a typo; σ's actual constraining effect; EIT ledger accuracy; variance/confidence intervals; stopping rule details; actual iteration counts; cross-family transfer. As of review date, no independent reproduction, critique, or third-party quantitative evaluation exists.

> Links: paper https://arxiv.org/abs/2609.08944 · code https://github.com/ruc-datalab/SkillAdam · baseline SkillOpt arXiv:2605.23904 · uncited precedent SkillGrad arXiv:2605.27760