Source post from zhichai.net forum: a critical read of the Chain-of-Experience paper (arXiv:2608.18027v1, posted 2026-08-18), a UC Santa Cruz × ByteDance Seed collaboration with co-first authors Haoqin Tu and Yunhao Fang, senior authors Cihang Xie and Shen Yan.
One-sentence takeaway: don't rush to fine-tune the model — let it walk in with a mistake notebook.
Mechanism: Don't Throw Away the Scratch Paper
Traditional test-time scaling discards drafts: Self-Consistency compresses many reasoning traces into one answer, losing all the detours. Self-Refine and Reflexion internalize feedback within a single context, which the paper criticizes as "fragmented and shallow."
CoE instead records a binary tuple \(e = (answer, feedback)\) per iteration and keeps the entire history in context. The next-round generation conditions on the full chain:
Zero weight updates — pure in-context reuse. This is orthogonal to cross-task "cheat sheet" methods like Dynamic CheatSheet / ACE: CoE is intra-task with full trajectories; DC/ACE are cross-task lossy compressions. The "Continual" in the title does not mean continual learning — parameters never move.
Four Feedback Tiers; the Ceiling Is Feedback Quality
1. none: no feedback, self-reflection only. 66.8% average across six benchmarks. 2. model feedback: an LLM judge gives textual critiques (may be the same model). 71.0%, +5.6% over no feedback. 3. executor feedback: run code, read errors, check test pass rates. Coding tasks 66.4→75.0. 4. correctness feedback: direct right/wrong signal, requires ground truth. 79.3%, +11.1% — but this is an oracle; the paper itself admits it is "often unavailable in reality" and positions it as an upper bound.
Setup: six benchmarks (AIME 2025, OmniMath, LiveCodeBench V6, LiveBench Code, EvaLearn, GPQA Diamond); closed models including GPT-5, GPT-5-mini, o4-mini, o3, o3-mini, Gemini-2.5 Pro, Claude 4.5 Sonnet; default 20 iterations with best-of selection; 3 runs per experiment.
Hard findings:
- Gains come early: +16.7% on AIME in the first 20 iterations on average; only +2.2% more from iterations 20–50. Don't over-budget iterations.
- Self-feedback even saves money: $4.6 vs $8.8 on AIME (no feedback), with +4.4 points. Token counts across the three multi-turn methods differ by less than 2K — gains come from reallocating compute into feedback iterations, not longer prompts.
- Robust to fake feedback: constant "all-correct" or "all-wrong" signals degrade AIME by only 7.6% on average; stronger models are more robust (GPT-5 mini drops only 2.5%). All-wrong feedback first dips then recovers — "constructive noise can stimulate deeper reasoning."
- Attribution: among 6,630 error→correct samples, 47.7% are feedback-driven; coding adds 30.0% spec-retrospection (re-aligning to problem requirements). Human-model agreement κ=0.768.
- Base ability ↔ improvement positively correlated: average Pearson +0.50 across five benchmarks, up to 0.97 on LiveBench Code. The paper calls learning-from-experience "an emergent property that scales with model capability" — take that half-sentence with a discount (see below).
- ✅ Safe to cite: self-feedback test-time iteration gives +5.6%, gains concentrate in the first 20 iterations, feedback quality ∝ verifier capability.
- ⚠️ Cite with caveats: +11.1% must be labeled an oracle upper bound; "outperforms DC/ACE" must note the 10x token gap.
- ❌ Don't cite: SelMV "universal improvements" (AIME actually drops under normal feedback); "effective across all tasks and models" (two built-in counterexamples).
The Accounting Needs Scrutiny: Critique Pass
The paper is more honest than most peers — oracle clearly labeled as upper bound, negative results reported. But four items deserve discounting:
1. The +11.1% hangs on a perfect grader. Nearly half the gap between self-feedback's 5.6% and the oracle's 11.1% depends on a grader that doesn't exist in practice. Correctness feedback pushes GPQA to 99.52% — best-of-20 plus per-round grading approximates repeated enumeration against the answer space. That's an open-book exam, not "learning from experience."
2. "Beats DC/ACE" is a victory at 10x the compute. Dynamic CheatSheet reaches 74.7% on AIME with 11,233 tokens; no-feedback CoE gets 74.1% with 106,825 tokens. At equal compute, DC doesn't lose. Moreover DC still underperforms even when curation uses ground-truth answers, hinting the baseline's failure may be an adaptation issue — no ablation rules this out.
3. "Emergent property" evidence is thin. Each correlation has n≈8 models, 5 of them from the OpenAI family; points like r=0.33 and 0.24 are highly unstable at n=8; no p-values or confidence intervals anywhere. Math correlations are actually weak; the pretty 0.97 comes from a single benchmark. Confounders — release date, reasoning-training intensity — go undiscussed.
4. A built-in counterexample is hidden in the appendix. On BrowseComp-Plus (deep retrieval), most models do worse with self-feedback than without — self-critique can't patch knowledge gaps. Gemini-2.5 Pro is the only model that doesn't save money. The main text's "across all tasks and models" contradicts its own data.
Minor errata noted: the abstract says 8 models, the main text lists 7; attribution agreement is 76.8% in text vs 76.0% in Table 4 (κ likely mislabeled as agreement); ACE mean is 64.0% in text vs 63.0% recomputed from tables.
How to Cite
Closing Nail
Two gaps deserve watching: first, open-weight models' experience-utilization ability — "emergence" should be tested on Qwen / DeepSeek, but this paper is all closed API models; second, weight internalization — the paper's own limitations section admits that distilling experience chains back into parameters is the real continual learning, where it would squarely meet training frameworks like AgentEvolver.
Verification point to bookmark: if a CoE + open-weight reproduction appears within three months (especially at the 7B scale) and the +5.6% holds, the "experience chain" stands; if not, it's merely a footnote to frontier models' context-window dividend.