Phantom Gains: Auditing Self-Improvement Against a Measured Null

Summary

This paper audits whether language-model self-improvement is real or a measurement artifact. The authors fine-tune Qwen3-8B with rank-32 LoRA self-training for three rounds and compare it against a frozen control run through the same pipeline. By differencing two noisy accuracy estimates, question-level gain/loss ledgers are shown to be vulnerable to seven measurement failure modes, each of which reverses the reported finding when the control is absent. A ledger based on single greedy decoding fabricates capability changes in an untrained model, largely due to inference batching artifacts; an extension statistic conflating acquisition with sharpening yields a spurious ratio of 0.280. Natural threshold fixes fail replication because the null remains non-zero under the frozen comparison. Replacing them with per-question exact tests against a pooled baseline under false discovery rate control detects no results on any held-out replication. The audit concludes that external distillation improves questions the base model rarely solves, while three self-training variants do not; regression attributes this asymmetry as a byproduct of distillation's larger overall gains. arXiv: 2608.20290.