Phantom Gains: Auditing Self-Improvement Against a Measured Null

Summary

This paper, authored by Cheng Xu, Nan Yan, Liming Chen, and M-Tahar Kechadi (arXiv:2608.20290), examines whether language model self-improvement claims are genuine or measurement artifacts. Because self-improvement is increasingly judged by which individual questions a model gains or loses—rather than average accuracy—tracking these transitions requires differencing two noisy estimates, making it vulnerable to measurement artifacts. The authors audit three rounds of rank-32 LoRA self-training on Qwen3-8B against a frozen control run through the same pipeline. They identify seven measurement failures, each capable of reversing reported findings without the control: single-greedy-decoding ledgers fabricate capability changes on an untrained model (mostly inference batching artifacts), extended statistics conflating acquisition with sharpening yield a 0.280 ratio, and natural threshold fixes fail replication with a non-zero null under frozen comparison. Replacing these with per-question exact tests against a pooled baseline under false discovery rate control detects no results on any held-out replication. The audit finds that external distillation improves questions the base model rarely solves, while three self-training variants do not; regression rejects attributing this asymmetry as a byproduct of distillation's larger overall gains.