Phantom Gains: Auditing Self-Improvement Against a Measured Null (arXiv 2608.20290)

Summary

This paper, 'Phantom Gains: Auditing Self-Improvement Against a Measured Null' by Cheng Xu, Nan Yan, Liming Chen, and M-Tahar Kechadi (arXiv:2608.20290, released 2026-08-22, NLP), audits claims of language-model self-improvement. Rather than judging improvement by average accuracy, it tracks which individual questions a model gains or loses, differencing two noisy estimates—a procedure vulnerable to measurement artifacts. The authors audit three rounds of rank-32 LoRA self-training on Qwen3-8B against a frozen control run through the identical pipeline, identifying seven measurement failures, each capable of reversing the reported finding when the control is absent. Single-pass greedy decoding bookkeeping fabricates capability changes on an untrained model, largely inference-batching artifacts; an extension statistic distinguishing acquisition from sharpening yields a ratio of 0.280. Natural threshold fixes fail replication because their null remains nonzero under the frozen comparison. Replacing them with per-question exact tests against a pooled baseline under false discovery rate control detects no effects on any held-out replication. The audit finds external distillation improves problems the base model rarely solves, while three forms of self-training do not; regression attributes this asymmetry as a byproduct of distillation's larger overall gains.