AI Self-Training Doesn't Flatten Language — It Restructures It: Surface Markers Amplify While Deep Syntax Dies

Paper Overview

| Item | Detail | |------|--------| | Title | Self-Training Doesn't Flatten Language — It Restructures It: Surface Markers Amplify While Deep Syntax Dies | | Author | Ming Liu (Amazon) | | arXiv | 2605.20602 | | Date | 2026-05-20 | | Category | cs.CL (Computational Linguistics) | | Core claim | Self-training does not uniformly 'flatten' language; it structurally reorganizes it: surface markers (however, perhaps, dashes) amplify while deep syntactic structures (questions, passive voice, subjunctive) systematically die. This asymmetric collapse is predicted by a unified theory — the Structural Depth Hypothesis (SDH). |

Self-training — generating text with a model, fine-tuning on that output, and repeating — is increasingly common as high-quality human data becomes scarce. The prevailing intuition, reinforced by the famous 2024 Nature model-collapse paper, is that this loop leads to 'flattening': language grows duller and more homogeneous.

This paper says that is wrong. The process is not flattening — it is something stranger: structural reorganization.

Experimental Setup

The author ran 11 generations of self-training (generate → fine-tune → generate) on five models: GPT-2 (124M), three Pythia sizes (410M, 1.4B, 2.8B), and OPT (1.3B), with fixed decoding parameters and prompts throughout. 17 linguistic features — pre-registered before the experiment — were tracked per generation, stratified by 'structural depth':

• Depth 0: surface markers — discourse connectives (however, moreover, therefore), hedges (perhaps, maybe), dashes
• Depth 1: local syntax — regular past tense, sentence-initial conjunctions, coordination
• Depth 2: clausal structures — questions, passive voice, parentheticals, irregular past tense, relative clauses
• Depth 3: cross-clausal/mood — subjunctive

Core Findings: Two Destinies

By GPT-2 generation 10:

Amplifying:

• Discourse connectives: +126.2%
• Hedges: +44.2%
• Dashes: +28.6%
• Sentence-initial conjunctions: +19.0%

Dying:

• Question marks: -91.7%
• Colons: -64.8%
• Semicolons: -64.4%
• Parentheticals: -56.8%
• Passive voice: -55.5%
• Subjunctive: -52.7%
• Irregular past tense: -52.3%

Group averages decline monotonically with depth: d=0 +24.9%, d=1 -10.0%, d=2 -47.2%, d=3 -52.7%. Generation-10 text *looks* formal and academic — full of 'however' and 'perhaps' — but it no longer asks questions, uses passive voice, inserts parentheticals, or entertains hypotheticals.

Structural Depth Hypothesis

SDH: a feature's decay rate is governed primarily by its structural depth (how many levels of syntactic dependency must be nested to use it legitimately), not its frequency. This contradicts classic collapse theory, which predicts rare features die fastest.

Verification via Spearman rank correlations across the 17 features:

• Depth vs. decay rate: rho = 0.540 (p < 10^-6)
• Frequency vs. decay rate: rho = 0.225 (marginal)
• Partial correlation controlling for frequency: rho_partial = 0.490 (p < 10^-6)

Depth explains 29–44% of decay variance. The pattern holds across all five models, and larger models show a steeper depth gradient (Pythia-2.8B: rho = 0.705).

Key Paired Comparisons

• Regular vs. irregular past tense: same grammatical category, similar frequency. Regular (d=1) rose +79.7%; irregular (d=2) fell -52.3%. Frequency theory cannot explain this divergence; depth can.
• Dashes vs. parentheses: both mark asides. Dashes (d=0) rose +28.6%; parentheses (d=2) fell -56.8%.
• Sentence-initial vs. mid-sentence coordination: the freer form rose +19.0%; the more constrained form fell -14.4%.

The Surface Complexity Paradox

Using standard aggregate complexity proxies, everything looks *richer* at generation 10:

| Metric | Change | |--------|--------| | Dependency tree depth | +45.5% | | Clausal nesting | +33.4% | | Mean word length | +25.3% | | Lexical diversity (TTR) | +9.8% |

But this is an illusion: the longer sentences and deeper dependency trees are propped up by chains of discourse connectives. Current AI-text detectors relying on these aggregate metrics would be misled after multiple self-training generations. Depth-stratified feature panels are needed instead.

The Exception That Proves the Rule

Exclamation points (d=0) collapsed -99.3% — apparently contradicting SDH. The resolution: SDH's full form includes a sampling-dependency term. Connectives and hedges have high sampling dependence and sufficient baseline frequency to enter a 'rich-get-richer' loop; exclamation marks' baseline was too low (1.03 per 1,000 tokens) for the loop to start. Only the joint equation explains why connectives and exclamation marks — both rare, both d=0 — meet opposite fates.

Implications

• AI-text detection: the ubiquity of 'however'/'furthermore' in AI text reflects survival of surface markers, not learned politeness. Aggregate metrics fail after multi-generation self-training.
• Data curation: when training on synthetic text, deliberately oversample human text rich in deep syntax — literary fiction, legal text, scientific prose.
• Model collapse theory: self-training performs *structural selection* — the criterion is structural depth, not frequency.

Honest Limitations

• Internal heterogeneity at d=2: the depth-2 tier shows the largest internal variance across models; finer-grained depth definitions are needed.
• Cross-model variance: only 10 of 17 features had 95% CIs excluding zero; several features can move in opposite directions across models. The bootstrap may underestimate standard errors.
• English only: structural depth may be defined very differently in other languages (e.g., Mandarin morphology).
• Scale ceiling: largest model was 2.8B; behavior at 70B+ is unknown.
• Causal mechanism open: SDH is descriptive. Why do autoregressive models systematically under-sample deep structures — probabilistic multiplication, or bias in training data? Untested.
• Decoding parameters: fixed at top-p=0.95, T=0.9; the depth gradient is robust to other schemes, but coefficient magnitudes change.

References

1. Liu, M. (2026). *Self-Training Doesn't Flatten Language — It Restructures It: Surface Markers Amplify While Deep Syntax Dies.* arXiv:2605.20602. 2. Shumailov, I. et al. (2024). *AI models collapse when trained on recursively generated data.* Nature, 631, 755-759. 3. Dohmatob, E. et al. (2024). *A Tale of Tails: Model Collapse as a change of scaling laws.* arXiv:2402.07043. 4. Guo, Y. et al. (2024). *The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text.* NAACL 2024 Findings. 5. Kobak, D. et al. (2025). *Delving into ChatGPT usage in scientific writing through excess vocabulary.* arXiv:2505.13478.