Learning Music Before Language: An Unexpected Finding About Transformer Structural Priors
> Structural priors for data-efficient language learning > arXiv: 2609.11505 | Veitsman, Martins, Lautenschlager, Beinborn > Pretraining on music, probabilistic grammars, and cellular automata, then learning language
---
A Seemingly Absurd Experiment
Suppose you want to train a language model but lack sufficient language data. The standard approaches: scrounge up more text, or just train from random initialization.
This research group tried something different: first teach the model something else — music, probabilistic grammars, cellular automata — then let it learn language.
It sounds like nonsense. Why would a Transformer that has studied Bach's Well-Tempered Clavier learn Chinese better?
Yet the experimental results were surprising: these "non-language" datasets did help models learn language better afterward. At least on the next-token prediction metric, the effect is real.
---
Experimental Design: Substituting Structure for Data
The core hypothesis: certain non-linguistic data, while not "language," contains structural regularities similar to those in language. If the model learns these structures first, it starts from a better position when learning language.
They tested four kinds of "non-language" data:
- Music (MIDI-encoded melody sequences)
- Probabilistic grammars (symbol sequences generated according to specific grammatical rules)
- Cellular automata (spatiotemporal pattern sequences, e.g., from Conway's Game of Life)
- Random sequences (as a control baseline)
- Music has motif development and tonal return
- Probabilistic grammars have recursive structure and long-range dependencies
- Cellular automata produce global patterns from local rules
- Next-token prediction is a structural task ("what distribution should the next token follow") → transferable
- Downstream linguistic tasks are semantic tasks ("what does this sentence mean") → not transferable
The pipeline: pretrain a Transformer on these data, fine-tune on language, then compare next-token prediction loss, weight change magnitude, and downstream linguistic benchmark performance.
---
Three Key Findings
Finding 1: Music, Grammars, and Cellular Automata Do Help
Models pretrained on music, probabilistic grammars, or cellular automata achieved lower next-token prediction loss when subsequently learning language than randomly initialized models.
This means: the model learned some "something" from non-linguistic data, and that something helped when learning language.
The authors used "weight displacement" to verify this: models starting from non-linguistic pretraining changed their weights less while learning language. From an optimization perspective, the non-linguistic data placed the model in a region of parameter space "closer to the language solution" — it didn't need to travel as far to find linguistic regularities.
Finding 2: But Downstream Linguistic Tasks Didn't Improve
Here comes the paper's most honest twist: although next-token prediction loss was lower, performance on downstream linguistic benchmarks did not systematically improve.
This is an important distinction. Next-token prediction is a relatively "shallow" task — it measures the probability distribution of the next token given context. Downstream linguistic tasks (syntactic parsing, semantic role labeling, question answering) test deeper language competence.
The model learned "statistical regularities of language," but not "structural capabilities of language." These are not the same thing.
Finding 3: Non-Linguistic Data Is Worse Than Equivalent Language Data
Pretraining on the same amount of actual language data outperforms non-linguistic data. Non-linguistic data "partially substitutes" for language data, not "equivalently substitutes."
This conclusion may seem to dismiss the value of non-linguistic data, but the researchers emphasize: in scenarios with extremely scarce language data (e.g., low-resource languages), "partial substitution" still matters.
---
Why Does Music Help? A Structural Isomorphism Hypothesis
The paper doesn't give a definitive answer, but we can speculate: music, probabilistic grammars, and cellular automata share certain "structural features" — repetition, variation, and hierarchically nested patterns.
Language shares these features: syntax trees are recursive, discourse has anaphora and coreference, words undergo morphological change. If the model first learns "how to recognize and generate hierarchical patterns" on purer structural data, then facing language, it doesn't need to learn these structures from scratch.
But this hypothesis also explains why downstream tasks didn't improve: language is more than structure. Language has semantics, world knowledge, and pragmatic inference. Structural priors help with "form," not "content."
---
A Deeper Insight: What Transfers, and What Doesn't
The most memorable takeaway isn't "music helps models learn language," but a finer distinction:
Structure transfers; semantics doesn't.
---
Connections to Existing Concepts
The paper's findings fit into what might be called the "evaluation blind-spot law": the next-token prediction loss metric cannot distinguish "structural capability" from "semantic capability." Two models with the same loss might differ — one learned only structure, the other both — yet the loss numbers look identical.
This also contrasts with the "positional encoding paradox": positional encoding, a strong inductive bias, works in-distribution but becomes a shackle out-of-distribution; whereas music/grammar "weak structural priors" help little in-distribution but prove valuable in data-scarce (out-of-distribution) settings. Strong priors lock in paths; weak priors leave room.
---
An Open Question
One question from this paper keeps coming back to me: if non-linguistic data helps next-token prediction but not downstream tasks, what does "learning structure" actually mean?
One possible answer: what the model learns isn't "structure" itself, but "how to quickly adapt to new sequence distributions." Music teaches the model the metacognition that "sequences have discoverable regularities," but the specific regularities of language still need to be filled in by language data.
In other words, what non-linguistic data teaches the model isn't "the structure of language," but "the existence of structure." This meta-level learning may matter more than any specific structure.
---
Paper link: arxiv.org/abs/2609.11505
Code: No open-source repository provided by the paper