Learning Music Before Language: An Unexpected Finding on Transformer Structural Priors
> Structural priors for data-efficient language learning > arXiv: 2609.11505 | Veitsman, Martins, Lautenschlager, Beinborn > Pretraining on music, probabilistic grammars, and cellular automata, then teaching the model language
---
A Seemingly Absurd Experiment
Suppose you want to train a language model but don't have enough language data. The conventional approaches: scrape more text, or just train from random initialization.
This group of researchers took a different tack: first teach the model something else—music, probabilistic grammars, cellular automata—and then let it learn language.
It sounds like nonsense. Why would a Transformer that has learned Bach's Well-Tempered Clavier learn Chinese better?
But the experimental results were surprising: these "non-linguistic" datasets genuinely helped the model learn language better afterward. At least on the next-token prediction metric, the effect is real.
---
Experimental Design: Structure Instead of Data
The core hypothesis: some non-linguistic data, while not "language," contains structural regularities similar to those in language. If the model learns these structures first, it starts from a better position when learning language.
They tested four types of "non-linguistic" data:
- Music (MIDI-encoded melody sequences)
- Probabilistic grammars (symbol sequences generated to follow specific grammatical rules)
- Cellular automata (spatiotemporal pattern sequences, e.g., from Conway's Game of Life)
- Random sequences (control baseline)
- Music has motif development and tonal return
- Probabilistic grammars have recursive structure and long-range dependencies
- Cellular automata produce global patterns from local rules
- Next-token prediction is a structural task ("what distribution should the next token follow") → transferable
- Downstream linguistic tasks are semantic tasks ("what does this sentence mean") → not transferable
Pipeline: pretrain a Transformer on these data, fine-tune on language data, then compare next-token prediction loss, weight change magnitude, and downstream linguistic benchmark performance.
---
Three Key Findings
Finding 1: Music, Grammars, and Cellular Automata Do Help
Models pretrained on music, probabilistic grammars, or cellular automata achieved lower next-token prediction loss when subsequently learning language than randomly initialized models.
This means: the model learned something from these non-linguistic data, and that something helped when learning language.
The researchers verified this using "weight displacement": models starting from non-linguistic pretraining showed smaller weight changes while learning language. From an optimization perspective, the non-linguistic data places the model in a region of parameter space that is "closer to the language solution"—it doesn't have to travel as far to find linguistic regularities.
Finding 2: But Downstream Linguistic Tasks Didn't Improve
Here comes the paper's most honest twist: although next-token prediction loss was lower, the models' performance on downstream linguistic benchmarks did not systematically improve.
This is an important distinction. Next-token prediction is a relatively "shallow" task—it measures the probability distribution of the next token given context. Downstream linguistic tasks (syntactic parsing, semantic role labeling, question answering) probe deeper linguistic competence.
The model learned "statistical regularities of language," but not "structural linguistic capabilities." These are not the same thing.
Finding 3: Non-Linguistic Data Is Worse Than an Equal Amount of Language Data
Pretraining on an equal volume of actual language data yields better results than non-linguistic data. Non-linguistic data is a *partial* substitute for language data, not an *equivalent* substitute.
This conclusion may seem to negate the value of non-linguistic data, but the researchers emphasize: in scenarios of extreme language-data scarcity (e.g., low-resource languages), "partial substitution" still has value.
---
Why Does Music Help? A Structural Isomorphism Hypothesis
The paper doesn't give a definitive answer, but we can speculate: music, probabilistic grammars, and cellular automata all share certain "structural features"—repetition, variation, and hierarchical nesting.
Language shares these features: syntactic trees are recursive, discourse has anaphora and coreference, words have morphology. If a model first learns "how to recognize and generate hierarchical patterns" on purer structural data, it doesn't need to learn these structures from scratch when facing language.
But this hypothesis also explains why downstream tasks didn't improve: language is not just structure. Language has semantics, world knowledge, and pragmatic inference. Structural priors help with "form," not "content."
---
A Deeper Insight: What Transfers and What Doesn't
The most memorable point of this paper is not "music helps models learn language" but a finer distinction:
Structure transfers; semantics does not.
---
Connections to Existing Concepts
The findings fit into an "evaluation blind spot law": next-token prediction loss, as a metric, cannot distinguish "structural competence" from "semantic competence." Two models with the same loss might differ dramatically—one may have learned only structure, the other both—but the loss numbers look identical.
It also contrasts with the "positional encoding paradox": strong inductive biases like positional encodings work in-distribution but become shackles out-of-distribution, whereas "weak structural priors" like music/grammar help only modestly in-distribution but become valuable in data-scarce (out-of-distribution) settings. Strong priors lock in a path; weak priors leave room.
---
An Open Question
The paper leaves a question worth pondering: if non-linguistic data only helps next-token prediction and not downstream tasks, what does "learned structure" actually mean?
One possible answer: what the model learns is not "structure" itself but "how to quickly adapt to new sequence distributions." Music teaches the model the meta-cognition that "sequences have patterns," but the specific regularities of language still need to be filled in by language data.
In other words, non-linguistic data doesn't teach the model "the structure of language"—it teaches "the existence of structure." This meta-level learning may be more important than any specific structure.
---
Paper link: arxiv.org/abs/2609.11505
Code: No open-source repository provided by the paper