What a 7-Month-Old Baby Can Do, a Standard Transformer Cannot
When you hear the sentence "Ming pushed Hong, and then she cried," you instantly know that Ming did the pushing and Hong is the one crying. This requires two steps: identifying *who did what* at the lexical level, and figuring out *who did what to whom* at the structural level.
Developmental psychology shows that infants as young as seven months can extract and generalize structural patterns like ABA or ABB from syllable sequences to entirely new syllables. Standard Transformers, however, fail at this when training data is limited.
The reason lies in the mathematics of attention. In standard self-attention, the query-key dot product simultaneously handles two jobs: encoding *relations* between tokens (who points to whom) and routing *value* vectors (what content to pass along). Relational information and object information are entangled in the same matrix multiplication — you cannot change "who did what to whom" without also changing "who is what."
It is like a kitchen with only one blender: vegetables, spices, and meat all get blended together. Stir-frying them separately is impossible.
Two Attentions, Each With Its Own Job
The core change in Relational BabyLM is splitting one attention into two:
- Self-attention routes object-level features — "what are this word's semantic features?"
- Relational attention handles structural relations — "how does this word relate to other words in context?"
The paper also introduces a Next-Latent Prediction (NextLat) training objective. Instead of having the hidden state predict the next token, NextLat forces the hidden state to compress history into a "dense belief state" — predicting the next hidden state rather than the next word. This pushes the model toward more compact compression of history instead of relying on short-range pattern matching.
RoPE as a Zero-Parameter Symbol Library
Relational attention needs a "symbol library" representing discrete structural positions. Prior work learned a trainable symbol library, adding parameters. This paper instead uses RoPE (rotary position embeddings) directly as the symbol library — zero extra parameters, performing as well as a learned library.
It is like using latitude and longitude instead of city names: coordinates are not learned, yet they pinpoint any city. RoPE's rotation encoding, originally designed for positional encoding, is repurposed as a structural symbol, saving an entire parameter matrix.
The BabyLM Challenge: Training on a Child's Data Budget
The BabyLM 2026 Challenge rules are simple: train on only 100M words of data. For comparison, GPT-3 used 500 billion words — 5000 times more. A child hears roughly 100M words by age 13. BabyLM asks: with data limited to human scale, what can models still learn?
This is the ideal setting for testing inductive biases: when data is scarce, architectural priors matter more than scale.
Results:
| Model | BLiMP | EWoK | Overall | NLP-task | |-------|-------|------|---------|----------| | GPT-2 baseline (12L) | 74.73 | 54.37 | 53.03 | 40.73 | | DAT 12L (NTP) | 79.81 | 56.66 | 54.11 | 41.39 | | DAT 18L (NextLat) | 80.62 | 56.99 | 54.65 | 43.23 | | DAT 18L (NextLat, curric.) | 78.94 | 57.25 | 55.50 | 43.86 |
On the strict track (100M words), their best model ranked 6th out of 55 systems, 3rd on the NLP-task subset, and achieved the highest EWoK score in the strict track. EWoK tests world knowledge rather than pure grammatical patterns — meaning DAT did not just learn syntax; it learned "understanding."
Architecture Is the Main Factor; Objective Is Secondary
The paper offers an honest finding: the architecture is the dominant factor; the training objective is secondary.
DAT's three relational attention variants (full RA, RCA, DisRCA) performed similarly at 10M words; only at 100M words did full RA pull ahead. This shows that *how* relational attention is implemented is not critical — what matters is separating relations from objects.
The NextLat objective does help — it improves 5 of 7 SuperGLUE tasks and better predicts the variance in human reading times — but its contribution is smaller than the architectural change.
The design lesson: get the architectural inductive biases right first, then optimize the training objective. In data-constrained settings, architectural priors matter more than training tricks.
A Deeper Implication
The paper touches a fundamental question: is standard self-attention the "right" inductive bias for Transformers?
Standard self-attention assumes that all token relations should be computed through the same query-key-value matrices. With unlimited data, this is fine — the model can learn to disentangle. With limited data, the entanglement becomes a bottleneck.
The human brain clearly does not work this way. Neuroscience shows the brain's "what" pathway (ventral stream) and "how/where" pathway (dorsal stream) are physically separated. Seven-month-olds can perform relational abstraction not because they have seen enough data, but because their brains process these two kinds of information separately from birth.
DAT encodes this biological prior into the Transformer architecture — and the results confirm that this prior helps, especially when data is scarce.
This offers an interesting counterpoint to the "scaling is all you need" narrative. When scaling hits its limits, perhaps it is time to ask: which human cognitive inductive biases have yet to be encoded in our architectures?
---
Paper: Relational Attention for Data-Efficient Language Modeling
Authors: Adrian Brasoveanu (UC Santa Cruz), Ece Takmaz (Utrecht University), Jakub Dotlačil (Utrecht University)
Venue: BabyLM 2026 Challenge