The task a 7-month-old can do, but a Transformer can't
Imagine hearing: "Ming pushed Hong, and then she cried." You instantly know who pushed and who cried. That judgment takes two steps: identifying who did what (object-level, lexical information), and who did what to whom (structure-level, relational information).
Developmental psychology shows seven-month-old infants can abstract ABA/ABB structural patterns from syllable sequences and generalize them to new syllables. Standard Transformers, at least with limited data, cannot.
The reason lies in the math of attention. In standard self-attention, the query-key dot product serves two roles simultaneously: encoding relations between tokens (who points to whom) and routing value vectors (what content gets passed). Relational and object information are mixed into the same matrix multiplication — you can't change "who did what to whom" without also changing "what is what." Like a kitchen with only one blender: vegetables, spices, and meat all blended together.
Two attentions, each with its own job
The core change in Relational BabyLM: split one attention into two.
- Self-attention routes object-level features — "what are this word's semantic features."
- Relational attention handles structural relations — "how does this word relate to other words in context."
The paper also introduces Next-Latent Prediction (NextLat). Instead of next-token prediction, hidden states compress history into a "dense belief state" and predict the next hidden state — forcing tighter compression of history rather than relying on short-range pattern matching.
RoPE as a zero-parameter symbol library
Relational attention needs a "symbol library" for discrete structural positions. Previous work learned a trainable symbol library — extra parameters. This paper uses RoPE (rotary position embeddings) directly as the symbol library, with zero extra parameters, matching the performance of learned libraries.
It's like using latitude-longitude coordinates instead of city names — not learned, but precisely locates any city. RoPE's rotations, designed for positions, are repurposed as structural symbols, saving an entire parameter matrix.
BabyLM: training on a child's data budget
The BabyLM 2026 Challenge limits training to 100M words. For context: GPT-3 used 500 billion words — 5000x more. A child hears roughly 100M words by age 13. With data capped at human scale, architectural priors matter more than data volume.
Results (strict track, 100M words):
| Model | BLiMP | EWoK | Overall | NLP-task | |-------|-------|------|---------|----------| | GPT-2 baseline (12L) | 74.73 | 54.37 | 53.03 | 40.73 | | DAT 12L (NTP) | 79.81 | 56.66 | 54.11 | 41.39 | | DAT 18L (NextLat) | 80.62 | 56.99 | 54.65 | 43.23 | | DAT 18L (NextLat, curric.) | 78.94 | 57.25 | 55.50 | 43.86 |
Their best model ranked 6th of 55 systems overall, 3rd on NLP tasks, and achieved the highest EWoK score on the strict track. EWoK tests world knowledge, not just grammar — meaning DAT didn't just learn syntax, it "understood."
Architecture first, objective second
An honest finding: architecture is the dominant factor; the training objective is secondary.
The three relational attention variants (full RA, RCA, DisRCA) performed similarly at 10M words; only at 100M did full RA pull ahead. What matters isn't the specific relational attention implementation — it's whether relations and objects are separated at all.
NextLat did help — improving 5 of 7 SuperGLUE tasks and better predicting variance in human reading times — but its contribution was smaller than the architectural change.
Design lesson: get the architectural inductive biases right first, then optimize the training objective. In data-constrained settings, priors beat tricks.
A deeper implication
The paper touches a fundamental question: is standard self-attention the "right" inductive bias?
Standard attention assumes all token relations should flow through one query-key-value matrix. With unlimited data, the model can learn to disentangle; with limited data, the entanglement is a bottleneck.
Human brains don't work this way. Neuroscience shows the ventral stream ("what") and dorsal stream ("how/where") are physically separated. Infants do relational abstraction not because they've seen enough data, but because their brains separate these information types from birth.
DAT encodes this biological prior into a Transformer — and it works, especially when data is scarce. As the "scaling is all you need" narrative hits limits, perhaps it's time to ask: which human cognitive inductive biases haven't yet been encoded into our architectures?
---
Paper: Relational Attention for Data-Efficient Language Modeling
Authors: Adrian Brasoveanu (UC Santa Cruz), Ece Takmaz (Utrecht University), Jakub Dotlačil (Utrecht University)
Venue: BabyLM 2026 Challenge