Overview
Paper: arXiv: 2609.11870 | Lisa Bylinina | BabyLM Workshop 2026
In 397 CE, Saint Augustine described in his *Confessions* how he learned language: adults pointed at objects while naming them, and he associated sounds with things — the idea philosophers later called ostensive definition. This post discusses an experiment that directly implements that 1,600-year-old theory in a modern language model.
Experimental Design
- Model: DeBERTa (masked language model), trained on a 10M-word BabyLM-scale corpus.
- Before training, embeddings of a subset of concrete words (e.g., "dog", "red", "big") are replaced with visual feature vectors extracted from image region annotations.
- Function words and abstract words keep random initialization.
- Training proceeds normally; the "visually seeded" model is compared against a purely random-init baseline across benchmarks.
- The seed leaves a persistent imprint. After training, seeded words' embeddings retain detectable traces of visual features in the embedding space.
- But most BabyLM benchmarks detect no difference. Abstract syntactic tasks (subject-verb agreement, island constraints, binding theory) show no benefit.
- The exception: COMPS (Misra et al. 2023), testing object–property knowledge (e.g., "bananas are yellow"). Visual seeding gives a zero-shot advantage in every configuration.
- A custom Visual-Property Swap benchmark (color, material, size, shape), controlling word frequency and seeding status, confirms: seeding yields a persistent, reproducible advantage only on seeded words.
- Paper: https://arxiv.org/abs/2609.11870
- Code: not yet released
- Related benchmarks: BabyLM Challenge, COMPS, Visual-Property Swap
Results
Synthetic Seeding Transfer (Causal Validation)
The author artificially gave synthetic visual embeddings to originally unseeded words. These "late-seeded" words gained the same advantage — showing the benefit comes from the seeding operation itself, not from any word being inherently easier to learn.
The Ghost of Function Words
A deeper finding: function and abstract words also ended up with visual seeds (indirectly) and retained them throughout training. Masked-prediction loss for seeded words was lower in every seed configuration — the training objective itself exploits the seeds.
> The model "knows" these words are different. But no existing benchmark can detect it. As the paper concludes: *what kind of evaluation could capture this signal remains an open question.*
Why It Matters
The paper's value is not that visual seeding makes models stronger — the effect is modest and task-limited. Its insight is an evaluation problem:
A model can retain visual priors internally, exploit them in its training objective, and still be indistinguishable on standard benchmarks from a randomly initialized model. Our evaluation methods may systematically miss what actually happens inside models.
Comment
The author of the post adds: many debates about "what models learn" are really debates about "how we measure learning." Model internals are richer than benchmark scores suggest — we need smarter probes, not just bigger models. Also notable: translating a 1,600-year-old philosophical theory directly into an experimental design shows philosophy can generate experiments, not just papers.
Links: