A 1,600-Year-Old Theory of Language Learning, Tested on a Neural Network
In 397 AD, Saint Augustine recalled in his *Confessions* how he learned to speak:
> "When they (my elders) named some object... I grasped that the thing was called by the sound they uttered."
This is one of the earliest theories of language acquisition—ostensive definition. An adult points at a banana and says "banana"; the child binds the sound to the visual impression. In 1953, Wittgenstein opened the *Philosophical Investigations* by challenging this: ostension presupposes the child already has a conceptual framework. The debate remained unresolvable for 1,600 years—until, in September 2026, Lisa Bylinina (Utrecht Institute of Language Sciences) ran a clever computational experiment: she let a DeBERTa model "see images before reading text," to test whether Augustine's theory holds inside a neural network.
Experimental Design: Giving Word Embeddings "Visual Eyes"
In standard language model training, all word embeddings are randomly initialized and learned from word co-occurrence in large text corpora. The model never "sees" a banana—it only knows the distribution of the word.
Bylinina modified this pipeline. Using the BabyLM strict-small track (~10 million words of training data, mimicking child language input), she intervened at a key point:
Before training begins, some word embeddings are initialized with visual features.
Specifically: for words appearing in visually annotated data (e.g., "banana," "red," "dog"), she extracted feature vectors from the corresponding image regions using a frozen visual encoder (SAM), averaged them, projected them into the embedding space, and wrote them into the corresponding rows of the embedding matrix. Words without visual support (e.g., "however," "democracy") kept random initialization.
Then the model trained normally on text. Visual information was injected only at initialization. The question: How long does this visual prior persist? Can standard evaluations detect it?
Finding 1: The Visual Imprint Persists Through Training
Using Representational Similarity Analysis (RSA)—which measures how the similarity structure of visual-initialized embeddings correlates with the similarity structure of their corresponding visual features:
- Visually initialized model: after 100M words of training, RSA = 0.31
- Text-only baseline: RSA = 0.10 on the same words
- Text-only baseline: 37.91
- Visual initialization: 39.63
- Visual initialization + synthetic grounding extension: 40.62 (5th place, strict-small track)
That is a huge gap. Even after 100M words of pure text training, the relative geometry of visually initialized embeddings partially preserves the structure of their visual anchors—while the baseline, trained on identical text, cannot recover this geometry from distribution alone.
More precisely: absolute positions moved almost entirely, but relative geometry was partially preserved. Each embedding drifted substantially during training, but the relative distances between words—who is near whom—retained traces of the visual initialization. It's like a city completely rebuilt where every building moved, but old neighborhoods remain neighbors. The visual prior left relational structure, not coordinates.
Finding 2: Standard Evaluations Barely Detect It
The visual imprint is clearly visible in embedding space, but standard BabyLM evaluations show almost no difference: BLiMP (grammaticality), EWoK (world knowledge), entity tracking—all show no significant gap between visual-initialized and baseline models.
The only exception is COMPS (object–property knowledge). On this task, visual initialization improved performance across all 9 encoder×vocabulary configurations (+1.37 points), with larger gains (+3.65) on the harder "novel concepts" subtask. This makes sense: COMPS tests visually perceivable properties—color, material, shape—exactly what visual initialization provides priors for.
Finding 3: VP-Swap Proves the Effect Is Causal
To verify causality, Bylinina built the VP-Swap (Visual-Property Swap) benchmark, testing color, material, size, and shape knowledge, with each item annotated for training frequency and visual initialization status.
Results: the advantage of visual initialization is precisely confined to initialized words. Uninitialized words showed no advantage, even at identical training frequencies.
The crucial causal test is synthetic grounding: for words that were never visually initialized, synthetic visual features were assigned (transferred from related words' visual features). Result: these words immediately gained the same advantage as naturally initialized ones.
This is a clean causal proof: the effect comes from visual initialization itself, not frequency, part of speech, or other confounds.
Finding 4: Abstract Words Benefit Too—But Nothing Can Measure It
The most surprising finding: function words and abstract vocabulary also received strong visual seeds and retained them through training.
How can "the," "of," "and," "democracy," or "justice" have visual seeds? Bylinina visually initialized *all* words appearing in the visual annotation data, regardless of concreteness. Function words appear in image-caption data ("the dog is on the table"), so they received visual features too.
Result: held-out mask-prediction loss for these words dropped under every random seed—the training objective itself exploits the visual prior.
But no standard evaluation detects this effect. Bylinina concludes: "What evaluation would pick this up remains an open question."
That sentence carries weight. It implies our evaluation systems have a systematic blind spot—they test "grammatical knowledge" and "world knowledge," but not "visual grounding of word meaning." A model could gain substantial semantic advantage from visual initialization, and we would see nothing on the leaderboard.
Leaderboard Competitiveness
Despite the evaluation blind spot, the model is competitive on the official BabyLM 2026 leaderboard:
This means visual initialization makes the model's acquisition order more human-like. Words with visual anchors are learned faster and earlier—just as human children learn "ball" and "dog" before "however" and "nevertheless."
What Does This Mean?
Was Augustine right? Not entirely. Wittgenstein's critique still stands—ostension requires a prior conceptual framework. But the experiment shows that even with an imperfect conceptual scaffold (visual encoder features are far from human cognition), visual initialization leaves a measurable trace. Augustine was not entirely right, but not entirely wrong either. The truth is in between.
"Visual grounding" is not a luxury—it's a structural advantage. Current LLM training skips the visual stage entirely; every word starts from random noise. This study shows that injecting visual information only at initialization (with purely textual training afterward) still yields lasting semantic advantage. Implication for multimodal design: vision–language alignment shouldn't happen only at the training-objective level, but also at embedding initialization.
Evaluation blind spots are bigger than we think. The most unsettling finding is that function and abstract words benefit undetected. We assume the BabyLM suite is comprehensive—grammar, semantics, world knowledge. Yet the visual-initialization advantage is invisible across it. This suggests current LM evaluations may systematically ignore important semantic dimensions, while we make architectural choices, tune hyperparameters, and compare models on these incomplete benchmarks.
My Take
The most brilliant aspect of this paper is not its conclusions but that it turns a 1,600-year-old philosophical debate into a computable, verifiable experiment. Bylinina doesn't claim Augustine was right—she asks a precise question: "What happens if we give a language model a visual prior?" The answer: a lot happens, but mostly we can't see it—because we're looking in the wrong direction.
This resembles the "scalar illusion" problem in AI safety—managing vectors with scalars, managing multi-dimensional capabilities with incomplete evaluations. The visual-initialization effect is a vector (grammar + world knowledge + visual semantics + abstract-word grounding), but BabyLM measures only the first two.
The most memorable sentence isn't a number but the open question: "What evaluation would pick this up?"—a question that applies to everything we assume we've already measured.
---
Paper: Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model
Author: Lisa Bylinina (Utrecht University)
Code: github.com/bylinina/augustinian_babylm
Model: huggingface.co/augustinian-babylm
Published: 2026-09-10