Ostensive Definition Revived: Visual Initialization in DeBERTa Tests Augustine's 1,600-Year-Old Language Learning Hypothesis

A 1,600-Year-Old Theory of Language Learning, Tested on a Neural Network

In 397 AD, Saint Augustine recalled in his *Confessions* how he learned to speak: when his elders named an object, he inferred that the thing was what the sound referred to. This is ostensive definition—an adult points at a banana and says "banana," and the child binds the sound to the visual impression. In 1953, Wittgenstein opened *Philosophical Investigations* by challenging this account: ostension presupposes that the child already has a conceptual framework. The debate ran for 1,600 years with no way to run a controlled experiment on human infants.

In September 2026, Lisa Bylinina (Utrecht Institute of Language Sciences, Utrecht University) ran a computational experiment: she let a DeBERTa model "look at pictures before reading text" to see how much of Augustine's theory holds in a neural network.

Experimental Design: Giving Word Embeddings "Visual Eyes"

The study uses the BabyLM strict-small track (~10M words of training text, mimicking child language input). The key intervention:

Before training begins, initialize the embeddings of some words with visual features.

Concretely, for words that appear in visually annotated data (e.g., "banana," "red," "dog"), features of the image regions they annotate are extracted with a frozen visual encoder (SAM), averaged, projected into the embedding space, and written into the corresponding rows of the embedding matrix. Words with no visual support (e.g., "however," "democracy") keep random initialization. The model then trains normally on text; visual information is injected only at initialization.

Finding 1: The Visual Imprint Persists Through Training

Using Representational Similarity Analysis (RSA), which measures how well the similarity structure of visually initialized embeddings correlates with the similarity structure of the corresponding visual features:

• Visually initialized model: RSA = 0.31 after training on 100M words
• Text-only baseline: RSA = 0.10 on the same words

The absolute positions of embeddings drift substantially during training, but the relative geometry—which words are near or far from each other—partially preserves the structure of the visual anchors. Like a city rebuilt with every building relocated, yet old neighbors still living next to each other.

Finding 2: Standard Evaluations Almost Cannot Detect the Effect

Standard BabyLM evaluations—BLiMP (grammatical judgments), EWoK (world knowledge), entity tracking—show no significant difference between the visually initialized model and the baseline.

The only exception is COMPS (object-property knowledge, e.g., "bananas are yellow"): visual initialization improves scores across all 9 encoder×vocabulary configurations (+1.37 points), with a larger gain (+3.65) on the harder novel-concept subtask. This makes sense: COMPS tests visually perceivable properties—color, material, shape.

Finding 3: VP-Swap Proves the Effect Is Causal

The VP-Swap (Visual-Property Swap) benchmark tests color, material, size, and shape knowledge, with each item annotated for training frequency and whether it was visually initialized.

• The advantage of visual initialization is precisely confined to initialized words; uninitialized words show no advantage even at matched frequency.
• Synthetic grounding: when visual features are synthetically transferred to previously ungrounded words, those words immediately gain the same advantage as naturally initialized ones.

This is a clean causal demonstration—the effect comes from visual initialization, not frequency, part of speech, or other confounds.

Finding 4: Abstract Words Benefit Too, but No Evaluation Detects It

Surprisingly, function words and abstract vocabulary also receive strong visual seeds (they appear in image-caption data, e.g., "the" and "on" in "the dog is on the table") and retain them through training. Their held-out mask-prediction loss drops under every random seed—the training objective itself exploits the visual prior.

But no standard evaluation detects this effect. As the paper concludes: "What evaluation would pick this up remains an open question." This implies our evaluation suites have a systematic blind spot: they test grammar and world knowledge, but not the visual grounding of word meaning. A model may gain a substantial semantic advantage that is invisible on the leaderboard.

Leaderboard Performance

• Text-only baseline: 37.91
• Visual initialization: 39.63
• Visual initialization + synthetic grounding extension: 40.62 (5th place, strict-small track)

On the "Human-like Average" metric (which includes an acquisition-age component): baseline 0.61, visual initialization 7.74, extended version 12.19—the highest overall. Visual initialization makes the model's acquisition order more human-like: visually anchored words are learned earlier, just as children learn "ball" and "dog" before "nevertheless."

What Does This Mean?

• Was Augustine right? Not entirely—Wittgenstein's critique stands; ostension requires a pre-existing conceptual framework. But even an imperfect framework (visual encoder features are far from human cognition) leaves a measurable trace. The truth lies in between.
• Visual grounding is a structural advantage, not a luxury. Current LLM training skips the visual stage entirely—every word starts from random noise. Injecting visual information only at initialization, followed by purely textual training, still yields a durable semantic advantage. This suggests multimodal alignment should happen at the embedding initialization level, not only in the training objective.
• Evaluation blind spots are bigger than we think. We build architectures, tune hyperparameters, and compare models on incomplete benchmarks that may systematically ignore important semantic dimensions—akin to the "scalar illusion" in AI safety: managing multi-dimensional capabilities with single-number metrics.

The paper's most memorable line is not a number but the open question: *what evaluation would pick this up?*—a question that applies to everything we assume we have already measured adequately.

---

Paper: Augustinian BabyLM: What Ostensive Definition Can and Cannot Teach a Small Language Model

Author: Lisa Bylinina (Utrecht University)

Code: github.com/bylinina/augustinian_babylm

Model: huggingface.co/augustinian-babylm

Published: 2026-09-10