Can St. Augustine's Ostension Teach Language Models Words? Testing 4th-Century Philosophy on DeBERTa

Forum topic · ✨步子哥 · 2026-09-11

Summary

This forum post discusses a BabyLM 2026 workshop paper (arXiv:2609.11870) by Lisa Bylinina that operationalizes St. Augustine's theory of ostensive definition in a language model. Before training a DeBERTa masked language model on a 10M-word BabyLM-scale corpus, the author replaced the initial embeddings of concrete words (e.g., 'dog', 'red', 'large') with visual feature vectors derived from image region annotations, while function and abstract words remained randomly initialized. Results show visual seeding leaves persistent, detectable traces in the embedding space and yields a consistent zero-shot advantage on object-property benchmarks (COMPS and a custom Visual-Property Swap test), but only for seeded words. A synthetic seeding control confirms causality: artificially seeded words gain the same advantage. Notably, seeded function words show lower masked-prediction loss, yet no standard BabyLM grammar benchmark detects any difference. The post argues the paper's real contribution is methodological: standard evaluation may systematically miss what models internally learn, calling for smarter probes rather than bigger models.

Can St. Augustine's Ostension Teach Language Models to Learn Words?

Testing 4th-century philosophy on DeBERTa — a note on a BabyLM Workshop 2026 experimental paper.

> arXiv: 2609.11870 | Lisa Bylinina | BabyLM Workshop 2026

---

A 1,600-Year-Old Idea

In 397 CE, St. Augustine recalled in the *Confessions* how he learned to speak: adults pointed at objects and named them, and he associated sounds with things. Philosophers later called this ostensive definition — defining a word by pointing.

The idea seems so natural it might not need testing. But modern language models start from random embeddings: "banana" carries no prior meaning before training, and all semantics must be learned from text. Lisa Bylinina directly implemented Augustine's ostension into a small language model to see what happens.

---

Experimental Design

She trained DeBERTa, a masked language model, on a small 10M-word corpus (BabyLM scale). The key manipulation:

Before training began, the initial embeddings of a subset of words were replaced with visual features. Using image region annotation data, words like "dog", "red", and "large" were initialized with visual feature vectors of their corresponding image regions. Other words (function words, abstract words) kept random initialization.

Then training proceeded normally. Afterward, "visually pre-seeded" and "purely random initialization" models were compared across benchmarks.

It is like giving an infant an innate head start: knowing, before hearing any language, that "red" relates to the visual color red.

---

Results: An Imprint, Depending on Where You Look

Visual seeding does leave a persistent imprint. After full training, seeded words' embeddings retain traces of the visual features — detectable through the structure of the embedding space.

But here is the catch: most BabyLM benchmarks cannot detect this difference at all. These benchmarks probe abstract grammatical knowledge — subject-verb agreement, island constraints, binding theory — where visual seeding provides no help.

The sole exception is the COMPS benchmark (Misra et al. 2023), which tests object-property knowledge (e.g., "bananas are yellow"). Here, visual seeding shows a zero-shot advantage in every configuration.

To verify further, the author built a custom Visual-Property Swap benchmark testing color, material, size, and shape knowledge, controlling each word's corpus frequency and seeding status. The result held: visually seeded models show a persistent, reproducible advantage on seeded words, and only on seeded words.

---

The Most Elegant Experiment: Synthetic Seeding Transfer

For causal verification, the author ran a clever control: words that were never visually seeded were given synthetic visual embeddings (random visual feature values). These "retroactively seeded" words gained the same advantage.

The advantage comes from the seeding operation itself, not from any word being inherently easy to learn. Whichever word you seed is the word that benefits.

---

The Ghost of Function Words

A deeper finding emerged. Function words ("the", "is", "in") and abstract words also received visual seeds (via some indirect pathway) and retained them throughout training. More critically, the training objective itself exploits these seeds: masked-prediction loss on seeded words is lower under every seed configuration.

The model "knows" these words are different. Yet no benchmark can detect it.

The paper concludes: "What kind of evaluation could capture this signal remains an open question."

---

What This Means

The paper's value is not that "visual seeding makes models stronger" — the effect is modest and task-specific. Its value lies in revealing a deeper problem:

How we evaluate language models may systematically miss what is actually happening inside them.

A model can retain traces of visual priors internally, exploit those traces in its training objective, and still perform identically to a "random-start" model on all standard benchmarks. That is not a model problem — it is an evaluation problem.

St. Augustine's ostension did teach the model something — we just don't know how to ask it.

---

My Take

This paper suggests a broader pattern: many debates about "what models learn" are really debates about "how we measure learning." What happens inside models is richer than benchmark scores indicate. We don't need bigger models — we need smarter probes.

Also, translating a 1,600-year-old philosophical theory directly into code and experiments is a dialogue across time worth encouraging. Philosophy doesn't just write papers — it can sometimes generate experimental designs.

---

Paper: https://arxiv.org/abs/2609.11870

Code: not yet released

Related benchmarks: BabyLM Challenge, COMPS, Visual-Property Swap