English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

What Babies Can Do That AI Can't: Learning Theories from Pure Observation (arXiv:2611.08177)

Forum topic · 小凯 · 2026-11-06

Summary

BabyTheorist (arXiv:2611.08177) is a benchmark that measures observational theory learning: the ability to induce reusable theories from purely observing before/after visual transitions, with no labels, task identities, or demonstrations — mirroring how infants discover object permanence, gravity, and causality. Tasks are generated by hidden compositional programs over 15x15 grids, arranged in an eight-level curriculum with a 'level-skipping' entrance exam that tests whether learned rules genuinely generalize. Seven baselines spanning autoregressive transformers, frontier LLMs, world models, and a compositional DSL learner (NEO) all stall within the first two of eight levels; GPT-5.6 Sol scores only 51–64% on level one, below the 80% promotion threshold, and all learned baselines fail level one in the multi-object setting. Crucially, a PPO agent given ground-truth DSL programs solves all single-object levels, showing the bottleneck is theory acquisition, not application. The post analyzes three obstacles: unsupervised program search explosion, latent-variable models capturing displacement but not nameable operators, and LLMs' imitation rather than compression. It concludes that machines can apply theories but cannot yet invent them.

What Babies Can Do That AI Can't: Inducing Theories from Pure Observation

— S5 Finale: When Benchmarks Confront the Definition of "Understanding"

> Paper: *BabyTheorist: Benchmarking Observational Theory Learning* > Authors: D. Baek, J. Baek, M. Jo et al., 7 authors (arXiv:2611.08177, 2026-11, cs.AI)

1. The Problem of Defining Understanding

How do we know whether a machine truly understands the world? Standard practice in visual-reasoning benchmarks is a few few-shot puzzles: give a few examples, test on new ones. That measures the ability to apply rules — the rules are laid out; can you follow them?

But developmental psychology has long pointed out that the real miracle of human intelligence lies upstream: with no teacher, no labels, no demonstrations, infants watch the world change and induce reusable theories — object permanence, gravity, causality — then use those theories to handle situations they have never seen. The paper calls this observational theory learning: acquiring reusable theories from pure observation and transferring them to new scenarios.

The difference from application ability is the difference between memorizing worked examples and inventing theorems. This paper builds a machine to measure that difference.

2. Benchmark Design: A "Theory Pressure Gauge"

BabyTheorist's test items are sequences of before/after changes in 15×15 grid images. Each "before → after" pair is generated by a hidden compositional program of the form step(selector, primitive) — 14 primitives in three families (transformations: rotate/flip/recolor; movement: translation scaled by the object's own size; relations: color swap/alignment), plus a family of selectors (all / by color / by position, etc.).

There are three key design layers. Layer one: pure observation. The learner sees only before/after transitions — "no program labels, no task identities, no support sets" — theories must be squeezed out of the raw observation stream. Layer two: curriculum. Eight difficulty levels with increasing program depth, 250,000 observations per level, ≥80% pass rate to advance. Layer three, the sharpest: level skipping. Before each level begins, an entrance exam is given; passing it means skipping that level's entire training data and jumping ahead. Level skipping tests exactly a theory's "reusability": is your theory a real regularity, or overfit muscle memory?

3. The Verdict: Seven Baselines, All Stuck in the First Two Levels

Seven representative baselines appear, covering three schools: implicit (autoregressive Transformers, two frontier LLMs), holistic-latent (continuous/discrete world models and variants), and compositional (NEO, which infers discrete primitive sequences from a learned codebook). Out of a perfect 800, the result in one sentence:

> All seven baselines stall within the first two of eight curriculum levels, and the gap keeps widening as the curriculum advances.

The most striking data point is the frontier LLM: GPT-5.6 Sol scores only 51–64% on level one, below the 80% promotion threshold. The strongest baseline, NEO (≈137 points), barely reaches level two. In "multi-object" mode, all learned baselines fail even level one.

But the paper includes a decisive control: given the ground-truth DSL directly, a PPO agent clears all eight single-object levels. The conclusion is clear — the bottleneck is not applying theories, but acquiring them. Once rules are handed over, existing methods can use them; extracting rules from unlabeled observations is the common breakpoint of all current approaches.

4. Why It's So Hard: Three Walls

Wall one: the search explosion of unsupervised induction. Theory learning is fundamentally induction over program space — no labels means no gradient signal to guide the search, and the compositional space grows exponentially; blind search is doomed.

Wall two: the glue-ification of latent variables. Holistic latent models compress "change" into a continuous vector — they memorize the displacement from one frame to the next but cannot extract nameable, recombinable operations. The recurring subroutines in the observation stream ("select the red object, then move it one cell right" — templates carrying object placeholders) leave no trace in glue-like latent space.

Wall three: LLMs' imitation inertia. Implicit models excel at continuing statistical regularities (predicting what the next frame looks like), while theory learning is an act of compression — folding myriad observations into a few operators. Imitation and compression point in opposite directions.

5. Finale: Mathematics Is the Answer Humanity Already Has

The nine articles of the S5 series circled the theme of a "mathematical foundation": impossibility proofs (Ising), complexity bounds (boson sampling), spectral dynamics (LoRA), measure-theoretic depths (conformal prediction), symmetry groups (Lie), topological thresholds (quantum memory), measurement theory (integrated information)... Their common undertone is theory — compressing countless phenomena into a few reusable laws.

BabyTheorist turns that undertone into the test itself: can machines do what human experts do daily — squeeze reusable structure out of observation streams? The current answer is no, and the gap grows exponentially with complexity. This gives the season an honest ending: we have built machines that can apply theories, but machines that "invent theories" remain on the first step.

Over three centuries ago, Newton squeezed three laws out of chaotic planetary observations, and humanity saw for the first time the power of "theory." BabyTheorist says: making machines retrace Newton's path is beyond us today. The next leap in intelligence may hide at some waypoint along that road.

---

Further Reading

1. Baek et al., *BabyTheorist: Benchmarking Observational Theory Learning from Pure Visual Transitions*, arXiv:2611.08177 (2026) 2. Chollet, *On the Measure of Intelligence* (2019, precursor to the ARC "application vs. acquisition" distinction) 3. Piaget, *The Construction of Reality in the Child* 4. Lake, Ullman, Tenenbaum, Gershman, *Building machines that learn and think like people* (2017)

*This is article 10 of Season 5 of zhichai.net's "Mathematical Foundations and Frontier Intersections" series (S5-Round 10, finale).*

Tags

#benchmark#observational-theory-learning#world-models#large-language-models#program-induction#visual-reasoning#baby-cognition#arxiv-2611.08177

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178636813