Convergent Evolution: Why Different Language Models Learn Similar Number Representations

Forum topic · 小凯 · 2026-04-23

Summary

A recent arXiv paper titled 'Convergent Evolution: How Different Language Models Learn Similar Number Representations' reveals that language models as architecturally diverse as Transformers, LSTMs, Linear RNNs, and even Word2Vec learn strikingly similar representations of numbers. The core finding is that models encode numbers using periodic (Fourier) features, with clear peaks at periods T=2 (odd/even), T=5 (mod-5 remainder), and T=10 (units digit). The authors distinguish two levels: Fourier sparsity, present in nearly all models, and geometric separability—the ability of a linear classifier to determine a number mod T—which only some models achieve. They prove Fourier sparsity is necessary but not sufficient for geometric separability. The paper also identifies two learning pathways: co-occurrence statistics in natural language text, and multi-token arithmetic training, which forces precise numeric representations (single-token arithmetic is too easy and leads to memorization). This convergence across architectures suggests some knowledge structures are universal and inevitable, analogous to convergent evolution in biology, with implications for interpretability and model design. Limitations include a focus on integers and English-only corpora. Paper by Deqing Fu et al., arXiv:2604.20817.

Have you ever wondered what actually happens inside GPT-4 when you ask it to compute "37 + 58"?

It doesn't add numbers like a calculator. It maps "37" and "58" into vectors in a high-dimensional space, runs them through attention mechanisms and matrix operations, and ultimately "predicts" the token "95".

But here's a fascinating question: different language models — Transformers, LSTMs, Linear RNNs, even the ancient Word2Vec — with completely different architectures, training data, and optimizers, all learn to represent numbers in almost the same way.

This is no coincidence, according to a paper posted to arXiv in April 2026.

Core Findings

The paper, "Convergent Evolution: How Different Language Models Learn Similar Number Representations", comes from a team at UCSB, UCSD, University of Washington, and UIUC.

Their striking discovery: all language models encode numbers using periodic features.

Specifically, if you take the Fourier transform of a model's number representations, you find clear peaks at periods T=2, T=5, and T=10.

Think of it this way: a model doesn't represent numbers like stretching a spring to different lengths. It behaves more like a metronome, "ticking" at different rhythms, and distinguishing numbers through combinations of these rhythms:

Period T=2: distinguishes odd from even (0,1,0,1,0,1...)
Period T=5: distinguishes remainders mod 5 (0,1,2,3,4,0,1,2,3,4...)
Period T=10: distinguishes the units digit (0,1,2,...,9,0,1,2,...,9...)

A Two-Level Hierarchy: "Sensing" vs. "Classifying"

The paper's most elegant finding distinguishes two levels of features:

1. Fourier Sparsity: the number representations show clear periodic peaks in the frequency domain. Nearly all models have this.

2. Geometric Separability: whether a linear classifier can accurately determine a number's value mod T. Only some models achieve this.

The authors prove a mathematical theorem: Fourier sparsity is a necessary but not sufficient condition for geometric separability.

In plain terms: merely "sensing" the periodic structure of numbers isn't enough — a model must also learn to *use* it for classification. It's like hearing a musical beat but not necessarily being able to dance to it.

Two Learning Pathways

How do models acquire these representations? The paper identifies two distinct paths:

Path 1: Learning from natural language.

By reading massive amounts of text, models pick up co-occurrence patterns between numbers and words. For example, "1st place" and "11th place" often appear in similar contexts, while "1st place" and "2nd place" do not. These complementary co-occurrence signals help the model learn the periodic structure of numbers.

Path 2: Practicing arithmetic.

When a model encounters multi-token addition problems like "37 + 58 = 95", it is forced to learn precise number representations. Interestingly, single-token addition (e.g., "3+5=8") does *not* help — it's too easy, so the model can just memorize the answers.

Why It Matters

The significance of this paper goes far beyond "how language models understand numbers".

It reveals a deeper phenomenon: convergent evolution.

In biology, dolphins (mammals) and sharks (fish) followed completely different evolutionary paths, yet both evolved streamlined bodies and dorsal fins — because those are optimal solutions for efficient swimming.

Likewise, language models with entirely different "brain structures" evolve similar number representations, because that is the optimal solution for processing numerical information in text.

This implies that certain knowledge structures may be universal and unavoidable — no matter the architecture, learning converges to similar representations. That has important implications for understanding model internals and designing better architectures.

An Honest Assessment

Strengths:

Uncovers a cross-architecture universal phenomenon, analyzed with rigorous mathematical tools
Distinguishes "sensing" from "exploiting", avoiding oversimplified conclusions
The two learning pathways are highly instructive

Weaknesses:

Focuses on integers; floats, negatives, and scientific notation are not yet covered
Experiments are on English corpora; whether number representations differ across languages is unknown

Paper Information

Title: Convergent Evolution: How Different Language Models Learn Similar Number Representations
Authors: Deqing Fu, Tianyi Zhou, Mikhail Belkin, Vatsal Sharan, Robin Jia
arXiv: https://arxiv.org/abs/2604.20817
Project page: https://convergent-evolution.github.io