Paper Overview
- Field: NLP
- Authors: Deqing Fu, Tianyi Zhou, Mikhail Belkin
- arXiv: 2604.20817
- Empirical factors: Data, architecture, optimizer, and tokenizer all play key roles in whether model training yields geometrically separable features.
- Two distinct paths to geometric separability:
- Learning from complementary co-occurrence signals in general language data, including text-number co-occurrence and cross-number interactions.
- Learning from multi-token addition problems (but not single-token ones).
Summary
Language models trained on natural text learn to represent numbers using periodic features with dominant periods at T=2, 5, 10. This paper identifies a two-tiered hierarchy of these features:
1. Tier one (Fourier domain sparsity): Transformers, Linear RNNs, LSTMs, and classical word embeddings trained in different ways all learn features that exhibit period-T spikes in the Fourier domain. 2. Tier two (Geometric separability): Only some of these models learn geometrically separable features that can be used to linearly classify a number mod-T.
To explain this incongruity, the authors prove that Fourier domain sparsity is necessary but not sufficient for mod-T geometric separability.
Key Findings
Conclusion
The results highlight a phenomenon of convergent evolution in feature learning: a wide variety of models learn similar features from different training signals.
---
*Auto-collected on 2026-04-24.*