Paper Information
| Item | Detail | |------|--------| | Title | HRM-Text: Efficient Pretraining Beyond Scaling | | Authors | Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori | | arXiv | 2605.20613 | | Date | 2026-05-20 | | Category | cs.CL | | Core claim | A brain-inspired hierarchical recurrent model (HRM) replaces the Transformer, trained with instruction-response pairs and PrefixLM masking. With a ~$1,500 compute budget, a 1B-parameter model competes with 2-7B open models while using 100-900x fewer training tokens. |
All animal brains are energy-efficient machines: the human brain runs on about 20 watts, a macaque's on ~5 watts, a zebrafish's on less than 0.1 W. Your GPT, by contrast, runs on thousands of A100 GPUs at 300 watts each. Nature spent billions of years solving "intelligence per watt"; AI research over the past decade has mostly asked only "how fast can we go with more fuel?"
On May 20, 2026, an arXiv preprint took a different direction: borrowing from biological brain architecture, redesigning the language model architecture from the ground up, and training a 1B-parameter model from scratch with only 40B tokens and $1,500 of compute — competing on multiple benchmarks with 2-7B models trained with thousands of times more resources.
What the Brain's Frontoparietal Loop Teaches
The human frontoparietal network has a well-known division of labor: the prefrontal cortex handles "slow" processes — strategic planning, goal setting, high-level decisions — while the parietal cortex handles "fast" processes — real-time perception, motor coordination, immediate execution. This resolves a fundamental computational problem: a single processing timescale either gets flooded by high-frequency signals or never allows sufficiently long chains of computation. The brain's solution is hierarchical timescales: a slow strategic layer maintains global direction and long-term goals, a fast execution layer captures short-term patterns, and feedback loops coordinate between the two.
The HRM Architecture: Coding the Fast-Slow Split
HRM-Text's core is a Hierarchical Recurrent Model (HRM). In a standard Transformer, each layer is a one-shot feedforward pass with no state or recurrence. HRM does two things:
1. It decomposes computation into two timescales — a strategic layer that updates at lower frequency, managing long-range context and global reasoning, and an execution layer that updates at higher frequency, handling token-level local patterns. 2. It connects them with deep recurrent connections — the strategic layer's state evolves continuously over the whole input sequence, so the model's "high-level goals" and "low-level details" are always in dialogue.
It's like reading a book: your eyes process dozens of characters per second (execution layer), while your brain builds an understanding of what the book is about on a completely different rhythm (strategic layer).
MagicNorm and Warmup Deep Credit Assignment
Hierarchical recurrence has a fatal problem: training stability. Gradients must backpropagate across timesteps, and the strategic layer's gradient paths span the entire sequence. The paper introduces two fixes:
- MagicNorm: a normalization mechanism tailored to gradient flow in hierarchical recurrence — adaptively scaling gradients at the boundary between the strategic and execution layers, so slow signals aren't drowned by high-frequency noise and fast signals aren't over-compressed. Its exact mathematical form is not fully disclosed.
- Warmup Deep Credit Assignment: a progressive strategy where the strategic layer initially updates at near-execution frequency; as training proceeds, its update frequency gradually drops, letting the model learn local responses before tackling long-range credit assignment.
- Compute comparability. "$1,500" and "96-432x less" depend heavily on estimation assumptions (GPU pricing, cloud provider, software stack). Training configuration details are not disclosed, making independent verification difficult. The unquantified cost of skipping pretraining — potential open-domain losses — is not controlled for.
- Generalizability of the stability techniques. MagicNorm and Warmup Credit Assignment are described very briefly. Without full implementations, we can't tell whether they are necessary conditions of the HRM architecture; if others can't reproduce it, the claim is a myth.
- Quality ceiling. MMLU 60.7% was a good 2024 score; frontier models in 2026 approach 90%. $1,500 pushes "entry-level intelligence" to extreme cheapness, not "advanced intelligence." Whether the efficiency advantage survives at 10B or 100B parameters is unaddressed.
- Limits of bio-inspiration. A two-layer hierarchical recurrent model likely captures only the most surface-level feature of the brain — "two timescales exist" — not the decentralized, sparse, heavily lateral structure of real cortex. Bio-inspiration is a metaphor, not a blueprint.
- Instruction data bias. With no raw-text pretraining, the model's entire capability is bounded by the quality, coverage, and bias structure of its instruction data — it may excel on benchmarks yet fail systematically on real-world edge cases.
Skipping Pretraining: Instruction-Response Pairs Only
Perhaps the boldest decision: HRM-Text skips pretraining on raw text entirely. It is trained from scratch only on instruction-response pairs, with the objective being task completion rather than next-token prediction. A PrefixLM masking scheme lets instruction tokens attend bidirectionally while response tokens use causal (leftward) attention — so the model fully understands the instruction's semantics and still generates replies in correct order.
The benefit is extreme: standard pretraining requires trillions of tokens; HRM-Text used 40B — 100-900x fewer — dropping compute from "only tech giants can play" to "a few grad students with one credit card." The cost: the model's worldview is defined entirely by instruction data, so open-domain chat and creative writing may be limited — dimensions the paper does not report on.
Benchmarks: What $1,500 Buys
| Benchmark | HRM-Text (1B) | Context | |-----------|--------------|---------| | MMLU (general knowledge) | 60.7% | Competitive with 2-7B open models | | ARC-C (science reasoning) | 81.9% | Notably above many larger models | | DROP (numerical reasoning) | 82.2% | On par with larger models | | GSM8K (grade-school math) | 84.5% | Strong math performance | | MATH (competition math) | 56.2% | Rare at the 1B scale |
The paper claims these results use ~100-900x fewer training tokens and an estimated 96-432x less compute than standard baselines. The 84.5% on GSM8K is especially notable — many 7B models need special prompting to reach that level, and HRM-Text achieved it from scratch without chain-of-thought prompting, suggesting the hierarchical recurrent architecture may be naturally suited to multi-step reasoning.
Honest Caveats: What We Don't Know
References
1. Wang, G. et al. (2026). *HRM-Text: Efficient Pretraining Beyond Scaling.* arXiv:2605.20613. 2. Kaplan, J. et al. (2020). *Scaling Laws for Neural Language Models.* arXiv:2001.08361. 3. Hoffmann, J. et al. (2022). *Training Compute-Optimal Large Language Models.* NeurIPS 2022. 4. Hasson, U. et al. (2015). *Hierarchical process memory.* Trends in Cognitive Sciences. 5. Graves, A. et al. (2014). *Neural Turing Machines.* arXiv:1410.5401. 6. Gu, A. & Dao, T. (2023). *Mamba: Linear-Time Sequence Modeling with Selective State Spaces.* arXiv:2312.00752.