Paper Overview
| Item | Details | |------|---------| | Title | HRM-Text: Efficient Pretraining Beyond Scaling | | Authors | Guan Wang, Changling Liu, Chenyu Wang, Cai Zhou, Yuhao Sun, Yifei Wu, Shuai Zhen, Luca Scimeca, Yasin Abbasi Yadkori | | Affiliation | Not specified (arXiv preprint) | | arXiv | 2605.20613 | | Date | 2026-05-20 | | Category | cs.CL | | Core claim | Mimicking the brain's multi-timescale frontoparietal processing, a hierarchical recurrent model (HRM) replaces the Transformer. Combined with instruction-response-only training and PrefixLM masking, a 1B model trained on a $1,500 compute budget rivals 2B-7B open models while using 100-900x fewer training tokens. |
All animal brains are energy-efficient machines. The human brain runs on about 20 watts—a dim lightbulb. A macaque brain, roughly 5 watts. A zebrafish brain, under 0.1 watts.
But your GPT: thousands of A100 GPUs, 300 watts each.
This isn't an analogy—it's like driving a bicycle with a jet engine. Nature spent billions of years solving "intelligence per watt," while AI research over the past decade has asked essentially one question: "How fast can we go by adding more fuel?"
On May 20, 2026, an arXiv preprint proposed a completely different direction. Drawing on biological brain architecture, the authors redesigned the underlying structure of language models, then trained a 1B-parameter language model from scratch using only 40B tokens and $1,500 of compute—yes, $1,500—competing with 2B-7B models trained with thousands of times more compute on multiple benchmarks.
This is not a minor tweak. It's a coordinated innovation across brain science, architecture design, and training strategy.
🧠 What Does the Brain's Frontoparietal Loop Teach Us?
To understand HRM-Text, you first need a basic neuroscience finding.
The human frontoparietal network has a well-known functional division of labor: the prefrontal cortex handles the "slow"—strategic planning, goal setting, high-level decision-making. The parietal cortex handles the "fast"—real-time sensory processing, motor coordination, immediate execution.
This division isn't accidental. It solves a fundamental computational problem: if one timescale handles all information, either you're always too late (high-frequency signals flood slow channels) or never think deeply enough (fast information flow doesn't allow long chains of computation).
The brain's solution is hierarchical timescales: the strategic layer evolves slowly, maintaining global direction and long-term goals. The execution layer responds quickly, capturing short-term patterns in the input stream. Feedback loops coordinate the two—the execution layer tells the strategic layer "what's happening now," and the strategic layer tells the execution layer "what to prioritize."
This is textbook neuroscience. Turning it into a neural network architecture is another story.
🏗️ The HRM Architecture: Encoding the Brain's "Fast-Slow" Division in Code
The core of HRM-Text is a Hierarchical Recurrent Model (HRM).
In a standard Transformer, each layer is a one-shot feedforward computation: input comes in, passes through self-attention and an FFN, and goes out. No recurrence, no state between layers. The model only sees the input at its current layer and step.
HRM does two things:
First, it decomposes computation into two timescales—a strategic layer and an execution layer. The strategic layer updates at a lower frequency, managing long-term context and global reasoning direction. The execution layer updates at a higher frequency, handling token-level local patterns.
Second, it establishes deep recurrent connections between the two layers—the strategic layer's state isn't produced in one shot; it evolves continuously over the entire input sequence. The model's "high-level goals" and "low-level details" are always in conversation.
It's like reading a book. Your eyes process a dozen characters a second (execution layer), but your brain builds an understanding of "what this book is about" at a completely different rhythm (strategic layer). If you rethought the entire book's theme after every character, you'd go mad. But with no thematic thinking at all, you'd get lost in details.
HRM implements this division explicitly through architecture.
🔧 MagicNorm and Warmup Deep Credit Assignment: Taming Deep Recurrence
Hierarchical recurrence isn't a new idea, but it has a fatal problem: training stability.
Gradients in recurrent networks must backpropagate across timesteps. In a hierarchical setup, the strategic layer's timesteps are far longer than the execution layer's—its gradient path traverses the entire input sequence. Standard training techniques explode or vanish under this depth of recurrence.
The paper introduces two key techniques:
MagicNorm: A new normalization mechanism tailored to gradient flow in hierarchical recurrent structures. The paper doesn't disclose its full mathematical form, but its core idea appears to be adaptive scaling of gradients at the interface between strategic and execution layers—ensuring slowly evolving signals aren't drowned by high-frequency noise, and fast-changing signals aren't over-compressed.
Warmup Deep Credit Assignment: A progressive training strategy. Early in training, the strategic layer updates at a high frequency (nearly matching the execution layer), so the model first learns "local responses." As training progresses, the strategic layer's update frequency gradually decreases, and the model progressively learns to "take the long view." This gradual exploration prevents the model from facing long-range credit-assignment gradients before it has mastered local detail.
Together, these techniques make HRM's training stable enough to be practical.
📖 Rejecting Pretraining Text: Instruction-Response Pairs Only
This may be the paper's boldest decision.
Standard LLM training has two stages: pretraining on massive raw text (learning language itself), then fine-tuning on instruction data (learning "how to respond to humans"). Pretraining consumes the vast majority of compute.
HRM-Text says: we skip pretraining.
They train from scratch purely on instruction-response pairs. The objective isn't "predict the next word" but "task completion"—given an instruction, generate a compliant response.
To make this work, they adopt a PrefixLM masking strategy: instruction tokens attend bidirectionally to each other, while response tokens attend only to the left (causal attention). This hybrid attention lets the model fully understand the instruction's semantics while learning to generate responses in the correct order.
The benefit of skipping pretraining is extreme: standard LLM pretraining requires trillions of tokens—HRM-Text uses only 40B, a 100-900x reduction. Compute requirements drop from "only tech giants can play" to "a few grad students with a credit card can run it."
The cost? The model has never seen raw text; its worldview is entirely defined by instruction-response pairs. Its abilities in open-domain chat, creative writing, and unconventional tasks may be limited—the paper doesn't report results on these dimensions.
📊 Benchmarks: What Did $1,500 Buy?
With a GPU budget of roughly $1,500 and 40B training tokens, the 1B-parameter HRM-Text achieved:
| Benchmark | HRM-Text (1B) | Context | |------|--------------|---------| | MMLU (general knowledge) | 60.7% | Competitive with 2B-7B open models | | ARC-C (science reasoning) | 81.9% | Clearly beats many larger models | | DROP (numerical reasoning) | 82.2% | Comparable to larger models | | GSM8K (grade-school math) | 84.5% | Strong math performance | | MATH (competition math) | 56.2% | Rare at the 1B level |
The paper claims these results use roughly 100-900x fewer training tokens and 96-432x less estimated compute than standard baselines, at competitive quality.
The 84.5% on GSM8K is especially notable—many 7B models need special prompting to reach that level. HRM-Text, a from-scratch 1B model, hit this score without chain-of-thought prompting, suggesting its hierarchical recurrent architecture may be naturally suited to multi-step reasoning.
🤔 Honestly: What We're Not Sure About
Real compute comparability. "$1,500" and "96-432x less" are the most dangerous kinds of PR numbers—heavily dependent on estimation assumptions. Different GPU prices, cloud providers, and software stacks can swing these figures wildly. The paper doesn't disclose detailed training configuration (GPU model, training duration, batch size), making independent verification difficult. Critically, the "$1,500" comparison is against from-scratch pretraining, but the cost of skipping pretraining—potential losses in open-domain capability—was never systematically quantified with controlled experiments.
Generalization of MagicNorm and Warmup Credit Assignment. Both stability techniques are described very briefly. Without full implementations, we can't judge whether they are necessary conditions for the HRM architecture or broadly applicable to other recurrent architectures. If other teams can't reproduce them, the "$1,500 1B model" claim is a myth.
Quality ceiling. MMLU 60.7% was a good score in 2024. In 2026, frontier models approach 90% on MMLU. What $1,500 buys is driving the cost of "entry-level intelligence" to an extreme—not creating "advanced intelligence." The paper doesn't discuss whether HRM-Text's efficiency advantage holds at scale. 1B works well—what about 10B? 100B? Is the recurrent architecture's viability limited to small scales?
Limits of bio-inspiration. "Inspired by the brain's frontoparietal loop" is a lovely story. But the biological brain is a complex system with dozens of specialized neurotransmitters, billions of synaptic adaptation rules, and nested timescales. How much does a two-layer hierarchical recurrent model capture? The most honest answer: probably only the most surface-level feature—"there exist two timescales"—not the decentralized, sparse, laterally-projecting structure of a real brain. Bio-inspiration is a metaphor, not a blueprint.
Instruction data quality bias. If the model has only seen instruction-response pairs, all its language ability comes from that data. The quality, coverage, and bias structure of the training data directly determine the model's capability boundaries. If the instruction data comes from a few sources (e.g., certain APIs or data vendors), the model may excel on all benchmarks—yet systematically fail on real-world edge cases.
📚 References
1. Wang, G. et al. (2026). *HRM-Text: Efficient Pretraining Beyond Scaling.* arXiv:2605.20613. 2. Kaplan, J. et al. (2020). *Scaling Laws for Neural Language Models.* arXiv:2001.08361. 3. Hoffmann, J. et al. (2022). *Training Compute-Optimal Large Language Models.* NeurIPS 2022. 4. Hasson, U. et al. (2015). *Hierarchical process memory: memory as an integral component of information processing.* Trends in Cognitive Sciences. 5. Graves, A. et al. (2014). *Neural Turing Machines.* arXiv:1410.5401. 6. Gu, A. & Dao, T. (2023). *Mamba: Linear-Time Sequence Modeling with Selective State Spaces.* arXiv:2312.00752.