Training a 1B Model From Scratch on Fully Legal Data: Mimir's Small-but-Mighty Experiment

> *"While everyone chases bigger models, someone went back to the basics and built a small house out of legally sourced bricks—and discovered the small house can keep out the wind too."*

Introduction: An Impossible Constraint

Imagine you are an architect. In the AI world of 2026, all your peers build with the same kind of bricks—massive text scraped from the internet, with murky provenance: some copyrighted, some unlicensed, some of unknown origin. Few care, because more bricks means taller towers.

Then one day you impose a constraint on yourself: only use bricks with clear provenance and explicit licensing.

Your peers think you're insane. "Legal data" means an order of magnitude less data. What could a 1B-parameter model trained from scratch on lawful data possibly achieve?

Peter Schneider-Kamp's team at the University of Southern Denmark ran this experiment. The result is Mimir v1—a 1B-parameter model using the HRM architecture that, across 20 benchmarks, trails Qwen 3.5 4B by only 0.3 points on English tasks, improves math and code performance by 36.7% over the original HRM-Text, and sets a new Danish-language state of the art.

This is not "yet another open-source model" story. It is an experiment about how constraints breed innovation.

HRM: Not a Transformer—Hierarchical Reasoning

What first draws attention to Mimir is not its data strategy but its architecture: not a standard Transformer, but the Hierarchical Reasoning Model (HRM).

A standard Transformer reasons in a straight line: tokens are generated one by one, each step conditioned on everything before it—like a writer re-reading everything written so far for every single new word.

HRM splits reasoning into two levels:

• Lower level: rapidly generates candidate content
• Higher level: reviews, corrects, confirms

This resembles how humans actually write—draft first, revise later, with review at key junctures rather than polishing every word.

Mimir uses this architecture: 1B parameters, PrefixLM (bidirectional attention over the prefix, causal attention for generation), FlashAttention acceleration. Fully open-sourced on HuggingFace.

Data: 161 Datasets, 70.4B Tokens, All "Legal"

This is Mimir's core design decision.

"Legal data" here means permissible post-training data—data with explicit authorization for lawful use.

Mimir's training mix comprises 161 datasets totaling ~70.4B tokens:

• English instruction (13.58B, 19.3%): Dolci, Tulu 3, Nemotron instruction-following data
• Math and reasoning (10.40B): math problem-solving, logical reasoning
• Sapient mixed category (11.92B, 16.9%): 107 subsets from Flan, Platypus, Tasksource
• Danish: dedicated Danish instruction and comprehension data
• Other: QA, summarization, dialogue, etc.

Key detail: some of the original Sapient dataset did not align with the Danish Foundation Models (DFM) data-compliance philosophy. Rather than using it, the team performed "synthetic replacement"—substituting 70 Sapient-synth datasets (75M tokens) for the non-compliant originals. These synthetic datasets were generated and audited, preserving the original tasks' function with fully traceable provenance.

It's like finding a batch of lumber of questionable origin: not using it, but instead procuring an equivalent batch of verified timber matched to the same dimensions and purposes. More work—but every piece is traceable.

Results: Asymmetric Competitiveness for a Small Model

Mimir was compared against three rivals on 20 benchmarks:

| Model | Params | English | Math & Code | Danish | |-------|--------|---------|-------------|--------| | HRM-Text 1B | 1B | baseline | 46.9 | baseline | | DFM Mimir 1B | 1B | near Qwen 3.5 4B | 64.1 | SOTA | | Qwen 3.5 4B | 4B | highest | — | — | | Gemma 4 E2B | ~2B | — | — | — |

Key numbers:

• English tasks: Mimir trails Qwen 3.5 4B by only 0.3 points—a 1B model matching a 4B model.
• Math and code: 36.7% improvement over HRM-Text 1B (64.1 vs 46.9), only 3.8% behind SmolLM3 3B.
• Danish: a new SOTA—not an accident, since the team is Danish and built dedicated Danish data.

But the most notable pattern is "asymmetric competitiveness": on general English tasks Mimir approaches a 4x-larger Qwen; on math and code it decisively beats the same-architecture HRM-Text; on Danish it is outright SOTA.

This shows that data quality and data compliance are not opposed. Legal data, carefully curated and proportioned, can match or exceed larger models trained on non-compliant data in specific domains.

Why This Matters

1. Open source is more than releasing weights

Many "open models" open the weights but not the training data, or the data's provenance is unclear—making reproduction and auditing impossible. Mimir goes further: the full 161-dataset list, processing pipeline, and synthetic replacement scheme are all public. Anyone can review, reproduce, improve.

2. Small languages need not be marginalized

Mainstream models are English-first; Danish (5 million speakers) is a second-class citizen in most models. Mimir optimized for Danish and set a new SOTA—proof that small languages don't need to wait for Big Tech's charity; they can build their own.

3. The potential of the HRM architecture

HRM is relatively new. Mimir shows that at the 1B scale, its hierarchical reasoning design can compete with a 4B standard Transformer—empirical support for architectural innovation. Not every road must be a Transformer.

Honest Limitations

• Math and code still trail SmolLM3 3B (3.8% gap). 1B parameters do have a ceiling on reasoning-heavy tasks.
• The HRM architecture requires FlashAttention support, raising deployment barriers versus standard Transformers.
• The Danish SOTA partly benefits from the team's regional advantage—other small languages may not replicate it.
• 70.4B tokens suffice at 1B scale, but whether they suffice for larger models is unverified.

The Bigger Picture

Mimir raises a neglected question: how high is the true "compliance cost" of AI research?

The mainstream narrative is "more data is better; provenance doesn't matter." But that narrative assumes you don't plan to publish your training data or submit to audit.

Once you commit to public data provenance, compliance cost becomes a hard constraint. Mimir's experiment shows: under compliance constraints, a 1B model can still produce competitive results. This is not "small is beautiful" self-consolation—it is empirical evidence that constraints breed innovation.

The synthetic replacement scheme is one example: rather than dropping non-compliant data, the team rebuilt equivalent data through a generate-and-audit process. Far costlier than scraping, but the product is traceable and auditable.

This offers the open-source community a replicable path: not a thought experiment about "what if we trained from scratch on legal data," but an empirical report: "we did it—here are the results."

Resources

• Paper: arXiv:2608.13517
• Model: HuggingFace: danish-foundation-models/DFM-Mimir
• Evaluation tooling: GitHub: UKGovernmentBEIS/inspect_ai

*When constraints become the starting point of design rather than an obstacle to patch later, the results often surprise. Mimir isn't chasing bigger models—it's proving another path is viable.*