The Contrast in a Single Number
Reproducing Llama-3.2-3B costs over $1.5 million. SmolLM3-3B costs $700K. Those are the "market prices" of small language model pretraining in 2026.
Then Puro-2B arrived: 1.4 trillion tokens, FP8 precision, consumer-grade RTX 5090 GPUs, total cost $5,090. Not fine-tuning, not distillation — pretraining from scratch. The final model approaches Qwen2.5-1.5B on 15 benchmarks and even beats Qwen2-1.5B on some tasks.
$5,090 is roughly the price of a well-specced desktop PC. The model that came out of such a machine approaches industrial-grade baselines.
This isn't magic — it's a carefully engineered "budget recipe."
Five Key Ingredients of the Recipe
Puro-2B's core isn't a single breakthrough but the synergy of five components. None is entirely novel on its own, but combined they compress costs by orders of magnitude.
Ingredient 1: RTX 5090 as the Cost-Effectiveness Tipping Point for Consumer GPUs
The NVIDIA RTX 5090 is a consumer graphics card, not a data center card. But it has 32GB of VRAM, FP8 compute support, and relatively low power draw. The Puro-2B team found that with a carefully designed parallelism strategy, this card can handle pretraining a 2B model.
The key isn't how strong a single card is, but what "consumer-grade" means: no data center contracts, no cloud provider quotas, no waiting in line for H100s. Buy a few cards, plug them into a workstation, and go. This turns "can we pretrain at all" from a resource question into an engineering question.
Ingredient 2: FP8 Mixed-Precision Training
FP8 is an 8-bit floating point format, half the bit-width of the commonly used BF16. Training directly in FP8 is numerically unstable, but Puro-2B implements a stable FP8 mixed-precision scheme — key computations run in FP8 while accumulations and sensitive operations fall back to higher precision.
This cuts both memory usage and compute roughly in half. On consumer GPUs, that effectively doubles the usable compute.
Ingredient 3: Hyperball Optimizer and MuonH
The most technical ingredient. Instead of AdamW, Puro-2B uses the Hyperball optimizer with a learning-rate scheduler called MuonH.
The core insight is "effective learning rate" — different optimizers have different sensitivities to the learning rate, so comparing raw numbers is meaningless. MuonH makes effective learning rate explicit, turning schedule design from "tuning by experience" into "deriving from curves." The team found that as long as the effective learning rate curve is matched, different schedules produce nearly identical validation loss. This turns LR tuning from a dark art into engineering.
Ingredient 4: Curriculum-Based Model Averaging
Training data isn't fed in all at once but in curriculum order. What's learned first, what comes later, and what data finishes the final phase are all co-designed with the model averaging strategy.
Model averaging is an old technique — averaging multiple late-training checkpoints often outperforms any single checkpoint. Puro-2B ties curriculum order to model averaging: the final phase's data determines the distribution of the last few checkpoints, and averaging them consolidates the final phase's learning.
Ingredient 5: The Data Recipe
Puro-2B's data isn't just heaped Common Crawl. The team designed explicit mixture ratios across two phases: Phase 1 uses 438 billion tokens to build foundations; Phase 2 uses 960 billion tokens focused on reasoning and code. Total: 1.4 trillion tokens.
Details are in §3.6 of the paper, but the core idea is: under a limited budget, data ordering and ratios matter more than total data volume.
The Puro Cost Scaling Law: A Counterintuitive Finding
Puro-2B isn't just one model but a family — the team trained multiple variants with different token budgets to fit a cost-performance curve.
The resulting rule is called the Puro Cost Scaling Law. It predicts: just $4,400 is enough to reach Qwen2-1.5B-level performance. $5,090 isn't the floor — it's the ceiling.
The practical value of this scaling law: it gives small teams a budget estimate for "good enough." No need to spend $1M on trial and error first — extrapolate with the scaling law to know what $5,000 buys.
Benchmark Results
Across 15 benchmarks:
- Math & code (4 generative tasks): Puro-2B averages 43.50, beating Qwen2-1.5B by 3.21 points and trailing Qwen2.5-1.5B by 4.02 points
- Reasoning & knowledge (11 tasks): Puro-2B averages 63.02, beating Qwen2-1.5B by 2.48 points and trailing Qwen2.5-1.5B by 2.51 points
- Paper: https://arxiv.org/abs/2608.27370
- HTML version: https://arxiv.org/html/2608.27370v1
- Model weights: https://huggingface.co/thu-pacman/Puro-2B-Base
- Training system: https://github.com/thu-pacman/Puro-Megatron
- License: Apache 2.0
Among open-recipe models, Puro-2B outperforms Instella-3B, OLMoE-A1B/7B, and MiniCPM5-1B-Base. Yulan-Mini-2.4B, SmolLM3-3B-Base, and MobileLLM-R1-950M score higher on some tasks, but they either have more parameters or are specifically optimized for math/code.
Why This Matters
Puro-2B's significance isn't "yet another 2B model trained." Qwen2.5-1.5B, SmolLM3-3B, and OLMo-2 are stronger or comparable in performance. The significance is the downward shift of the cost boundary.
In 2024, reproducing a 3B model cost over $1M. In August 2026, Puro-2B pushed that number down to $5,090 — a 200x reduction. This means:
1. Academic labs can pretrain. No industrial cluster needed — one workstation suffices. 2. Pretraining recipes are reproducible. Puro-2B open-sources weights, data manifests, and training scripts under Apache 2.0. 3. A cost scaling law can guide budgets. Not "spend first, see later" but "extrapolate first, then spend."
Honest Caveats
Puro-2B isn't free of costs. Training took 17.6 days and 22,514 GPU-hours. The RTX 5090 is consumer-grade, but the 32GB version isn't cheap, and multi-GPU parallelism is required. The $5,090 covers only GPU rental — no hardware depreciation, electricity, or labor.
Also, Puro-2B's performance approaches but doesn't exceed Qwen2.5-1.5B, which is the result of an industrial team's tuning. For a small team to match or surpass it, more work is needed on data recipes and post-training.
But as a "starting point," Puro-2B has moved the starting line to an unprecedented position.
Paper and Code
Puro's Chinese name references Prometheus, who stole fire for humanity — Puro steals pretraining for poor labs. $5,090, the price of a workstation, for a working 2B model. It's not the end, but it's a significant milestone in the democratization of pretraining.