A Stark Contrast in Numbers
Reproducing Llama-3.2-3B reportedly costs over $1.5 million. SmolLM3-3B: around $700K. That has been the "market rate" for small language model pretraining.
Then Puro-2B arrived: 1.4 trillion tokens, FP8 precision, consumer-grade RTX 5090 GPUs, and a total cost of $5,090. Not fine-tuning, not distillation — pretraining from scratch. The final model approaches Qwen2.5-1.5B on 15 benchmarks and even surpasses Qwen2-1.5B on some tasks.
$5,090 is roughly the price of a decent desktop PC. This is not magic — it is a carefully engineered "budget recipe."
Five Key Ingredients
No single breakthrough explains the result. It is the synergy of five components, each individually not novel, but together yielding an order-of-magnitude cost reduction.
1. RTX 5090 as the consumer-GPU inflection point
The NVIDIA RTX 5090 is a consumer card, not a datacenter GPU — but it offers 32GB VRAM, FP8 compute support, and relatively low power draw. Under a carefully designed parallelism strategy, these cards can shoulder 2B-model pretraining. The key point: "consumer-grade" means no datacenter contracts, no cloud quotas, no waiting for H100s. Pretraining shifts from a resources problem to an engineering problem.
2. FP8 mixed-precision training
FP8 halves the bit width of BF16. Naive FP8 training is numerically unstable, but Puro-2B implements a stable FP8 mixed-precision scheme — FP8 for key computations, higher precision for accumulations and sensitive operations. This roughly halves memory and compute, effectively doubling usable compute on consumer GPUs.
3. The Hyperball optimizer and MuonH scheduler
Instead of AdamW, Puro-2B uses the Hyperball optimizer with a learning-rate scheduler called MuonH. The core insight is "effective learning rate": different optimizers have different sensitivities, so raw values aren't comparable. MuonH makes the effective learning rate explicit, turning schedule design from intuition into curve-fitting. The team found that matching effective learning-rate curves yields nearly identical validation loss across different schedules.
4. Curriculum-coupled model averaging
Training data is fed in a curriculum order, with the sequencing coupled to a model-averaging strategy — averaging late-stage checkpoints often beats any single checkpoint. Here, the final-stage data distribution shapes the final checkpoints, and averaging them consolidates that final stage of learning.
5. The data recipe
Not a naive Common Crawl dump. Phase 1 uses 438.8 billion tokens for foundations; Phase 2 uses 960 billion tokens targeting reasoning and code — 1.4 trillion tokens total. Details are in §3.6 of the paper. Core idea: under a fixed budget, data ordering and mixing ratios matter more than raw volume.
The Puro Cost Scaling Law: A Counterintuitive Finding
Puro-2B is actually a family of models trained at various token budgets to fit a cost-performance curve. The resulting Puro Cost Scaling Law predicts that $4,400 suffices to reach Qwen2-1.5B-level performance — $5,090 is the ceiling, not the floor. For small teams, this means budget estimation by extrapolation, not million-dollar trial and error.
Benchmark Results
Across 15 benchmarks:
- Math & code (4 generative tasks): Puro-2B averages 43.50, beating Qwen2-1.5B by 3.21 points and trailing Qwen2.5-1.5B by 4.02 points
- Reasoning & knowledge (11 tasks): Puro-2B averages 63.02, beating Qwen2-1.5B by 2.48 points and trailing Qwen2.5-1.5B by 2.51 points
- Paper: https://arxiv.org/abs/2608.27370
- HTML version: https://arxiv.org/html/2608.27370v1
- Model weights: https://huggingface.co/thu-pacman/Puro-2B-Base
- Training system: https://github.com/thu-pacman/Puro-Megatron
- License: Apache 2.0
Among fully open recipes, Puro-2B outperforms Instella-3B, OLMoE-A1B/7B, and MiniCPM5-1B-Base. Yulan-Mini-2.4B, SmolLM3-3B-Base, and MobileLLM-R1-950M score higher on some tasks, but have more parameters or math/code-specific optimization.
Why It Matters
The significance is not "another 2B model" — Qwen2.5-1.5B, SmolLM3-3B, and OLMo-2 match or beat it. The point is the downward shift of the cost boundary. In 2024, reproducing a 3B model cost $1M+. In August 2026, Puro-2B compressed that to $5,090 — a 200× reduction. This means:
1. Academic labs can pretrain. A workstation suffices, no industrial cluster needed. 2. Pretraining recipes are reproducible. Weights, data manifests, and training scripts are open-sourced under Apache 2.0. 3. Cost scaling laws can guide budgets. Estimate first, spend second.
Honest Caveats
Training took 17.6 days and 22,514 GPU-hours. Even the 32GB RTX 5090 is not cheap, and multi-GPU parallelism is required. The $5,090 covers GPU rental only — no hardware depreciation, electricity, or labor. And while Puro-2B approaches Qwen2.5-1.5B, it does not surpass it; closing that gap will require further work on data recipes and post-training. But as a starting point, it moves that starting line unprecedentedly far.
Paper & Code
The name "Puro" evokes Prometheus stealing fire for humanity — Puro steals pretraining for poor labs. $5,090, the price of a workstation, for a working 2B model. Not the endpoint, but a real milestone in the democratization of pretraining.