MiniMind: Train a 64M LLM From Scratch for ~$0.40 in 2 Hours on One RTX 3090

Forum topic · ✨步子哥 · 2026-09-01

Summary

MiniMind is an open-source (Apache 2.0) educational project that trains a complete 64M-parameter LLM from scratch using pure PyTorch, with no reliance on high-level libraries like transformers, trl, or peft. The full pipeline covers pretraining, SFT, LoRA, RLHF-DPO, RLAIF with PPO/GRPO/CISPO, tool use, agentic RL, adaptive thinking, knowledge distillation, and YaRN long-context extension. Training takes about 2 hours on a single RTX 3090 at a GPU rental cost of roughly 3 RMB (~$0.40). Model architecture aligns with the Qwen3 / Qwen3-MoE ecosystem, and a 198M-A64M MoE variant is also available. Unlike nanoGPT (pretraining only) or LLaMA-Factory (wrapper-based full pipeline), MiniMind implements every core algorithm—including attention, RoPE, MoE routing, GRPO, and CISPO—line by line, aiming to turn LLM training from a black box into a white box for learners. It gained around 1,005 GitHub stars per day at the time of writing.

> Repository: jingyaogong/minimind > Language: Python (native PyTorch) > License: Apache 2.0 > Trending: 1005 stars/day

In One Sentence

A complete LLM training project built from scratch without high-level wrappers like transformers/trl/peft — covering pretraining, SFT, LoRA, RLHF-DPO, RLAIF (PPO/GRPO/CISPO), Tool Use, Agentic RL, and distillation. 64M parameters, trained in 2 hours on a single RTX 3090, at a cost of about 3 RMB.

The Problem It Addresses

Most learners run inference with a few lines like AutoModelForCausalLM.from_pretrained(...) and fine-tune with SFTTrainer, but never learn what happens inside: how loss is computed, how attention is implemented, how RoPE encodes positions. As the author puts it:

> "Building a plane yourself out of Lego is far more exciting than flying first class."

What MiniMind Does

MiniMind is not a "model" — it is a complete from-scratch training pipeline. Key numbers:

| Dimension | Value | |---|---| | Main model size | 64M (Dense) / 198M-A64M (MoE) | | Model volume | 1/2700 of GPT-3 | | Training time | 2 hours (single RTX 3090, 1 epoch SFT) | | Training cost | ~3 RMB (rented GPU) | | Training stages | Pretrain → SFT → LoRA → DPO → PPO/GRPO/CISPO → Tool Use → Agentic RL → Distillation | | Dependencies | Native PyTorch; no transformers/trl/peft wrappers | | Architecture alignment | Qwen3 / Qwen3-MoE ecosystem |

Key design: all core algorithms (attention, RoPE, MoE routing, PPO, GRPO, CISPO, LoRA, distillation) are implemented from scratch in PyTorch. Every line of code is visible.

Complete Training Chain Covered

| Stage | From-scratch implementation | |---|---| | Pretrain | Autoregressive LM training, data cleaning and deduplication | | SFT | Instruction tuning, multi-turn dialogue format, chat_template | | LoRA | Low-rank adapters implemented without peft | | RLHF-DPO | Direct preference optimization from scratch | | RLAIF | PPO, GRPO, and CISPO all implemented from scratch | | Tool Use | Tool-calling capability mixed into SFT data | | Agentic RL | GRPO/CISPO in multi-turn tool-use scenarios | | Adaptive thinking | <think> tags with an open_thinking switch | | Distillation | Native white-box distillation | | YaRN | RoPE long-context extrapolation |

Notably, the RLAIF stage implements three algorithms: GRPO (used by DeepSeek in R1) and CISPO (a 2025 algorithm), alongside PPO.

Engineering Philosophy: Radical Simplicity

1. Minimal parameter count

64M parameters vs. GPT-3's 175B — a 2700x gap. This means:

Trainable on a single RTX 3090 (24GB), no 8x A100 needed
Model files are hundreds of MB, not terabytes
Inference runs even on CPU
Training finishes in 2 hours, not 2 weeks

A 64M model cannot do useful tasks, but the algorithms are the same as training a 175B model. You learn algorithms, not scale.

2. Minimal dependencies

No transformers, trl, or peft — only PyTorch. Attention is hand-written, RoPE is hand-written, LoRA is hand-written, PPO is hand-written. Every line can be read, modified, and debugged.

3. Minimal data

By default only two files are needed: pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl. No TB-scale data pipelines required.

Why 1005 Stars/Day

MiniMind hits two simultaneous needs:

1. The "from scratch" movement in LLM education. After DeepSeek-R1's open-source release, the community recognized that real understanding comes from implementing, not just calling packages. 2. Democratization of RLHF/RLAIF. GRPO and CISPO were paper-only topics for most engineers; MiniMind translates them into runnable PyTorch code on a model small enough to watch train on a laptop.

Comparison With Similar Projects

| Project | Params | Stages | From scratch | Ecosystem alignment | |---|---|---|---|---| | MiniMind | 64M | Pretrain→SFT→LoRA→DPO→PPO/GRPO/CISPO→Tool Use→Agentic RL→Distillation | ✅ All | Qwen3 | | nanoGPT | ~1M | Pretrain only | ✅ Partial | GPT-2 | | minGPT | ~1M | Pretrain only | ✅ Partial | GPT-2 | | lit-gpt | 7B+ | Pretrain→SFT | ❌ Uses Lightning | Llama | | LLaMA-Factory | Various | Full pipeline | ❌ Wrapper layer | Multi-model |

MiniMind is unique in combining "from-scratch implementation + full pipeline + frontier algorithms (GRPO/CISPO/Agentic RL)."

Bottom Line

MiniMind does not compete with Qwen3 — a 64M model cannot handle practical tasks. Its goal is to turn the full LLM training pipeline from a black box into a white box. "Radical simplicity" is an engineering decision: when you shrink scale, minimize dependencies, and simplify code, the essence of complex systems becomes visible. 3 RMB cannot buy a coffee, but it can buy a complete LLM training education.