> Repository: jingyaogong/minimind > Language: Python (native PyTorch) > License: Apache 2.0 > Trending: 1005 stars/day
In One Sentence
A complete LLM training project built from scratch without high-level wrappers like transformers/trl/peft — covering pretraining, SFT, LoRA, RLHF-DPO, RLAIF (PPO/GRPO/CISPO), Tool Use, Agentic RL, and distillation. 64M parameters, trained in 2 hours on a single RTX 3090, costing about 3 RMB (~$0.40).
The Problem It Addresses
Most learners can run inference with from_pretrained and fine-tune with SFTTrainer, but never learn what happens inside: how loss is computed, how attention is implemented, how RoPE encodes positions. You're *using* LLMs, not *understanding* them. The author puts it plainly:
> "Building a plane from LEGO bricks is far more exciting than flying first class."
What It Does
MiniMind is not just a model — it's a complete from-scratch training pipeline:
| Dimension | Value | |---|---| | Model size | 64M (Dense) / 198M-A64M (MoE) | | Scale | 1/2700 of GPT-3 | | Training time | 2 hours (single RTX 3090, 1 epoch SFT) | | Cost | ~3 RMB (GPU rental) | | Stages | Pretrain → SFT → LoRA → DPO → PPO/GRPO/CISPO → Tool Use → Agentic RL → Distillation | | Dependencies | Native PyTorch only | | Ecosystem | Aligned with Qwen3 / Qwen3-MoE |
All core algorithms (attention, RoPE, MoE routing, PPO, GRPO, CISPO, LoRA, distillation) are implemented from scratch in PyTorch.
Full Training Pipeline Coverage
| Stage | From-Scratch Implementation |
|---|---|
| Pretrain | Autoregressive LM training, data cleaning & dedup |
| SFT | Instruction tuning, multi-turn dialogue, chat_template |
| LoRA | Low-rank adapters without peft |
| RLHF-DPO | Direct preference optimization |
| RLAIF | PPO, GRPO, and CISPO all implemented |
| Tool Use | Tool-calling mixed into SFT data |
| Agentic RL | GRPO/CISPO in multi-turn tool-use settings |
| Adaptive Thinking | <think> tags + open_thinking switch |
| Distillation | Native white-box distillation |
| YaRN | RoPE long-context extension |
Notably, the RLAIF stage implements all three algorithms: GRPO (used in DeepSeek-R1) and CISPO (a 2025 algorithm), keeping the educational project at the research frontier.
Engineering Philosophy: Simplicity
1. Minimal parameters (64M): Trainable on one 3090 (24GB), a few hundred MB on disk, runs inference on CPU. The algorithms are identical to those used for 175B models — you learn the algorithms, not the scale.
2. Minimal dependencies: No transformers, trl, or peft. Attention, RoPE, LoRA, and PPO are all hand-written — every line is readable, modifiable, debuggable.
3. Minimal data: Two default files (pretrain_t2t_mini.jsonl and sft_t2t_mini.jsonl) — no TB-scale pipelines or distributed data loading.
Why 1005 Stars/Day
Two converging demands:
1. The "from scratch" movement in LLM education: Post-DeepSeek-R1, the community recognizes that true understanding comes from implementation, not package calls.
2. Democratization of RLHF/RLAIF: GRPO and CISPO were paper-only concepts for most engineers. MiniMind translates them into runnable PyTorch code on a model you can watch train on a laptop.
Comparison with Similar Projects
| Project | Size | Stages | From Scratch | Ecosystem | |---|---|---|---|---| | MiniMind | 64M | Pretrain→SFT→LoRA→DPO→PPO/GRPO/CISPO→Tool Use→Agentic RL→Distill | ✅ Full | Qwen3 | | nanoGPT | ~1M | Pretrain only | ✅ Partial | GPT-2 | | minGPT | ~1M | Pretrain only | ✅ Partial | GPT-2 | | lit-gpt | 7B+ | Pretrain→SFT | ❌ Lightning | Llama | | LLaMA-Factory | Various | Full pipeline | ❌ Wrapper | Multi |
MiniMind uniquely combines from-scratch implementation, full pipeline coverage, and frontier algorithms.
Takeaway
MiniMind doesn't compete with Qwen3 — a 64M model can't do useful tasks. Its goal is to turn the full LLM training pipeline from a black box into a white box. When you shrink the scale, minimize dependencies, and simplify the code, the essence of complex systems reveals itself. 3 RMB can't buy a coffee, but it can buy a complete LLM training education.