MiniCPM5-1B: ModelSense AI's AI-Written Training Framework ForgeTrain Pushes 1B-Parameter Limits

Forum topic · 小凯 · 2026-06-16

Summary

ModelSense AI (Biao Mian Zhihui), Tsinghua University, and OpenBMB have released MiniCPM5-1B, a 1.08B-parameter model trained with ForgeTrain, a training framework reportedly generated entirely by AI with zero human-written code. ForgeTrain claims ~10% higher training throughput than Megatron on NVIDIA H100 and than MindSpeed on Huawei Ascend, with Ascend adaptation completed in 3–5 days. MiniCPM5-1B achieves an AA-Index of 17.9, ranking first among sub-2B models, outperforms Qwen3.5-2B with half the parameters, supports 131K context and Think/No-Think modes, and posts a 42.57 composite score with strengths in agent tool calling, code generation, and competition math. The pipeline uses three-stage training (base, mid, post-training with SFT, RL, and online policy distillation), lifting math/code/instruction scores by 16 points while cutting overlong responses by 29 points. Open-sourced assets include Ultra-FineWeb-L3 (over 1T tokens of Chinese synthetic data), seven inference backends (vLLM, SGLang, llama.cpp, Ollama, etc.), five fine-tuning frameworks, and deployment across nine chip architectures via FlagOS.

ModelSense AI (Mianbi Zhihui), Tsinghua University, and OpenBMB have released MiniCPM5-1B, a 1.08B-parameter language model notable for being trained with ForgeTrain — a training framework reportedly generated entirely by AI, with zero human-written code. Source: OpenBMB/MiniCPM on GitHub.

Key Benchmarks

| Metric | Value | Notes | |--------|-------|-------| | Parameters | 1.08B | ~0.5GB after INT4 quantization | | AA-Index | 17.9 | First among sub-2B models | | vs Qwen3.5-2B | Better performance with half the parameters | | | Context length | 131K | vs 32K for Qwen3.5-0.8B and Gemma3 1B | | Reasoning modes | Think / No Think | Single checkpoint, switchable | | Composite average | 42.57 | 7 points above strongest same-size baseline (35.61) |

The model's largest advantages are in agent tool calling, code generation, and competition math — capabilities critical for on-device use.

ForgeTrain: An AI-Written Training Framework

Humans only define goals (e.g., "train a 1B model on Ascend 910B with FP8, target throughput X tokens/sec"); ForgeTrain then handles code generation, testing, and tuning in an AI-driven loop:

1. Analyze target hardware (memory bandwidth, compute, interconnect topology) 2. Generate tailored distributed strategies and operator implementations 3. Auto-compile, test, profile 4. Iterate based on feedback

Claimed performance:

| Platform | Baseline | Result | |----------|----------|--------| | NVIDIA H100 | Megatron | ~10% faster | | Huawei Ascend | MindSpeed | ~10% faster | | Ascend adaptation | Manual (weeks/months) | 3–5 days, zero human code |

The stated rationale: general frameworks carry compatibility burdens across all hardware and models, while ForgeTrain is "forged on the spot" for a specific hardware/model/scale combination.

Training Recipe

Three-stage pipeline: Base training → Mid training → Post-training (SFT → RL → OPD).

SFT: 200B tokens deep-thinking + 200B tokens hybrid-thinking data
RL: Domain-specific teacher models (math, code, closed-book QA, writing, instruction following), Reasoning RL on DAPO-Math-17k, two-stage length scheduling
OPD (online policy distillation): Distills RL teachers into one release model using reverse-KL advantage estimation and top-k logits

RL + OPD reportedly yields +16 points on math/code/instruction-following and a 29-point reduction in overlong responses.

Open-sourced datasets include Ultra-FineWeb, Ultra-FineWeb-L3 (over 1T tokens, among the largest Chinese synthetic pretraining corpora), UltraData-Math, and UltraData-SFT-2605. Technical reports: UltraData, Rethinking OPD.

Deployment Ecosystem

7 inference backends: vLLM, SGLang, llama.cpp, Ollama, LM Studio, MLX, ArcLight
5 fine-tuning frameworks: TRL+PEFT, LLaMA-Factory, ms-swift, unsloth, xtuner
Agent Skills for Cursor / Claude Code that auto-select deployment paths
9 chip architectures via FlagOS: NVIDIA, Hygon, MetaX, Iluvatar, Enflame, Moore Threads, Kunlunxin, Ascend, ARM-v9

A bonus project, MiniCPM-Desk-Pet, is a local-LLM desktop pet with LoRA personality switching.

Takeaways

1. AI auto-generating hardware-specific training frameworks could bypass the "one framework fits all hardware" compatibility trap. 2. Density over scale: with extreme data curation and post-training, 1B parameters can beat 2B models in practical usefulness. 3. Self-evolution flywheel: stronger models → better frameworks and data → stronger next-generation models, a recursive loop the team is targeting at the L4 self-improvement level.

Links: GitHub | HuggingFace | ModelScope