Nemotron-Cascade 2: A 30B-Parameter MoE Model Reaching IMO, IOI, and ICPC Gold-Medal Level

Forum topic · 小凯 · 2026-03-21

Summary

Nemotron-Cascade 2 is an open-source 30B-parameter Mixture-of-Experts language model that activates only 3B parameters per token, yet reportedly achieves gold-medal-level performance on IMO 2025 (math), IOI 2025 (programming), and the ICPC World Finals. It is described as the second open model after DeepSeek-V3.2-Speciale (671B total, 37B activated) to reach this milestone, with roughly 1/12 the active parameters. Built on the Nemotron-Nano-V3 base with depth upscaling, the model uses a Cascade RL pipeline—sequential reinforcement learning stages covering math, code, agentic tasks, and alignment—combined with multi-domain on-policy distillation from the strongest intermediate teacher models in each domain. Carefully curated SFT data (AST-checked code, gap-filling synthesized math, LLM-judged general data) and progressive context extension to 128K tokens support the training recipe. This post explains the architecture and training methodology, compares benchmarks against larger models, and discusses implications for AI democratization, inference cost, and energy efficiency.

This post from zhichai.net discusses the paper *Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation* (arXiv:2603.19220), framing it as a modern David-vs-Goliath story: a 30B-parameter model (3B activated) reportedly matching models 20x its size.

Key points

Headline results: Nemotron-Cascade 2 is reported to reach gold-medal level on IMO 2025 (math), IOI 2025 (competitive programming), and the ICPC World Finals. It is described as the second open-source model to achieve this, after DeepSeek-V3.2-Speciale-671B-A37B — with only ~1/12 of DeepSeek's active parameters.
Architecture: A Mixture-of-Experts model built on the Nemotron-Nano-V3 base with 30B total / 3B activated parameters, extended via Layer-Predictor-based Depth Upscaling (DuS). Context is progressively expanded (4K → 32K → 128K tokens). Training stability techniques include gradient clipping, learning-rate scheduling, loss scaling, and model parallelism.
Stage 1 — Curated SFT: Rather than indiscriminate instruction data, the authors filter and synthesize per domain: AST analysis for code quality, gap-filling synthesis for math reasoning, and LLM-based quality scoring for general data, covering math, code, instruction following, dialogue, and tool use.
Stage 2 — Cascade RL: Instead of a single post-training RL pass, capabilities are built sequentially: (1) math reasoning, (2) code generation, (3) agentic abilities (tool use, multi-step reasoning), (4) alignment/safety. Each stage has dedicated reward models and data, improving stability and allowing per-domain mastery.
Stage 3 — Multi-domain on-policy distillation: During each RL stage, the model distills from the strongest intermediate teacher for that domain (e.g., math teacher for math, coding teacher for code). On-policy (live) distillation helps recover performance if an RL stage regresses on other benchmarks and transfers tacit knowledge from larger models.
Benchmarks (as reported in the post): MMLU 86.2, MMLU-Pro 78.3, GPQA Diamond 72.1, AIME 2025 81.8, LiveCodeBench v6 74.3 — reportedly surpassing Qwen3.5-35B and approaching Nemotron-3-Super-120B. A paper co-author (IMO 2015 gold medalist) personally verified the model's IMO solutions.
Efficiency claims: The post estimates ~30x better inference energy efficiency versus GPT-4-class dense models (~0.3Wh vs ~10Wh per query) and argues 3B active parameters enable deployment on consumer GPUs, high-end phones, and edge devices.
Acknowledged limitations: long-range dependency understanding, multilingual (especially low-resource) coverage, and creative tasks may still trail larger general-purpose models.

References cited

1. Yang, Z., et al. (2026). *Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation*. arXiv:2603.19220. 2. Yang, Z., et al. (2025). *Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models*. arXiv:2512.13607. 3. Liu, A., et al. (2024). *DeepSeek-V3 Technical Report*. arXiv:2412.19437. 4. Shazeer, N., et al. (2017). *Outrageously large neural networks: The sparsely-gated mixture-of-experts layer*. ICLR 2017. 5. Hinton, G., Vinyals, O., & Dean, J. (2015). *Distilling the knowledge in a neural network*. arXiv:1503.02531.

Takeaway

The post argues that parameter count is not the sole determinant of capability: staged training, multi-domain distillation, and MoE efficiency can deliver elite reasoning at a fraction of the compute, pointing toward a more sustainable and democratized AI landscape.