Small Model, Big Power: How a "Mini" AI Defeats the Giants

> *Understanding Nemotron-Cascade 2: conquering math olympiad gold with 3 billion active parameters*

Prologue: When a Lightweight Steps into the Heavyweight Ring

In the summer of 2025, something stunning happened at the International Mathematical Olympiad (IMO). It wasn't a record-breaking human prodigy — it was an AI model, Nemotron-Cascade 2, achieving gold-medal-level performance.

The most surprising part: the model has only 3 billion active parameters. By comparison, DeepSeekV3.2-Speciale — the other gold-level IMO model — has 671 billion total parameters with 37 billion active. In other words, Nemotron-Cascade 2 matched the achievement with less than one-tenth of the "brain capacity," like a lightweight boxer beating a heavyweight champion.

How did it do it?

Chapter 1: The Parameter War — Is Bigger Really Better?

The AI field has long assumed bigger models are smarter. But giant models bring serious costs:

• Cost: training and running them is extremely expensive
• Speed: slow inference degrades user experience
• Barrier: even large companies can't afford the biggest models
• Environment: training emissions rival those of a small city

The research team proposes a new concept: Intelligence Density — how much intelligence you get per unit of parameters, like comparing fuel efficiency rather than tank size.

With only 3B active parameters (30B total, ~10% activated per token), Nemotron-Cascade 2 achieved:

• IMO 2025: gold-level (35 points)
• IOI 2025: gold-level (439.28 points)
• ICPC 2025 World Finals: 10/12 problems solved (gold-level)

It is only the second open-weight model after DeepSeekV3.2 to reach gold level in all three competitions — with 1/20th the parameter count.

Chapter 2: MoE — Not Every Neuron Needs to Work

The first secret weapon is Mixture-of-Experts (MoE) architecture.

In traditional "dense" models, every input passes through all parameters — like requiring a full-staff hospital consultation for every patient. In MoE, parameters are divided into groups called "experts," and a router network acts as a triage desk: it examines the input and activates only the most relevant experts with appropriate weights.

Nemotron-Cascade 2 has 30B total parameters but activates only 3B (~10%) per input — like a hospital with 100 specialists where only 10 consult per case. Benefits include:

• Efficiency: only relevant parameters compute
• Capacity: experts can specialize
• Scalability: more experts don't linearly increase inference cost

MoE is hard to train, though: load balancing (avoiding router collapse onto a few experts), training instability, and memory access (all expert parameters must reside in memory even if unused — you must "employ" all 100 specialists even if only 10 work at a time).

Chapter 3: Cascade RL — Training Step by Step

Reinforcement learning (RL) trains models through reward signals rather than imitation of fixed answers — like training a dog with treats rather than step-by-step instructions.

Cascade RL applies RL in progressive stages, like learning piano from scales to sonatas:

1. Supervised Fine-Tuning (SFT) — build foundations from high-quality annotated data across math, coding, science, tool use, and dialogue 2. Instruction-Following RL (IF-RL) — strict adherence to user instructions 3. Multi-domain RL — STEM multiple-choice, agent tool calls, structured output 4. Math RL — deep math reasoning with verifiable rewards 5. Code RL — verified via execution results 6. Software Engineering RL (SWE RL) — real-world tasks like fixing GitHub bugs

Why not mix everything at once? Cascade RL:

• Avoids catastrophic forgetting — each stage consolidates prior knowledge
• Enables fine-grained optimization — each task type gets a tailored strategy (math verifies instantly; code needs execution; SWE needs environment interaction)
• Provides progressive difficulty — the model progresses stably without "crashing" on overly hard tasks

Chapter 4: Multi-Domain On-Policy Distillation — Teachers Guide Students

The third weapon is Multi-Domain On-Policy Distillation (MOPD), building on Hinton et al.'s 2015 knowledge distillation. Instead of only learning hard labels, the student learns the teacher's "soft labels" — full probability distributions conveying which answers are almost right.

MOPD's twist: during Cascade RL, each stage produces a specialist "intermediate model" (the math-stage model excels at math; the code-stage model at coding). MOPD distills knowledge from the best model of each prior stage, restoring capabilities that may have regressed — like an athlete training simultaneously with endurance, technique, and tactics coaches.

It's also on-policy: distillation happens *during* the RL process, so teachers and students evolve together, enabling timely repair of lost abilities, dynamic improvement, and training stability.

Chapter 5: The Stunning Scorecard

Math reasoning

• IMO 2025: 35 points (gold level; medals typically require ~28–35 of 42)
• AIME 2025: 92.4% accuracy (98.6% with tool use)

Competitive programming

• IOI 2025: 439.28 points (gold level)
• ICPC 2025 World Finals: 10/12 problems (gold level)
• LiveCodeBench v6: 87.2 (88.4 with tools)

Other capabilities

• Alignment/instruction: ArenaHard v2 83.5, IFBench 82.9
• Long context: NIAH@1M (RULER Subset) 99.0, with a 1M-token context window
• Agents: SWE Verified (OpenHands) 50.2, TerminalBench 2.0 21.1%

Chapter 6: Democratizing AI — The Era of Small Models?

Nemotron-Cascade 2's success may signal a paradigm shift:

• Cost democratization: 3B active parameters means far cheaper inference for SMEs, researchers, and individual developers
• Local deployment: quantized, it runs on consumer GPUs like the RTX 4090/5090 — no cloud API, data stays local, works offline
• Environmental friendliness: far better energy efficiency

It also marks NVIDIA formally entering the open-weight model race, previously led by Chinese companies (Qwen, DeepSeek) and Mistral — likely motivated by GPU demand, ecosystem building, and competition with Chinese open models. Either way, more open models mean more choice, faster innovation, and lower barriers.

The field may shift from "who has the biggest model" to "who achieves the best results with the fewest resources" — through architecture innovation (MoE, state-space models), training methods (Cascade RL, distillation), inference optimization (quantization, speculative decoding, caching), and specialized hardware. Nemotron-Cascade 2 proves small models can have big wisdom — possibly just the start of an efficiency revolution.

Epilogue: What Is Intelligence, Really?

True intelligence may lie not in how many resources you have, but how you use them. A 3-billion-parameter model with clever architecture and training can defeat giants with tens of billions — just as the human brain's ~86 billion neurons accomplish tasks no AI can match. We may not need to wait for ever-larger supercomputers; with smarter methods, we can do remarkable things now.

Small body, big energy — a description of one model, and perhaps a prophecy of AI's future direction.

References

1. Yang, Z., et al. (2026). Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation. arXiv:2603.19220. 2. Chen, Y., et al. (2025). Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models. arXiv:2512.13607. 3. Shazeer, N., et al. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR. 4. Guo, D., et al. (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948. 5. Hinton, G., Vinyals, O., & Dean, J. (2015). Distilling the Knowledge in a Neural Network. arXiv:1503.02531.

*This article is an accessible explainer of the Nemotron-Cascade 2 paper; in case of discrepancies, refer to the original paper.*