Molt: NVIDIA's 9,200-Line RL Framework That Scales to 1 Trillion Parameters
Change one loss function in a mainstream agent reinforcement learning framework, and how many places do you have to touch? Usually: one layer in the trainer, one in the distributed backend, one in the rollout glue—one change cuts through three layers. Agent RL research is fundamentally about constant iteration—new estimators, new pipeline stages, new sampling schemes—and every change pays that three-layer engineering tax. Half of research time goes to engineering overhead.
Molt, open-sourced by NVIDIA in June 2025, answers this question with line count: the full reinforcement learning codebase is about 9,200 lines. The repository accumulated 1,197 stars in four months, and the paper (on arXiv for three months) is already on its third revision. For comparison, verl is ~62K lines—Molt is 15% of it; slime is ~25K lines—Molt is 37%. The README puts it bluntly: you can read it in an afternoon.
Three Boxes, One Loop
Molt has only three components. Ray handles placement and asynchronous queues—it is the entire runtime. vLLM handles inference sampling. NVIDIA AutoModel + FSDP2 handles training, written in pure PyTorch. The three boxes form an asynchronous loop: agent sampling, training, and weight syncing overlap on the ring, with no component waiting on another.
The agent itself is an ordinary Python program. Write an Env.step() or ChatAgent.run(); the environment can contain graders, multi-turn tool use, visual environments, or LLM-as-judge—reward is any Python you care to write. It's compatible with OpenAI and Anthropic APIs, so existing agent code plugs in unmodified, with automatic context compression for long trajectories.
The keyword of this contract is token alignment: token ids, logprobs, action spans, rewards, and multimodal tensors produced by the sampling side stay matched end-to-end—from the vLLM queue into training. The paper includes a hard verification: on the 753B-parameter GLM-5.2, the log-probability gap between the sampling engine and the training engine is below 5×10⁻⁴, with zero importance-sampling filtering—the two engines speak the same probability language at model scale, with no tokens dropped, altered, or mismatched across versions.
Let the Numbers Talk
Small, yes—but the ledger deserves a line-by-line look.
One trillion parameters: the full asynchronous path—agent sampling, weight syncing, training updates—has run end-to-end on a 1T-parameter policy. Parallelism is pure configuration; the training code is still that same PyTorch. The README's example is DeepSeek-V3-scale MoE with expert parallelism at 256 ways and the largest actor using Adam with CPU offloading.
Continuous agent learning with 3B active parameters: Qwen3-30B-A3B (a 30B-total / 3B-active MoE) trained as a Python tool agent with up to ten tool calls, on 7.5K deduplicated DAPO-Math problems. After 1,464 updates, average accuracy across three benchmarks rose from 46.38% to 66.36%, stabilizing between 64 and 67 in the final phase. Final scores: AMC23 94.38%, Minerva 41.36%, AIME2025 63.33%.
Head-to-head with the Megatron stack: same model (Qwen3-30B-A3B), same 16 H100s, same asynchronous protocol—against slime (Megatron-Core + SGLang). Molt: 119.4±2.3 seconds per step; slime: 109.5±10.3 seconds, translating to Molt reaching roughly 92% of slime's throughput. The abstract calls it "statistically comparable"; the table's real number is 92%—trading 9,200 lines for Megatron-grade performance, and that 8% is the visible part of the bill.
One lineage worth noting: the paper itself states in related work that Molt originated from OpenRLHF. The latter is even smaller (~7.2K lines) but doesn't cover agentic scenarios or the 1T scale. Lead author Jian Hu is the creator of OpenRLHF, and this year also published FlashREINFORCE—already merged into Molt as a new recipe in September.
Small, But for Whom
Line count isn't an aesthetic question—it's a question of research economics. A framework is a researcher's harness, and the depth of cross-layer changes determines the tax on every experiment. Molt keeps agents in pure Python, compresses training to a single actor, and pushes distribution into three mature components—cutting the custom distributed middleware layer, the part you have to re-understand every time you change an algorithm.
The README carries a DeepWiki badge, meaning an AI can be turned loose on the repository directly. This move and the line count are two sides of the same coin: code small enough for AI to read whole is small enough for humans to read whole, moving the iteration bottleneck from engineering comprehension back to experiments itself. Production codebases treating "can AI navigate this" as a first-class design goal is spreading from the application layer into infrastructure—and research frameworks are the latest stop.
9,200 lines, 1 trillion parameters, 92% throughput. Those three numbers together are what this framework has to say.