Background
Post-training of large language models (LLMs) is widely described as an engineering challenge rather than an algorithmic one. The mathematics behind RLHF, GRPO, and PPO is well-documented; the difficulty lies in stitching together the required components into a single working reinforcement learning loop. A typical pipeline must coordinate a training framework (e.g., Megatron) for weight updates, an inference engine (e.g., SGLang or vLLM) for rollout generation, a data buffer for sample management, a reward module for scoring, and optionally an environment interaction module for agentic tasks.
slime is the open-source RL post-training framework released by Tsinghua's THUDM group. It is not a reference implementation: it is the actual tool used to train GLM-4.5, GLM-4.6, GLM-4.7, GLM-5, GLM-5.1, and GLM-5.2.
Core Design: One Path, Three Roles
The central design decision in slime is that training, rollout, and data generation travel the same path.
Many RL frameworks adopt a "modular service" architecture in which the trainer, rollout engine, and data generator are independent services communicating via APIs or message queues. While this approach improves modularity, it makes debugging painful because RL training bugs are usually silent — the model appears to be learning while actually learning the wrong objective. Locating the source of such bugs across multiple services is difficult.
slime instead uses a "pipeline" design. Megatron training, SGLang rollout, custom data generation, reward computation, verifier feedback, and environment interaction all flow through the same training/rollout/data-buffer path. The advantage is not throughput (pipeline and modular designs perform similarly) but debuggability: the complete data flow is visible at a single observation point, showing what the model generated, what scores the reward function assigned, what the buffer stored, and which samples updated the weights.
Why Megatron + SGLang
slime commits to Megatron for training and SGLang for inference, and exposes their native parameters directly rather than wrapping them in an abstraction layer.
This commitment reflects a philosophy of avoiding the lowest-common-denominator abstraction. Frameworks that aim to support multiple training backends (Megatron, DeepSpeed, FSDP) and multiple inference backends (vLLM, SGLang, TensorRT-LLM) typically introduce a generic "Trainer interface" and "Inference interface." Because backend capabilities differ, the abstraction can only expose the intersection of features supported everywhere. This sacrifices the unique capabilities of each backend.
slime rejects this trade-off. It supports only the Megatron + SGLang combination, but exposes every SGLang advanced feature (including the --sglang- prefixed arguments) directly, without waiting for framework-level adaptation.
Data Generation Flexibility: Beyond Math and Code
RL post-training is fundamentally about training data, and different tasks require different generation strategies:
- Math: math problems + answer verifiers
- Code: coding problems + test cases
- Search: search environments + answer matching
- Tool use: tool APIs + invocation verification
- Multi-agent: environment + interaction protocols
- Weight synchronization: after the trainer updates weights, the rollout engine must load them. Incorrect timing causes rollouts generated from stale weights.
- Failure recovery: RL training often runs for days or weeks, and node crashes are routine. Checkpointing must restore the system to a consistent state.
- Debugging paths: slime supports rollout-only and train-only modes, allowing inference and training to be debugged independently.
- Reproducibility: RL training has many randomness sources (sampling, data shuffling, distributed training). All random seeds must be recorded explicitly.
- Qwen series: Qwen2.5, Qwen3, Qwen3MoE, Qwen3Next, Qwen3.5, Qwen3.6
- DeepSeek V3 series: DeepSeek V3, V3.1, DeepSeek R1
- Llama 3
- GitHub: https://github.com/THUDM/slime
- Documentation: https://thudm.github.io/slime
- GLM series: https://github.com/THUDM
Many frameworks provide specialized trainers for each task type — a math trainer, a code trainer, an agent trainer — but these trainers do not share infrastructure, and adding a new task type requires re-implementing from scratch.
In slime, data generation and reward computation plug into the unified training path as plugins. Math, code, search, tool use, sandbox, verifier, environment, multi-agent systems, and long-horizon agentic workflows are all attached as data-generation or reward workflows without forking the training kernel. Adding a new task type requires writing only a data-generation script and a reward function — no training code changes.
What "Battle-Tested" Really Means
The slime README describes it as "one of the most battle-tested open RL post-training frameworks." The supporting evidence is that GLM-4.5 through GLM-5.2 were all trained with slime — not toy datasets but full model release pipelines.
"Battle-tested" in this context means the framework has survived the engineering issues that surface in real RL training:
These are not "features" in the marketing sense; they are fixes written after real failures. A framework that has not actually trained large-scale models tends to overlook these details.
Supported Model Ecosystem
Beyond the GLM series, slime supports:
This breadth demonstrates that slime is a general-purpose RL post-training framework developed and validated by the GLM team, not a GLM-only tool.
RL Post-Training as Infrastructure
slime points to a broader trend: RL post-training is transitioning from a "research experiment" to "engineering infrastructure."
Two years ago, RLHF required re-implementing algorithms from papers, writing custom training loops, and building rollout services from scratch. Frameworks like slime now package the entire pipeline: developers supply the model and data-generation logic, and slime handles the training loop, rollout scheduling, weight synchronization, and failure recovery.
This mirrors the role HuggingFace Transformers played for NLP pretraining — converting "research project" into "engineering task." The key difference is that slime targets RL post-training rather than pretraining. As RL post-training infrastructure matures, more teams can run RL training experiments, which is likely to catalyze new RL training methods in the same way HuggingFace expanded pretraining research.
---
Related Links