Post-training large language models with reinforcement learning is an engineering problem, not an algorithmic one. The formulas behind RLHF/GRPO/PPO are clearly written in papers — the hard part is stitching together a trainer (e.g., Megatron), an inference engine (e.g., SGLang/vLLM), a data buffer, a reward module, and possibly an environment interaction module into one working RL training loop.
slime is the open-source RL post-training framework from Tsinghua's THUDM team. The models behind it may look familiar: GLM-5.2, GLM-5.1, GLM-5, GLM-4.7, GLM-4.6, GLM-4.5. This is not a "reference implementation" — it is the tool that actually trained these models.
Core Design: One Path, Three Roles
Slime's key design decision: training, rollout, and data generation all flow through the same path.
Many RL frameworks are "building-block" designs: the trainer is one service, the rollout engine another, data generation a third, communicating via APIs or message queues. This is modular but hard to debug — RL training bugs are usually silent (the model appears to be learning but is learning the wrong thing), and when data flows through multiple services, locating the source of a problem is painful.
Slime is "pipeline"-style: Megatron training, SGLang rollout, custom data generation, reward computation, verifier feedback, and environment interaction all flow through one unified training/rollout/data-buffer path.
The benefit is not performance (pipeline and building-block designs perform similarly) but debuggability: when all data flows through one path, you can see the complete data flow in one place — what the model generated, what score the reward function gave, what's in the buffer, which samples the trainer used to update weights.
Megatron + SGLang: Why These Two
Slime uses Megatron (training) and SGLang (inference) as core engines and exposes their parameters natively — no wrapping abstraction layer, just direct pass-through.
The philosophy: no lowest-common-denominator abstraction. Frameworks that support multiple training backends (Megatron, DeepSpeed, FSDP) and inference backends (vLLM, SGLang, TensorRT-LLM) must abstract a "universal trainer interface" and "universal inference interface." Since backend capabilities differ — some SGLang optimizations don't exist in vLLM, some Megatron parallelism strategies aren't in DeepSpeed — unification means taking the intersection of features, sacrificing each backend's unique strengths.
Slime doesn't do this. It supports one path (Megatron + SGLang) but exposes all engine parameters directly. You can use every advanced SGLang feature (e.g., --sglang- prefixed arguments) without waiting for framework adaptation.
Data Generation Freedom: Not Just Math and Code
RL post-training revolves around data. Different tasks need different generation approaches:
- Math: problems + answer verifiers
- Code: coding tasks + test cases
- Search: search environments + answer matching
- Tool use: tool APIs + call validation
- Multi-agent: environments + interaction protocols
- Weight synchronization: after the trainer updates weights, the rollout engine must load them. Bad sync timing means rollouts generate data with stale weights.
- Fault recovery: RL training runs for days to weeks; node crashes are routine. Checkpointing must restore correct state.
- Debug paths: slime supports rollout-only and train-only modes for debugging inference and training separately.
- Reproducibility: RL training has many randomness sources (sampling, data shuffling, distributed training), requiring explicit logging of all random seeds.
- Qwen series: Qwen3.6, Qwen3.5, Qwen3Next, Qwen3MoE, Qwen3, Qwen2.5
- DeepSeek V3 series: DeepSeek V3, V3.1, DeepSeek R1
- Llama 3
- GitHub: https://github.com/THUDM/slime
- Documentation: https://thudm.github.io/slime
- GLM series: https://github.com/THUDM
Many frameworks provide a specialized trainer per task type (math trainer, code trainer, agent trainer), which don't share infrastructure — adding a new task type means starting from scratch.
Slime plugs data generation and reward computation into a unified training path as "plugins." Math, code, search, tools, sandboxes, verifiers, environments, multi-agent systems, long-horizon agentic workflows — all connect as data-generation or reward workflows without forking the training kernel. Adding a new task type requires only a data generation script and a reward function, no training-code changes.
What "Battle-tested" Really Means
Slime's README claims it is "one of the most battle-tested open RL post-training frameworks." The support for this claim: GLM-5.2, GLM-5.1, GLM-5, GLM-4.7, GLM-4.6, and GLM-4.5 were all trained with slime. This is a framework validated in real model release pipelines, not on toy datasets.
"Battle-tested" means slime has survived every engineering problem RL training throws at it:
These aren't "features" — they're "fixes born from pain." A framework that hasn't genuinely trained large models wouldn't anticipate these details.
Supported Model Ecosystem
Beyond the GLM series, slime also supports:
This list shows slime is not a GLM-exclusive tool — it is a general-purpose RL post-training framework that happens to be developed and validated by the GLM team.
RL Post-training as Infrastructure
Slime points to a trend: RL post-training is moving from "research experiment" to "engineering infrastructure."
Two years ago, RLHF meant reproducing algorithms from papers, writing your own training loop, and building your own rollout service. Now frameworks like slime package the entire pipeline: you provide the model and data generation logic; slime handles the training loop, rollout scheduling, weight synchronization, and fault recovery.
This is analogous to what HuggingFace Transformers did for NLP — turning large model training from a research topic into an engineering task. The difference: slime focuses on RL post-training, not pretraining.
As RL post-training infrastructure matures, more teams can run RL training experiments, potentially spawning new RL training methods — just as HuggingFace enabled more people to do pretraining experiments.
---
Related links: