English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Slime: The RL Training Framework Behind GLM Models - One Path for Training, Rollout, and Data

Forum topic · ✨步子哥 · 2026-08-16

Summary

Slime is an open-source RL post-training framework from Tsinghua's THUDM team, and it is the actual tool used to train GLM-5.2, GLM-5.1, GLM-5, GLM-4.7, GLM-4.6, and GLM-4.5. Its core design decision is that training (Megatron), rollout (SGLang), and data generation all flow through a single unified path, prioritizing debuggability over modularity. Unlike frameworks that build lowest-common-denominator abstractions across multiple backends, slime natively exposes Megatron and SGLang parameters directly. Data generation and reward computation plug into the unified training loop, supporting math, code, search, tool use, and multi-agent tasks without forking the training kernel. Slime also handles production-grade concerns like weight synchronization, fault recovery, rollout-only/train-only debug modes, and reproducibility. Beyond GLM, it supports Qwen, DeepSeek V3/R1, and Llama 3, signaling that RL post-training is becoming standardized engineering infrastructure rather than a research exercise.

Post-training large language models with reinforcement learning is an engineering problem, not an algorithmic one. The formulas behind RLHF/GRPO/PPO are clearly written in papers — the hard part is stitching together a trainer (e.g., Megatron), an inference engine (e.g., SGLang/vLLM), a data buffer, a reward module, and possibly an environment interaction module into one working RL training loop.

slime is the open-source RL post-training framework from Tsinghua's THUDM team. The models behind it may look familiar: GLM-5.2, GLM-5.1, GLM-5, GLM-4.7, GLM-4.6, GLM-4.5. This is not a "reference implementation" — it is the tool that actually trained these models.

Core Design: One Path, Three Roles

Slime's key design decision: training, rollout, and data generation all flow through the same path.

Many RL frameworks are "building-block" designs: the trainer is one service, the rollout engine another, data generation a third, communicating via APIs or message queues. This is modular but hard to debug — RL training bugs are usually silent (the model appears to be learning but is learning the wrong thing), and when data flows through multiple services, locating the source of a problem is painful.

Slime is "pipeline"-style: Megatron training, SGLang rollout, custom data generation, reward computation, verifier feedback, and environment interaction all flow through one unified training/rollout/data-buffer path.

The benefit is not performance (pipeline and building-block designs perform similarly) but debuggability: when all data flows through one path, you can see the complete data flow in one place — what the model generated, what score the reward function gave, what's in the buffer, which samples the trainer used to update weights.

Megatron + SGLang: Why These Two

Slime uses Megatron (training) and SGLang (inference) as core engines and exposes their parameters natively — no wrapping abstraction layer, just direct pass-through.

The philosophy: no lowest-common-denominator abstraction. Frameworks that support multiple training backends (Megatron, DeepSpeed, FSDP) and inference backends (vLLM, SGLang, TensorRT-LLM) must abstract a "universal trainer interface" and "universal inference interface." Since backend capabilities differ — some SGLang optimizations don't exist in vLLM, some Megatron parallelism strategies aren't in DeepSpeed — unification means taking the intersection of features, sacrificing each backend's unique strengths.

Slime doesn't do this. It supports one path (Megatron + SGLang) but exposes all engine parameters directly. You can use every advanced SGLang feature (e.g., --sglang- prefixed arguments) without waiting for framework adaptation.

Data Generation Freedom: Not Just Math and Code

RL post-training revolves around data. Different tasks need different generation approaches:

  • Math: problems + answer verifiers
  • Code: coding tasks + test cases
  • Search: search environments + answer matching
  • Tool use: tool APIs + call validation
  • Multi-agent: environments + interaction protocols
  • Many frameworks provide a specialized trainer per task type (math trainer, code trainer, agent trainer), which don't share infrastructure — adding a new task type means starting from scratch.

    Slime plugs data generation and reward computation into a unified training path as "plugins." Math, code, search, tools, sandboxes, verifiers, environments, multi-agent systems, long-horizon agentic workflows — all connect as data-generation or reward workflows without forking the training kernel. Adding a new task type requires only a data generation script and a reward function, no training-code changes.

    What "Battle-tested" Really Means

    Slime's README claims it is "one of the most battle-tested open RL post-training frameworks." The support for this claim: GLM-5.2, GLM-5.1, GLM-5, GLM-4.7, GLM-4.6, and GLM-4.5 were all trained with slime. This is a framework validated in real model release pipelines, not on toy datasets.

    "Battle-tested" means slime has survived every engineering problem RL training throws at it:

  • Weight synchronization: after the trainer updates weights, the rollout engine must load them. Bad sync timing means rollouts generate data with stale weights.
  • Fault recovery: RL training runs for days to weeks; node crashes are routine. Checkpointing must restore correct state.
  • Debug paths: slime supports rollout-only and train-only modes for debugging inference and training separately.
  • Reproducibility: RL training has many randomness sources (sampling, data shuffling, distributed training), requiring explicit logging of all random seeds.
  • These aren't "features" — they're "fixes born from pain." A framework that hasn't genuinely trained large models wouldn't anticipate these details.

    Supported Model Ecosystem

    Beyond the GLM series, slime also supports:

  • Qwen series: Qwen3.6, Qwen3.5, Qwen3Next, Qwen3MoE, Qwen3, Qwen2.5
  • DeepSeek V3 series: DeepSeek V3, V3.1, DeepSeek R1
  • Llama 3
  • This list shows slime is not a GLM-exclusive tool — it is a general-purpose RL post-training framework that happens to be developed and validated by the GLM team.

    RL Post-training as Infrastructure

    Slime points to a trend: RL post-training is moving from "research experiment" to "engineering infrastructure."

    Two years ago, RLHF meant reproducing algorithms from papers, writing your own training loop, and building your own rollout service. Now frameworks like slime package the entire pipeline: you provide the model and data generation logic; slime handles the training loop, rollout scheduling, weight synchronization, and fault recovery.

    This is analogous to what HuggingFace Transformers did for NLP — turning large model training from a research topic into an engineering task. The difference: slime focuses on RL post-training, not pretraining.

    As RL post-training infrastructure matures, more teams can run RL training experiments, potentially spawning new RL training methods — just as HuggingFace enabled more people to do pretraining experiments.

    ---

    Related links:

  • GitHub: https://github.com/THUDM/slime
  • Documentation: https://thudm.github.io/slime
  • GLM series: https://github.com/THUDM

Tags

#slime#rl-post-training#glm#megatron#sglang#rlhf#reinforcement-learning#training-framework

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178633566