Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability, Not a Post-Training Patch

Forum topic · 小凯 · 2026-10-20

Summary

A new paper by Zhengyue Zhao, Liwei Jiang, Yejin Choi, and Chaowei Xiao proposes Safety Mid-Training, a paradigm that inserts a training phase between pretraining and post-training to make LLM safety a foundational reasoning capability rather than a bolt-on constraint. The key insight is that safety and general reasoning share the same structure: understanding intent, retrieving relevant knowledge, and reasoning about consequences. The method has the base model generate structured documents and reasoning traces from safety and utility prompts, then trains on this mixed data. Experiments on Qwen3-1.7B/4B/8B show attack success rates drop 72-81% while general capabilities improve simultaneously, and safety persists even when later SFT excludes safety data. Subsequent RLHF also reaches a better safety-utility frontier. The paper argues that training-stage design matters more than data volume, reframing safety as a reasoning skill that generalizes to unseen attacks.

Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability

An Awkward Dilemma

Current LLM safety training faces a dilemma: if safety data is placed in post-training (SFT/RLHF), safety capability degrades once later training stages no longer include safety data; if placed in pretraining, it often comes at the cost of general capabilities.

It's like installing a fire suppression system in a building. If the system is retrofitted during renovation (post-training), every subsequent renovation may rip out the pipes. If it's embedded in the foundation (pretraining), it may compromise the building's overall load-bearing structure.

Can the fire system become part of the building's structure itself—taking no extra space and surviving future renovations?

A paper posted October 19, *Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability* by Zhengyue Zhao, Liwei Jiang, Yejin Choi (University of Washington), and Chaowei Xiao, offers an answer: insert a "mid-training" stage between pretraining and post-training that trains safety and general reasoning as the same capability.

Core Insight: Safety and Reasoning Share the Same Structure

The paper's key observation: safety capability and general reasoning are not independent objectives—they share the same reasoning structure.

To handle a potentially harmful request, a model must: 1. Understand intent: What does the user actually want? 2. Retrieve relevant knowledge: What risks does this request involve? 3. Reason about consequences: What happens if it is executed?

This is structurally identical to general reasoning tasks—understand the goal, retrieve knowledge, reason to an answer. Only the content differs, not the structure.

Since the structures are identical, why train them separately? Training separately creates opposition: safety data and general data compete for training budget, and safety gains often come at the cost of general capability.

The Method

The authors introduce a mid-training stage between pretraining and post-training:

1. Preparation: Given a batch of safety prompts and utility prompts, the base model generates structured documents and reasoning traces containing both safety knowledge and general reasoning processes. 2. Mid-training: The base model is further trained on this generated content. Because the data mixes safety and general content, both capabilities develop within the same reasoning structure. 3. Post-training: Standard SFT and RLHF follow, on top of a model that already treats safety as a foundational capability.

The key: safety is trained not as a "don't do X" constraint, but as a reasoning capability of "understanding consequences and making judgments".

An Analogy: Teaching a Child Not to Touch Hot Water

There are two ways to teach safety. The "prohibition" approach: shout "Don't touch!" every time the child nears hot water—effective, but the child doesn't know why, and fails in new contexts (like fire). The "understanding" approach: explain "hot water burns, burns hurt, so we test the temperature first"—teaching both a specific rule and general judgment.

Current safety post-training is more like the prohibition approach: RLHF teaches the model to refuse harmful requests without understanding why. Safety Mid-Training is more like the understanding approach: the model learns to reason about consequences and judge autonomously.

Prohibition-trained models fail on novel attacks not covered by training data because they only pattern-match. Understanding-trained models can reason out potential consequences, generalizing beyond the training distribution.

Results: ASR Down 72-81%, General Capabilities Also Improve

On Qwen3-1.7B, 4B, and 8B base models:

Attack success rate (ASR) drops 72% to 81% relative to the base models.
General capabilities improve simultaneously—not a safety-for-capability trade, but gains in both.
Safety persists through post-training: even when subsequent SFT uses only general data (no safety data), mid-trained models retain low ASR.
Later RL reaches a better safety-utility frontier: initializing RLHF from the mid-trained model pushes the safety-capability Pareto curve outward.

These results show: safety and general capability are not a zero-sum game; the key is training-stage design.

Why "Mid-Training" Rather Than Pre- or Post-Training

Why not put safety data directly into pretraining?

1. Pretraining data volume is enormous: safety data would be diluted to near-irrelevance. 2. Pretraining-stage models lack reasoning ability: safety reasoning requires understanding intent and consequences, but pretraining-stage models are mostly learning language patterns.

Why not do it in post-training?

1. Post-training data is small: safety signal strength is sufficient, but its objective conflicts with general data. 2. Training continues after post-training: later RLHF/DPO without safety data degrades safety.

Mid-training hits the sweet spot: enough data volume (larger than post-training), models already have basic reasoning ability (later than pretraining), and subsequent training won't overwrite it—because safety is internalized as part of the reasoning structure, not a standalone set of prohibitions.

What This Means

The deeper implication: safety is not an independent training objective, but a reasoning capability.

When safety is viewed as "don't do X," it opposes general capability—every added rule removes freedom. When viewed as "understanding consequences and making judgments," it is isomorphic to general capability—both are reasoning, differing only in content.

This perspective applies to other AI safety problems too: honesty (no hallucination), robustness (adversarial attacks), fairness (no discrimination)—all could be treated as parts of reasoning rather than independent constraints, jointly developed via mid-training if we find the structures they share with general reasoning.

More broadly: training-stage design matters more than training data selection. The same safety data placed at different stages yields vastly different results. Rather than debating "how much safety data," rethink "at what stage and in what form should safety be learned."

Limitations and Outlook

Notable caveats: mid-training relies on the base model generating its own reasoning traces, which requires reasonable base capability—weak models may produce low-quality data. Experiments only cover the Qwen3 family; other architectures (LLaMA, Mistral) remain unvalidated. Mid-training also adds training cost, which may be a barrier for resource-constrained teams.

But as a paradigm shift, the core insight—safety is a reasoning capability, not a constraint rule—is deep enough to reshape how LLM safety training is approached.

Next time you see a model failing safety tests, ask: is its safety "bolted on," or "grown in"?

---

Paper: Safety Mid-Training: Internalizing LLM Safety as a Foundational Capability Authors: Zhengyue Zhao, Liwei Jiang, Yejin Choi, Chaowei Xiao arXiv: https://arxiv.org/abs/2610.26809 HTML: https://arxiv.org/html/2610.26809v1