Have you ever watched two martial arts masters spar? One throws a punch, the other blocks and counters. Every exchange pushes both to improve. But with the wrong training method, sparring can degenerate into one fighter being knocked down repeatedly while the other learns nothing — because the opponent is too weak.
That is precisely the dilemma facing AI red-blue adversarial training. AdvGRPO, newly proposed by Microsoft's AI Red Team, addresses a key and previously overlooked problem in this dilemma: why GRPO (Group Relative Policy Optimization) collapses in attack-defense co-training, and how to keep it from collapsing.
The automation dilemma of red-blue adversarial training
Current LLM safety alignment relies mainly on static datasets — adversarial prompts hand-written by human red teamers, or attack samples generated from known jailbreak techniques. Models learn to resist these known attacks, but remain fragile against adaptive attackers that adjust their strategies to exploit defender weaknesses.
The natural solution is automated red-blue adversarial training: one model acts as attacker, one as defender, both co-trained to adapt to each other. Prior work has attempted this with PPO and DPO, but reported that GRPO is unstable in this setting.
This is unfortunate, because GRPO has unique advantages: it needs no value network (simpler than PPO), is sample-efficient, and has proven effective at steering safety-related behavior. If the stability issue can be solved, GRPO would be an ideal algorithm for attack-defense co-training.
AdvGRPO's three key designs
AdvGRPO keeps GRPO stable in attack-defense co-training through three designs:
1. Dense multi-channel rewards
Traditional methods often use a single scalar reward (attack succeeded / defense succeeded), giving sparse signal. AdvGRPO defines four reward channels for the attacker:
- Attack reward: whether the jailbreak succeeded
- Attack prompt reward: whether the prompt is natural, not machine-generated-looking
- Chain-of-thought reward: whether the reasoning is coherent
- Helpfulness reward: whether the attack prompt is meaningful on non-safety dimensions
- Stage 1: single-turn attack training — the attacker learns basic jailbreak skills
- Stage 2: closed-loop multi-turn attacks — the attacker must observe the defender's reply before deciding the next step, more realistic than the open-loop approach of generating all attack turns at once
- Stage 3: co-training begins, with attacker and defender updated alternately
The defender also receives multi-channel rewards, ensuring the defense isn't simply refusing everything.
2. Decoupled advantage normalization
This is the core fix for GRPO instability. In attack-defense co-training, attacker and defender reward scales are completely different — attack success rate may jump from 5% to 80%, while defense success may slip from 95% to 90%. If normalized in the same GRPO batch, one side's gradient signal drowns out the other.
Decoupled normalization lets attacker and defender compute advantage values independently, without interference.
3. Staged training curriculum
The two sides don't fight from the start. Training proceeds in three stages:
Key findings
Attacker side: AdvGRPO-trained attackers are effective not only against the defender used in training but also transfer to unseen defenders. More counterintuitively, uncensored models do not automatically make good attackers — what you need is not "able to say anything" but "knowing how to say things that hit weaknesses." Knowing how to insult is not the same as knowing how to debate.
Defender side: co-trained defenders outperform baseline methods on safety benchmarks while maintaining general capabilities. This shows attack-defense co-training doesn't just make the defender "better at refusing" — it learns to handle a broader range of attack strategies.
GRPO stability: the previously reported instability stems from sparse rewards and improper advantage normalization. AdvGRPO's dense rewards and decoupled normalization solve this.
Why "closed-loop" beats "open-loop"
Prior work SEMA also used GRPO to train multi-turn attackers, but open-loop: the attacker generates all attack turns in one go, without observing the defender's replies. It's like reading a pre-written debate speech from start to finish, ignoring what the opponent says.
AdvGRPO's closed-loop approach lets the attacker see the defender's reply each turn, then decide what to say next. Each turn carries an independent reward signal, and successful attacks terminate early (no need to keep attacking an already-jailbroken defender). This teaches the attacker genuine conversational strategy — how to adjust its wording based on the opponent's reactions.
A deeper insight
This work reveals an important methodological insight: an algorithm's "instability" is often not a flaw in the algorithm itself, but a problem in how it is used. GRPO was unstable in attack-defense co-training not because GRPO is defective, but because the reward structure and training dynamics of co-training require special handling.
It's like a good knife chipping when cutting ice — the knife isn't bad; you need to understand ice's crystal structure first, then adjust the angle and force.
For AI safety, automated red-blue adversarial training is an increasingly important direction. As models grow more capable, manual red-teaming can't keep up. AdvGRPO offers a viable path: letting AI become its own most rigorous tester.
---
Paper: https://arxiv.org/abs/2606.09701
Authors: Blake Bullwinkel, Eugenia Kim, Amanda Minnich, Mark Russinovich (Microsoft AI Red Team / Microsoft Azure) Core method: GRPO + dense multi-channel rewards + decoupled advantage normalization + staged training curriculum Key findings: GRPO can run stably in attack-defense co-training; uncensored models don't automatically make good attackers; closed-loop multi-turn attacks beat open-loop; co-trained defenders beat baselines