Have you seen two martial arts masters sparring? One throws a punch, the other blocks, then counterattacks. Every exchange pushes both to improve. But with the wrong training method, the sparring degenerates: one side gets knocked down repeatedly, and the other learns nothing—because the opponent is too weak.
This is exactly the dilemma facing AI red-blue adversarial training. Microsoft AI Red Team's newly proposed AdvGRPO framework solves a key overlooked problem: why GRPO (Group Relative Policy Optimization) collapses in attack-defense co-training, and how to prevent it.
The Automation Dilemma of Red-Blue Adversaries
Current LLM safety alignment relies mainly on static datasets—adversarial prompts hand-written by human red teamers, or attack samples generated from known jailbreak techniques. Models learn to resist these known attacks, but defenses remain fragile against adaptive attackers that adjust strategies based on the defender's weaknesses.
The natural solution is automated red-blue adversaries: one model attacks, one defends, and both co-train and adapt to each other. Prior work has attempted such co-training with PPO and DPO, but reports noted GRPO is unstable in this setting.
That's a pity, because GRPO has unique advantages: no value network (simpler than PPO), high sample efficiency, and proven effectiveness in steering safety-related behavior. If the stability issue can be fixed, GRPO is an ideal algorithm for attack-defense co-training.
AdvGRPO's Three Key Designs
AdvGRPO keeps GRPO stable in co-training through three core designs:
1. Dense multi-channel rewards
Traditional methods use a single scalar reward (attack succeeded or defense succeeded), giving sparse signals. AdvGRPO gives attackers four reward channels:
- Attack reward: whether the jailbreak succeeded
- Attack prompt reward: whether the prompt is natural, not machine-like
- Chain-of-thought reward: whether the reasoning is sound
- Helpfulness reward: whether the attack prompt is meaningful on non-safety dimensions
- Stage 1: single-turn attack training—attacker learns basic jailbreak skills
- Stage 2: closed-loop multi-turn attacks—the attacker must observe the defender's reply before deciding the next move (more realistic than the open-loop approach of generating all turns at once)
- Stage 3: co-training begins, with attacker and defender updating alternately
Defenders also receive multi-channel rewards, ensuring they don't just refuse everything.
2. Decoupled advantage normalization
This is the core fix for GRPO instability. In co-training, attacker and defender reward scales differ completely—attack success rate might jump from 5% to 80%, while defense success rate might slip from 95% to 90%. Normalizing both in the same GRPO batch means one side's gradient signal drowns out the other. Decoupled normalization lets attacker and defender compute advantages independently, without interference.
3. Staged training curriculum
Rather than starting the duel immediately, training proceeds in three stages:
Key Findings
Attacker side: AdvGRPO-trained attackers transfer to unseen defenders, not just the defender used in training. More counterintuitively, uncensored models do not automatically make good attackers—what matters isn't "saying anything" but "knowing how to hit weaknesses." Being able to insult someone isn't the same as being able to debate.
Defender side: Co-trained defenders outperform baseline methods on safety benchmarks while retaining general capabilities. This shows co-training doesn't just make defenders "better at refusing"—it teaches them to handle a broader range of attack strategies.
GRPO stability: The previously reported instability stems from sparse rewards and improper advantage normalization. AdvGRPO's dense rewards and decoupled normalization solve it.
Why "Closed-Loop" Beats "Open-Loop"
Earlier work (SEMA) also trained multi-turn attackers with GRPO, but open-loop: the attacker generates all turns at once without observing the defender's replies—like reading a prepared debate script regardless of what the opponent says.
AdvGRPO's closed-loop approach lets the attacker see the defender's reply each turn and decide the next move. Each turn gets its own reward signal, and successful attacks terminate early (no need to keep attacking an already jailbroken defender). This teaches the attacker genuine conversational strategy—adjusting phrasing based on the opponent's reactions.
The Deeper Insight
The work reveals an important methodological lesson: an algorithm's "instability" is often not the algorithm's fault but a usage problem. GRPO was unstable in attack-defense co-training not because GRPO is flawed, but because the reward structure and training dynamics of co-training require special handling.
It's like a good knife chipping when cutting ice—the knife isn't bad; you need to understand ice's crystal structure and adjust angle and force.
For AI safety, automated red-blue adversarial training is increasingly important. As models grow more capable, manual red teaming can't keep up. AdvGRPO offers a viable path: letting AI become its own strictest tester.
---
Paper: https://arxiv.org/abs/2606.09701
Authors: Blake Bullwinkel, Eugenia Kim, Amanda Minnich, Mark Russinovich (Microsoft AI Red Team / Microsoft Azure) Core method: GRPO + dense multi-channel rewards + decoupled advantage normalization + staged training curriculum Key findings: GRPO can run stably in attack-defense co-training; uncensored models don't automatically make good attackers; closed-loop multi-turn attacks beat open-loop; co-trained defenders outperform baselines