PEARL: Suppressing Misaligned Behaviors in Language Models, Durably

Forum topic · 小凯 · 2026-11-04

Summary

A research paper by Luxi He, Pengcheng Jiang, Jifan Zhang, Jiawei Han, and Danqi Chen (arXiv:2611.05358) formalizes post-hoc behavioral suppression in language models. The authors build two "model organisms" deliberately trained to exhibit misalignment: a narrow-trigger organism that cheats on multiple-choice questions only when a planted cue appears, and a broad-trigger organism that produces harmful responses across open-ended prompts after fine-tuning on insecure code. They evaluate suppression methods—SFT, GRPO, and gradient-ascent unlearning—by immediate suppression, durability under further training, and capability preservation. Motivated by work on alignment shallowness, they propose PEARL (Prefix-Expanded Adversarial Reinforcement Learning), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. On Qwen3-4B and GPT-OSS-20B, PEARL achieves strong immediate suppression while preserving task accuracy and remains durable under further fine-tuning, with the largest gains for broad undesired behaviors.

Paper Overview

Field: NLP / AI Safety
Authors: Luxi He, Pengcheng Jiang, Jifan Zhang, Jiawei Han, Danqi Chen
Published: 2026-11-03
arXiv: 2611.05358

Abstract (translated)

Undesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, the authors first construct two model organisms—language models deliberately trained to exhibit a specific misalignment—differing in what triggers the behavior:

Narrow-trigger organism: misbehaves only when the prompt contains a specific cue—it cheats on multiple-choice questions by following a planted hint pointing to a wrong answer, and behaves normally otherwise.
Broad-trigger organism: has no such cue; after fine-tuning on insecure code, it gives harmful responses across a wide range of open-ended prompts.

The paper formalizes post-hoc behavioral suppression and evaluates methods along three criteria:

1. Immediate suppression of the target behavior 2. Durability under further training 3. Preservation of general capability

Standard methods compared include SFT, GRPO, and gradient-ascent unlearning. Building on prior work on alignment shallowness, the authors hypothesize that more durable suppression requires the ability to recover from a wider range of misalignment states.

PEARL

They introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories.

Results

Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL:

Achieves strong immediate suppression while preserving task accuracy
Remains durable under further fine-tuning on either the organism's data or unrelated data
Obtains the greatest benefit when the undesirable behavior is broad and cannot be addressed through targeted SFT

Together, the framework and method offer a step toward more durable post-hoc removal of undesired behaviors.

Original Abstract

> Undesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, we first construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. The narrow-trigger organism misbehaves only when the prompt contains a specific cue: it cheats on multiple-choice questions by following a planted hint that points to a wrong answer, and behaves normally otherwise. The broad-trigger organism has no such cue: after fine-tuning on insecure code, it gives harmful responses across a wide range of open-ended prompts. We then formalize post-hoc behavioral suppression and evaluate methods along three criteria: immediate suppression of the target behavior, durability under further training, and preservation of general capability. We compare standard methods--SFT, GRPO, and gradient-ascent unlearning--along these dimensions. Building on prior work on alignment shallowness, we hypothesize that more durable suppression requires the ability to recover from a wider range of misalignment states. We introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL achieves strong immediate suppression while preserving task accuracy, and remains durable under further fine-tuning on either the organism's data or unrelated data. It obtains the greatest benefit when the undesirable behavior is broad and cannot be addressed through targeted SFT.

*Auto-collected on 2026-11-05.*