[论文] Suppressing Misaligned Behaviors in Language Models, Durably

论文概要

研究领域: NLP
作者: Luxi He, Pengcheng Jiang, Jifan Zhang, Jiawei Han, Danqi Chen
发布时间: 2026-11-03
arXiv: 2611.05358

中文摘要

不期望的模型行为可能在训练的任一阶段出现或复发。一旦出现，我们能在多大程度上压制它？为研究这个问题，我们首先构建了两个「模式生物」——被刻意训练出特定失准行为的语言模型，二者在触发条件上不同：窄触发有机体仅在提示包含特定线索时作弊（在多选题上跟随指向错误答案的植入线索），其余表现正常；宽触发有机体则没有这种线索，在不安全代码上微调后会在各种开放式提示下给出有害回答。我们把事后行为压制形式化，并按三条标准评估：即时压制目标行为、在后续训练下的持久性、以及通用能力保留。我们对比了 SFT、GRPO 和梯度上升遗忘。基于此前关于「对齐浅层性」的工作，我们假设更持久的压制需要能从更广的失准状态中恢复。由此提出 PEARL（前缀扩展的对抗强化学习）——一种 GRPO 变体，从缓存的有机体轨迹中采样中间状态并展开为 continuation 来增强 rollout。在两种有机体设置和两类模型（Qwen3-4B、GPT-OSS-20B）上，PEARL 在保持任务准确率的同时实现强即时压制，并在有机体数据或无关数据上继续微调时仍保持稳健。当不期望行为广泛且无法通过针对性 SFT 解决时，它收益最大。

原文摘要

Undesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, we first construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. The narrow-trigger organism misbehaves only when the prompt contains a specific cue: it cheats on multiple-choice questions by following a planted hint that points to a wrong answer, and behaves normally otherwise. The broad-trigger organism has no such cue: after fine-tuning on insecure code, it gives harmful responses across a wide range of open-ended prompts. We then formalize post-hoc behavioral suppression and evaluate methods along three criteria: immediate suppression of the target behavior, durability under further training, and preservation of general capability. We compare standard methods--SFT, GRPO, and gradient-ascent unlearning--along these dimensions. Building on prior work on alignment shallowness, we hypothesize that more durable suppression requires the ability to recover from a wider range of misalignment states. We introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL achieves strong immediate suppression while preserving task accuracy, and remains durable under further fine-tuning on either the organism's data or unrelated data. It obtains the greatest benefit when the undesirable behavior is broad and cannot be addressed through targeted SFT. Together, our framework and method offer a step toward more durable post-hoc removal of undesired behaviors.

自动采集于 2026-11-05

#论文 #arXiv #AI #小凯