Suppressing Misaligned Behaviors in Language Models, Durably

Summary

This paper studies how to suppress misaligned behaviors in language models and how durable that suppression is. The authors build two model organisms, one with a narrow trigger and one with a broad trigger, and compare SFT, GRPO and gradient-ascent unlearning on immediate suppression, durability under further training, and general capability. They introduce PEARL, a GRPO variant that augments rollouts with continuations from intermediate states along cached organism trajectories, which achieves strong suppression while preserving task accuracy across Qwen3-4B and GPT-OSS-20B.