Addition by Subtraction: Toward Mitigating Fragile Safety Alignment in Large Language Models

Summary

Researchers study how safety alignment in large language models depends on a small set of safety-critical neurons and a single refusal direction, so pruning or ablating them can undermine aligned behavior. They propose Addition by Subtraction, which gradually removes identified safety-critical neurons during finetuning and uses preference optimization to restore safe behavior through other model components. Across four models from three vendors, the resulting models resist pruning-based and black-box jailbreak attacks better, with attack success rates reduced by up to 90% for some attacks.

Mitigation

Addition by Subtraction: gradually remove identified safety-critical neurons during finetuning and use preference optimization to re-establish safety-aligned behavior through other parts of the model.

Related items

• MediumResearchers Uncover Prompt Adware Hidden in OpenAI Codex Jailbreak ToolkitSimilar attack · The Hacker News
• InfoSafeStart: Mitigating the Overlooked Safety Cost of Active Expert Reduction in MoE LLM ServingSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• MediumCritical Langflow flaws enable authentication bypass and RCESimilar attack · CSO Online
• MediumFree the Model, Sell the Gateway: Prompt Adware hidden in Jailbreak BundlesSimilar attack · Check Point Research
• InfoTraceRouter++: Causal Semantic Routing Realignment via Optimal TransportSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)