Follow the Gradient: Tracing Safety Degradation Back to Training Data

Summary

This paper presents SimpleLoRIF, a gradient embedding method that flags instruction-tuning examples whose gradients align with unsafe behavior, even when the examples look benign. Applied to Tulu 3, removing the 20% most aligned examples to produce SafeTulu reduced GCG attack success rate by up to 22% across Qwen, Llama, and Olmo, with capability largely preserved. On emergent misalignment, the authors report AUC as high as 0.865 in the gradient space of misaligned models.

Mitigation

The source describes removing the 20% of training examples most gradient-aligned with unsafe responses, producing the SafeTulu dataset, and releases SafeTulu along with a Hugging Face Space that lets researchers customize the removal fraction or filter subsets.

Related items

• InfoChart: Anthropic's cash pile towers over nearly every U.S. companySame vendor · CNBC Technology
• HighGHSA-wf6j-7hvr-gr93: Hydra legacy target blacklist permits unsafe PyYAML loadingSame vendor · Hugging Face Security Advisories
• InfoAnthropic accelerates – for the rest of us, the speed of growth spells precaritySame vendor · The Guardian Technology
• InfoAnthropic's IPO is risky businessSame vendor · CNBC Technology
• LowThe RealReal won’t let AI agents hog designer clothingSame vendor · The Verge (AI)