Identically Trained Language Models Break on Different Jailbreaks

Summary

This study tests whether models that share pretraining data and safety post-training break on the same jailbreak prompts. Across 25 data recipes, 150 one-billion-parameter models were compared, and per-prompt non-refusal rates correlated at 0.984 between post-training seeds of one pretraining run, falling to 0.811 across pretraining seeds and 0.779 across data recipes. The best attack template on one checkpoint stayed best after a new pretraining seed in only 27% of model pairs, versus 67% after a new post-training seed. Each pretraining run was also identifiable from its non-refusal pattern across 3,211 prompts.

Topics

• Prompt injection and jailbreaks

Related items

• InfoI-HITL: Inalienable Human-in-the-LoopSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoSteering for Safety, Hijacking for Profit: Trojan Threats in the LLM Activation Steering Supply ChainSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoThe Target Agent Is Its Own Best Critic but a Poor Proposer: Self-Improved Red TeamingSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoSentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety DeploymentsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• MediumSecuring our frontier model training environmentSimilar attack · OpenAI Blog