Systematic Evaluation of Adversarial Jailbreaking Attacks and Defenses Across Seven Large Language Models

Summary

This empirical study tests five adversarial jailbreaking attack families against seven LLMs, including Llama-3-8B, Mistral-7B, Qwen2.5-7B and Gemma2-9B, and four defenses. Obfuscation attacks reach a mean attack success rate of 54.3% and bypass every evaluated model with nonzero success. Model size does not predict robustness (Spearman r=-0.57, p=0.18), and self-reminder prompting cuts ASR from 44% to 10% on Mistral-7B and from 48% to 5% on Llama-3.1-8B with no reported false positives.

Mitigation

Self-reminder prompting is the most consistent defense in the evaluation, reducing ASR from 44% to 10% on Mistral-7B and from 48% to 5% on Llama-3.1-8B, with zero false positives and no additional inference cost.

Topics

• Prompt injection and jailbreaks

Related items

• InfoAmazon AWS posts 39% revenue growth as cloud rivals also accelerate Same vendor · CNBC Technology
• InfoYou can generate and edit Google Docs files directly in Gemini chats nowSame vendor · The Verge (AI)
• InfoClaude Model Builds Exploits Across Major Browsers & OSesSame vendor · Dark Reading
• InfoEndor Labs + Google Antigravity: Secure code by default in agentic workflows | Endor LabsSame vendor · Endor Labs Blog
• InfoGoogle Touts Powerful Cybersecurity Capabilities in Gemini 4 Carbon ModelSame vendor · SecurityWeek