Concerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success Rates

Summary

This paper examines how Attack Success Rate (ASR) for eight jailbreak attacks against three open-weight language models depends on the evaluation procedure. Low-resource language attacks score a mean ASR of 96.0% under keyword matching but 2.7% under a Qwen3 evaluator following StrongREJECT scoring instructions. Repeated sampling at Ollama's default temperature of 0.8 yields 100% ASR under keyword matching versus 50% under the Qwen3 evaluator, suggesting residual unsafe behavior can persist despite an initial refusal.

Mitigation

The paper recommends explicit success definitions, matched response budgets, validated evaluators, and reporting cumulative failure under repeated interaction rather than relying on single-generation tests.

Topics

• Prompt injection and jailbreaks

Related items

• InfoSafety Mid-Training: Internalizing LLM Safety as a Foundational CapabilitySimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoBeyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM JailbreaksSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoBudgeted Attack Scheduling for LLM Red-Teaming: The Roles of Static Priors, Semantic Guidance, and Online FeedbackSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoPeer Pressure: Execution-Verified Co-Evolution of Attacker and Defender Populations for Code SecuritySimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoThe Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free CheckSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)