Beyond Attack Success Rate: Cost-Aware Metric for Evaluating Iterative LLM Jailbreaks

Summary

This paper argues that attack success rate (ASR) for jailbreaks ignores repeated calls to the victim model, which give the deployed system more chances to detect and intervene. It introduces ASR_STEP, which penalizes additional victim-model calls while keeping binary success judgments. On JailBreakBench across three model families, GCG's scores drop by 7.0 to 8.7 percentage points after discounting, PAP's stay unchanged, and PAIR's drop by 3.4 to 12 percentage points.

Topics

• Prompt injection and jailbreaks

Related items

• InfoSafety Mid-Training: Internalizing LLM Safety as a Foundational CapabilitySimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoConcerns in Benchmarking Jailbreaks: An Empirical Study of Attack Success RatesSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoBudgeted Attack Scheduling for LLM Red-Teaming: The Roles of Static Priors, Semantic Guidance, and Online FeedbackSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoPeer Pressure: Execution-Verified Co-Evolution of Attacker and Defender Populations for Code SecuritySimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoThe Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free CheckSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)