Beyond AUROC: Deployment-Aware Evaluation of Model-Internal Jailbreak Detectors

Summary

Researchers evaluated four model-internal jailbreak detectors across four LLMs and twelve jailbreak families. Thresholds calibrated on curated benign prompts frequently failed to hold their false-positive rate budgets on real-user traffic, and recalibration exposed family-level coverage gaps, including high-AUROC cases with low true-positive rates. Changing the reference set could substantially shift which attack families were detected.

Mitigation

The authors recommend evaluating model-internal jailbreak defenses using realized FPR on deployment-like traffic and per-family TPR at explicit low-FPR operating points, with AUROC as a complementary ranking metric rather than the primary security measure.

Topics

• Prompt injection and jailbreaks

Related items

• InfoSuppression Is Not Erasure: Mechanistic Analysis and Reactivation Boundaries of Subspace Projection for LoRA Backdoor DefenseSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoAnyBreak: Towards Jailbreaking Any T2I Model for Any ConceptSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoHarness Matters: Measuring and Dissecting Security Resilience in Web AgentsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoA Layer-wise Study of How Style Rewriting Breaks the Calibration of Zero-Shot LLM Fake-News DetectorsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoGem5-Shark: Smaug-based Heterogeneous Deep Learning Accelerator Security and Robustness Evaluation and Characterization FrameworkSimilar attack · OpenAlex (peer-reviewed AI security)