Did the LLM Answer the Question? A Semantic Reconstruction Game for Jailbreak Evaluation

Summary

This research examines how jailbreak evaluation depends on the judge LLM, showing that swapping only the judge can change reported attack success rate by up to 82 percentage points. The authors introduce the semantic reconstruction game, a training-free evaluator that reconstructs the request a response answers and compares it against decoy requests, with abstention when neither matches. Across three off-the-shelf LLM judges it achieves the lowest mean error against human labels among the methods evaluated and reduces variation in reported ASR, prioritizing avoidance of false positives.

Topics

• Prompt injection and jailbreaks

Related items

• InfoSafeDriveVLA: Navigation-Conditioned World Model Dreaming for Conflict-Aware End-to-End Autonomous DrivingSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoDiagnose Before You Refine: Multi-Dimensional Judge Feedback for Query-Efficient Jailbreak AttacksSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoMoral Combat: Exploiting Moral Foundations Theory to Jailbreak LLMsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoDeveloping Gemini 4 Argon responsibly and safelySimilar attack · DeepMind Safety Research
• InfoRethinking Quality in Large Language Model Routing: Security-Conditioned Model-and-Budget SelectionSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)