Skip to content
AI Sec Watch
InfoResearchPreprintLLM-specific

Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers

Source
Arxiv (cs.CR + cs.CL + cs.LG)(opens in a new tab)
Published
Record updated
View JSON

Summary

This paper presents the first controlled adversarial evaluation of TypeSafe AI's Jev, a safety judge that returns probabilities for developer-defined questions rather than text or labels. Across four tasks and fifty attack variants from thirteen families, Jev was the most robust system on input moderation, bypassed by 9% of blocks under a benign-sounding question versus 56% for GPT-6 Astra. On phishing detection, forgeries bypassed 19-29% of flagged emails, and Jev detected only 62% of phishing emails without attack, against 77-81% for the GLM judges.

Related items