Beyond Binary Labels: Diagnostic Evaluation of LLM Safety with Prompt-Level Rubrics

Summary

DiagSafe is a diagnostic framework for evaluating LLM safety that replaces binary safe or unsafe labels with prompt-level rubrics covering safety and usefulness. Tested on a bilingual English-Chinese benchmark built on an 84-label safety taxonomy, it reached 95.34% agreement with human annotations, versus 69%-87% for baselines, and found that leading LLMs differ markedly in unsafe rates and are particularly susceptible in Chinese and multi-turn contexts.

Related items

• Medium“Next Exploit Opportunities”: How a Threat Actor Used AI to Help Plan and Execute Multiple Types of Cybercrime ActivitiesSimilar attack · Palo Alto Unit 42
• MediumAI Guardrails Aren’t Stopping Criminal HackersSimilar attack · Wired (Security)
• InfoBuilding Trust into Enterprise AI Agents: Cisco AI Defense and Workday Agent PassportSimilar attack · Cisco AI Blog
• InfoCombining Long-Term Memory and LLMs Comes at a High Cost: From Prompt Injection to Persistent CompromiseSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoPolarity Blindness in LLM Safety: A Systematic Study of Negation, Antonym Substitution, and Adversarial FramingSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)