ABench: Adversarily Benchmarking Large Language Models across Taxonomized Values

Summary

The authors introduce ABench, a handcrafted benchmark for testing LLM value alignment in Chinese, built on a hierarchical taxonomy of 158 value subcategories and 2,895 open-ended questions. The questions use role-playing and logical traps to test whether models keep their stances under adversarial induction, and the authors add a weighted automated evaluation framework. Across 16 mainstream LLMs, most models scored below 60 under these attacks, indicating significant vulnerabilities.

Related items

• InfoBuilding Trust into Enterprise AI Agents: Cisco AI Defense and Workday Agent PassportSimilar attack · Cisco AI Blog
• InfoPolarity Blindness in LLM Safety: A Systematic Study of Negation, Antonym Substitution, and Adversarial FramingSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoOrthogonal Boundary Unlearning for Dual-Use Concepts in Large Language ModelsSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoGlobal Security Blind Spots: Translation as a Safety Bypass in Low-Resource LanguagesSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoSafety Mid-Training: Internalizing LLM Safety as a Foundational CapabilitySimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)