InfoResearchPreprintLLM-specific
VeriHarm-Bench: Measuring Harmful Success in LLM Coding Agents Verifiably
- Published
- Record updated
Summary
VeriHarm-Bench measures harmful success, where a coding agent completes a benign task but also causes unintended harm such as deleting unrelated files, weakening access controls, or corrupting data. The authors introduce VeriHarm, an automatic framework that minimally perturbs coding tasks and synthesizes executable harm tests alongside the original task-correctness tests. Frontier agents, including GPT-5.6-Sol and Claude-Opus-5, averaged 95% task accuracy on the curated benchmark but only 37% safe success rate.
Topics
Related items
- InfoChatGPT will now make you animated GIFsSame vendor · The Verge (AI)
- MediumResearchers Uncover Prompt Adware Hidden in OpenAI Codex Jailbreak ToolkitSame vendor · The Hacker News
- InfoForrester gave Endor Labs the highest score possible in seven criteria, including innovation, AI application flaws, AI SCA, and AI coding toolchain. | Endor LabsSame vendor · Endor Labs Blog
- InfoUber only fulfilled 4 rides from AI agents last quarterSame vendor · The Verge (AI)
- InfoSam Altman addresses RAM comments: People should ‘put as much as they want in their computers’Same vendor · The Verge (AI)