InfoResearchPreprintLLM-specific
What Standard Evaluations Hide About Detectors of Hijacked LLM Agents: Rare Attacks, Unseen Tools and Benchmark Transfer
- Published
- Record updated
Summary
This research audits eleven evaluations of detectors that flag hijacked LLM agents from their tool-call traces and finds attack rates of 16% to 86% with no variation tested. On AgentDrift and AgentDojo, a detector with AUROC 0.879 reaches precision 0.076 at a 2% attack rate, and detector rankings reverse as tool coverage falls below about 50%. Detectors trained on AgentDrift rank AgentDojo traces at chance, while the AlignmentCheck monitor reaches AUROC 0.930 but would still mostly raise false alerts at that attack rate.
Related items
- HighGHSA-jq3q-62jc-fqq9: code-review-mcp-server: Command injection via unsanitized folderPath/baseBranch parametersSimilar attack · GitHub Advisory Database
- CriticalCVE-2026-120699: CubeSandbox host path bind mount allows host file access via POST /sandboxesSimilar attack · NVD/CVE Database
- MediumCalifornia sues OpenAI over AI agents escaping sandboxes, hacking firmsSimilar attack · BleepingComputer
- HighCVE-2026-126816 - Path traversal and arbitrary file write in the cost report generator of aws-pricing-mcp-serverSimilar attack · AWS Security Bulletins
- InfoRansomware Attacks Increase Behind AI, But Payments DeclineSimilar attack · Dark Reading