{"data":{"id":"2a0f22f9-3e04-49b2-8eb1-8ec8f7a6b0a1","title":"What Standard Evaluations Hide About Detectors of Hijacked LLM Agents: Rare Attacks, Unseen Tools and Benchmark Transfer","summary":"This research audits eleven evaluations of detectors that flag hijacked LLM agents from their tool-call traces and finds attack rates of 16% to 86% with no variation tested. On AgentDrift and AgentDojo, a detector with AUROC 0.879 reaches precision 0.076 at a 2% attack rate, and detector rankings reverse as tool coverage falls below about 50%. Detectors trained on AgentDrift rank AgentDojo traces at chance, while the AlignmentCheck monitor reaches AUROC 0.930 but would still mostly raise false alerts at that attack rate.","solution":"The authors turn their findings into a checklist for evaluating trace detectors, but the source text does not state any specific fix or workaround. Dropout on tool inputs is reported to largely remove the ranking reversal, which is a finding rather than a recommendation.","labels":["security","research"],"sourceUrl":"https://arxiv.org/abs/2610.41803v1","publishedAt":"2026-10-30T14:30:47.000Z","cveId":null,"cweIds":null,"cvssScore":null,"cvssSeverity":null,"severity":"info","attackType":["other"],"issueType":"research","affectedPackages":null,"affectedPackageNames":null,"affectedPackageRefs":null,"affectedVendors":[],"affectedVendorsRaw":["AgentDojo","AgentDrift","AlignmentCheck"],"classifierModel":"claude-haiku-5-5","classifierPromptVersion":"v4","summaryPromptVersion":"v2","headline":null,"headlinePromptVersion":null,"cvssVector":null,"attackVector":null,"attackComplexity":null,"privilegesRequired":null,"userInteraction":null,"exploitMaturity":null,"epssScore":null,"epssCheckedAt":null,"kevDateAdded":null,"advisoryAliases":null,"affectedPackagesSource":null,"affectedPackagesCheckedAt":null,"patchAvailable":null,"disclosureDate":"2026-10-30T14:30:47.000Z","fixReleasedAt":null,"daysToFix":null,"fixCheckedAt":null,"capecIds":null,"crossRefCount":0,"attackSophistication":"moderate","impactType":["integrity","safety"],"aiComponentTargeted":"agent","llmSpecific":true,"classifierConfidence":0.95,"researchCategory":"preprint","atlasIds":null}}