Trusting the Tool: Asymmetric Deference of LLM Vulnerability Auditors to Static-Analysis Evidence

Summary

This research measures how LLM vulnerability auditors respond to static-analysis notes, using 916 PrimeVul function pairs, eleven model configurations and 16,572 audits. Fabricated assurances that a missing check exists lowered detection of real vulnerabilities by up to 19 points relative to true notes, while fabricated alarms rarely reduced accuracy. Verifying each claim in a separate call and showing the refuted claims raised accuracy under false assurances by 12 points in a pre-specified test (p=0.002), though the gain for a second model was smaller and not significant.

Mitigation

Pipelines should verify a tool's guarantees before the model relies on them, and studies should measure deference in a direction-aware way. Verifying each claim in a separate call and showing the refuted claims improved accuracy under false assurances in the pre-specified test.

Related items

• MediumOpenAI, Anthropic disclosures spotlight growing AI containment challengeSimilar attack · CSO Online
• MediumBias in Qwen3-Coder ModelSimilar attack · Schneier on Security
• MediumCVE-2026-116104: Sony Neural Network Libraries code injection in NNP ConverterSimilar attack · NVD/CVE Database
• MediumCVE-2026-116103: Sony Neural Network Libraries code injection in encode param command handlerSimilar attack · NVD/CVE Database
• MediumPHP2Shell Exploitation Hits Government Entity Amid Global Scanning and Signs of AI-Assisted HackingSimilar attack · The Hacker News