How Much Does the Harness Matter, and for Whom? A Multi-Model Study of Context in LLM-Driven Vulnerability Discovery

Summary

Researchers tested 24 model-runner configurations of an open-source agent harness on a production codebase with non-public vulnerabilities, across 265 confirmed security root causes. The largest two-run configuration covered 60.4% of findings, and 14.3% were found by exactly one model. In a four-model experiment, an engineered prompt with CodeQL-derived call-graph context raised criticality-weighted yield by 20-86% for smaller and mid-tier models, versus 2% for the frontier model.