InfoResearchPreprintLLM-specific
On the Reliability of CyberGym Evaluation: Task Material Consistency and Reference-Target Identity
- Published
- Record updated
Summary
This research audits the reliability of CyberGym, a benchmark that scores vulnerability reproduction by checking whether a proof-of-concept input crashes a vulnerable program version but not its fixed version. Auditing all 1,507 tasks, the authors flag 338 (22.4%) as materially inconsistent, mostly through coverage gaps and descriptions and patches pointing to a different defect. Of 1,286 differential passes from one Level-1 GLM-5.3 run, 89 were judged non-target and 10 had insufficient evidence, lowering the pass rate from 85.3% to 78.8% when a target hit is required.