Separating signal from noise in coding evaluations

Summary

OpenAI audited SWE-Bench Pro, the successor to SWE-bench Verified, using an automated datapoint analysis pipeline plus multi-pass agent review and human annotation by five software engineers. The audit flagged 200 (27.4%) tasks automatically and 249 (34.1%) through human annotation, with issues in four categories: overly strict tests, underspecified prompts, low-coverage tests, and misleading prompts. OpenAI estimates that about 30% of SWE-Bench Pro tasks are broken and advises model developers to examine results carefully.

Related items

• LowOpenAI’s AI misalignment problem is growingSame vendor · CSO Online
• InfoOpenAI Employee Arrested for Allegedly Planting Hidden Recorder in Executive Conference RoomSame vendor · The Hacker News
• LowFBI arrests Saudi OpenAI employee who allegedly planted recording device in conference roomSame vendor · CNBC Technology
• InfoAn OpenAI Employee Allegedly Used a Chatbot to Help Plant a Recording Device at the OfficeSame vendor · Wired (Security)
• InfoEmployee Arrested After Listening Device Found in OpenAI Conference Room, FBI and the Company SaySame vendor · SecurityWeek