Separating signal from noise in coding evaluations

OpenAI discovered that SWE-Bench Pro, a widely-used benchmark for measuring AI coding abilities, has significant quality problems that make it unreliable for evaluating model capabilities. Approximately 30% of the tasks in the benchmark are broken due to issues like overly strict tests, unclear instructions, insufficient test coverage, or misleading prompts, meaning the benchmark no longer accurately measures whether AI models can actually write software.