InfoResearchPreprintLLM-specific
PrivacyPR: A Dataset of Real-World Privacy Fixes for Benchmarking Privacy-Aware Code Review
- Published
- Record updated
Summary
PrivacyPR is a dataset of 4,131 merged GitHub pull requests from 1,469 repositories, created between June 2018 and January 2024, that links real privacy defects to their accepted fixes and review context. In its evaluation harness, two coding agents detected only 42-51% of reintroduced defects under a generic review prompt, rising to 88-91% when explicitly asked to check privacy, while flagging 11-15% of non-fix PRs.