Narratives Mislead: An Empirical Study of How LLMs Misjudge CVSS Severity in Bug Bounty Reports

Summary

This study tests whether LLMs misjudge CVSS severity from narrative bug bounty reports, using 240 Huntr reports with vendor-validated CVSS vectors and eight models under four prompting strategies and fine-tuning. The best prompting configuration reaches only 49.58% level accuracy, and fine-tuning a 7B model raises this to at most 56.25%. Errors are directional: models overestimate far more often than they underestimate, and they almost never recognize low-severity vulnerabilities.