Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysis

Summary

This measurement study evaluates code generated by five widely used LLM families across Python, Java, C++, and C, using 200 manually curated tasks. Generated code was assessed for compilation success, semantic correctness with 4,000 per-program unit test files, and quality and security with SonarQube and CodeQL. Python and Java showed higher compilation and correctness rates and fewer security findings than C and C++, which showed more memory safety issues, hard-coded secrets, and cryptographic misuses.