RiskyPy: Toward Model-Calibrated Prompt Generation for Joint Security and Functionality Evaluation of LLM-Generated Code

Summary

RiskyPy is an automated pipeline that generates security-oriented Python code-generation prompts calibrated to model proficiency, using functionality and vulnerability canaries. It covers 18 CWEs from the 2025 CWE Top 25 and produces 101 prompts. In experiments on Qwen3-4B and Seed-Coder-8B, 57-72% of prompts yielded functional code, while only 5-10% of generations were free of Bandit findings.