Analysis of Local Open-Weight LLM Agents on Vulnerability-Reproduction Tasks

Summary

This study evaluates twelve configurations of local open-weight language models on ten CyberGym vulnerability-reproduction tasks, three runs each under one harness. Every configuration, down to the 4B model, solves most of the six easier tasks, while the remaining four separate the configurations and the 27B model is required for the harder ones. Quantization left the solved set unchanged in every matched pair but changed speed, and the mixture-of-experts model reached the same scores in less wall-clock time.