When Models Know but Can't Linearly Say: Relational Encoding Limits Linear Vulnerability Classification in Code LLMs

Summary

This study tests eight code LLMs, from 6.7B to 15B parameters, on C programs drawn from the DELTASECOMMITS, SVEN and PRECISE-BUGS datasets. The models encode which code patterns a snippet instantiates with high fidelity, but vulnerability is more accessible as a paired relational signal than as a separable categorical label. Standard classifiers on isolated samples therefore reach limited accuracy, because vulnerable and secure samples overlap in representation space, even though the models reliably rank which of two snippets is riskier.