{"data":{"id":"323c9f0b-b9e8-43a3-b325-ec23075012e4","title":"The Missing Compliance Class: LLM Graders That Collapse Jailbreak Success, and a Free Check","summary":"This research shows that dense LLM graders used to score jailbreak attacks can collapse compliant responses into the evasion class, so the reported attack success rate depends heavily on which grader is used. On two corpora of 900 responses and eight graders, the compliance rate differs by a factor of 22 to 25, while the non-refusal rate differs by 18 to 23 percentage points. The authors propose reporting the compliance rate with its per-class label counts and the named grader, alongside the non-refusal rate.","solution":"The authors propose a reporting minimum that keeps the compliance rate and adds its per-class label counts and the named grader. Beside it goes the non-refusal rate, whose disagreement across graders is additive.","labels":["research","safety"],"sourceUrl":"https://arxiv.org/abs/2610.24872v1","publishedAt":"2026-10-17T21:20:23.000Z","cveId":null,"cweIds":null,"cvssScore":null,"cvssSeverity":null,"severity":"info","attackType":["jailbreak"],"issueType":"research","affectedPackages":null,"affectedPackageNames":null,"affectedPackageRefs":null,"affectedVendors":[],"affectedVendorsRaw":[],"classifierModel":"claude-haiku-5-5","classifierPromptVersion":"v4","summaryPromptVersion":"v2","headline":null,"headlinePromptVersion":null,"cvssVector":null,"attackVector":null,"attackComplexity":null,"privilegesRequired":null,"userInteraction":null,"exploitMaturity":null,"epssScore":null,"epssCheckedAt":null,"kevDateAdded":null,"advisoryAliases":null,"affectedPackagesSource":null,"affectedPackagesCheckedAt":null,"patchAvailable":null,"disclosureDate":"2026-10-17T21:20:23.000Z","capecIds":null,"crossRefCount":0,"attackSophistication":"moderate","impactType":["safety"],"aiComponentTargeted":"model","llmSpecific":true,"classifierConfidence":0.9,"researchCategory":"preprint","atlasIds":["AML.T0054"]}}