Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,003 items
This research paper shows that finite safety assessments cannot rule out an adversary that behaves normally until certification and then activates. The authors prove that an adversary matching the full observation law until irrevocable certification can gain expected post-certification events of v E[(n-S_0)_+] over n trials. In simulated RoCo collaboration, switching produced commanded-separation violations in 34 of 36 post-certification episodes, and preregistered AgentDojo trials estimated 1.98 additional security violations per four-task continuation (95% interval [1.67, 2.29]).
Fix: With permanent revocation, a mixture likelihood-ratio monitor bounds cumulative excess above a prespecified reference rate at square-root order, up to logarithmic factors. Against a controller observing the monitor's rule and statistic, live mixture monitoring reduced violations from 12.00 to 3.81 per 24 assigned task opportunities, and revocation also reduced tasks executed.
Arxiv (cs.CR + cs.CL + cs.LG)Researchers propose SCRouter, an LLM router that picks both the model and the output-token budget for each query, weighing utility against security because the two objectives favor different choices. They release SCRouterBench, with 38,869 utility and security queries across nine security dimensions, each answered under all 144 model-budget combinations. SCRouter beats baselines by 13-29 points in-domain and 19-23 points out-of-domain.
SA-CUA is a dual-LLM framework for Computer Use Agents, which act visually on screens and lack a boundary between trusted and untrusted content. It derives conservative deterministic rules from the accessibility tree to unmask trusted regions from the privileged LLM's screenshot, and uses an endorsement channel to verify untrusted values before they reach the trusted domain. The authors report it keeps much higher utility than CaMeL-CUA while holding the attack success rate at zero.
Kairos is a reference monitor placed at the isolation boundary of an unmodified, opaque LLM agent, mediating the file and network operations that its tool calls, shell commands and generated scripts produce. A declarative trust policy decides most effects deterministically and defers the rest, which are held while the agent continues speculatively until a bounded decision oracle, a language model or a human, releases or refuses them. On AgentDojo across three agent models, it reduces attack success from 47.0% to 2.7% while benign utility stays at the undefended agent's level.
FedGraph is a graph-based aggregation framework for federated LLM fine-tuning that defends against backdoor attacks. It models clients as nodes in a dynamically constructed graph, using the structural consistency of malicious updates to separate them from benign ones and suppress them during aggregation. Experiments report attack success rates below 10%, 95.5% average detection accuracy, and a 2.33% false positive rate.
Fix: FedGraph, a graph-based aggregation framework that suppresses malicious client contributions during aggregation.
Arxiv (cs.CR + cs.CL + cs.LG)Linear probes on hidden states are used to screen agent tool outputs for indirect prompt injection, but they can learn conversation position and text length instead of the attack itself, reaching 0.91-0.98 AUROC without the model. The authors add matched harmless negatives to the training data, cutting false alarms on injection-free conversations from 41-80% to 4-27% across six open models at the default threshold. At a threshold set on normal traffic, the probe still catches a median 95% of attacks that fooled the agent, versus 91% without the new data.
Fix: Add matched negatives to the probe's training data: the same conversation without the injection, and harmless text formatted like an injection. The same data also helps a text-only filter.
Arxiv (cs.CR + cs.CL + cs.LG)Researchers present a targeted displacement attack that changes the account identifier embedded in multi-bit LLM watermarks without the secret key. The attacker needs only a few accounts that generate output and model logits, then rewrites the text while keeping its meaning so the decoder recovers the attacker's chosen message. Empirically, the attack succeeds in almost all cases across five watermarking methods and four LLMs, given enough accounts, with success tracking the paper's theoretical bounds.
Jev-Guard is a unified model that performs both safety classification and policy routing for agent harnesses, selecting an applicable audit policy from a policy tree before classifying the action. On 7,033 evaluation cases it reports 93.72% safety accuracy, a mean latency of 58ms, and 93.10% hierarchical routing accuracy, with the full pipeline averaging 240ms. Integrated into the Hermes harness framework, its permission gate cut mean review time per task from 4.80s for an SFT-LLM judge to 1.33s.
Researchers introduce Ciphered Skill Injection (CSI), which hides malicious instructions in encoded form inside third-party skill files so that LLM agents decode and follow them. Across eight models and 139 injection-task pairs, the Caesar and layered variants reach attack success rates of up to 47.5% and 40.3% in the primary setting with shortened skills. The tested skill scanners rarely flag the injected skills.
The survey from ACM Computing Surveys examines backdoor vulnerabilities and fairness challenges in generative models. The source text provided is limited to bibliographic details (Volume 58, Issue 16, pages 1-56, December 2026), so no methods or findings are described.
CompanionSafe is a framework for evaluating and mitigating safety risks in AI companions, which provide ongoing emotional support through sustained relational interaction. Its evaluation component, CompanionEval, builds 1,300 relationally grounded multi-turn settings across four turn tiers from diverse personas, matched companion characters, and dialogue scenarios. Its mitigation component, CompanionGuard, turns benchmark-exposed failures into turn-localized supervision for companion-specific risk detection. The authors report that even strong LLMs and existing guard models remain vulnerable to companion-specific relational harms, and that safety risks increase as interactions grow longer.
Researchers propose local contract-conditioned rewriting as an enforcement layer for LLM agents that draft and send emails on a user's behalf. The method builds an information-release contract for each outgoing draft, rewrites it with a local Gemma rewriter, and invokes a final PrivacyChecker selectively. Under normal prompts, leakage falls from 11.41% to 4.03% while UsableRate rises from 85.2% to 92.6%, and under strategic attack prompts checker-only cost amplification drops from 1.32x to 1.02x.
ProtoArena is a black-box benchmark of 80 cross-component protocol vulnerability-exploitation environments spanning 11 domains and 7 attack categories, primarily informed by published protocol-attack mechanisms. Five LLMs were evaluated, and Claude Opus 4.8 reached 48.8% success under L1 guidance and 12.5% under effect-directed black-box L3. Results show agents can achieve verifiable exploitation effects in some environments, but black-box success remains low and varies across domains.
This paper presents the first controlled adversarial evaluation of TypeSafe AI's Jev, a safety judge that returns probabilities for developer-defined questions rather than text or labels. Across four tasks and fifty attack variants from thirteen families, Jev was the most robust system on input moderation, bypassed by 9% of blocks under a benign-sounding question versus 56% for GPT-6 Astra. On phishing detection, forgeries bypassed 19-29% of flagged emails, and Jev detected only 62% of phishing emails without attack, against 77-81% for the GLM judges.
Researchers tested whether steering LLM activations along semantic directions (formality, sentiment, complexity, and a random control) could make a robust text watermark, comparing it against the standard logit-based KGW watermark. At matched fluency (within 1.15 times the original perplexity), KGW reached an AUC of 0.81 against a sentence-level paraphraser while the best steering direction reached 0.65. A random direction was as detectable as a meaningful one, so the signal was not semantic.
SkillScreen is a compact decision model that scores whether an agent skill is malicious directly from its SKILL.md file in a single forward pass. On MaliciousSkillBench it reaches 99% recall on unseen attack families while flagging 20-25% of benign skills, fewer than the word-level classifier and prompted LLM judges tested. It also generalizes better than a fully fine-tuned encoder to unseen sources, though unfamiliar benign sources sharply raise false positives.
The paper introduces SADDER, an extension of the Safety-Aware Denoiser (SAD) for text diffusion models. SADDER uses an embedding-space safety signal to gate which token positions SAD's update rule modifies, and derives new position-wise target distributions so the modified denoiser stays exact. Tested on Masked Diffusion Language Models and Uniform-State Diffusion Models in unsafe continuation and jailbreak settings, it reportedly balances safety, generation quality and inference cost better than existing methods.
Researchers propose FedChainDP, a federated learning framework for ECG classification across hospitals that adds update-level Gaussian noise, encrypts updates with AES-256-GCM, and logs them on a blockchain with Merkle-tree integrity protection. On MIT-BIH data split across 5 hospitals, the model reached 98.91% ± 0.14% accuracy, but the authors state the evaluation used a beat-level random split with duplication-based balancing, and no formal (ε, δ)-differential privacy guarantee is claimed.
BrowseSafe++ extends the BrowseSafe benchmark from 14,719 pages to 38,675 paired attacked and clean pages, most injections observed in the wild on production websites. The authors train BrowseSafe-Guard, two 0.8B-parameter detectors for text and screenshots, which achieve higher F1 scores than a frontier-model-based detector while keeping false-positive rates and inference cost low. BrowseSafe-Adapt revises injections from browser-agent feedback, raising attack success on tasks from a few percent when replayed directly to up to 95.41%.
VeriRouter is a verifiable framework for LLM routers, which select models and rewrite requests and can be compromised. It combines ZK proofs with a provider-issued serving certificate, under an endorsed public context binding a private semantic index and rewrite banks, to check integrity on both the control and data planes while keeping routing assets hidden from clients. On real-world workloads, model-selection and rewrite proofs average 21.92 s and 4.24 s, and verification adds 4.08 s per turn or 0.11 s with session-pinned selection.