Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,003 items
RoCoMark is a code watermarking framework designed to survive LLM-based rewriting that preserves functionality. The authors attribute existing methods' failures to a distribution mismatch between training perturbations and LLM rewriting behavior. Across four benchmarks and attacks from five LLMs, RoCoMark reaches an average detection AUROC of 74.2% and a Rewrite BitACC of 75.3%, with a code pass rate above 99.7%.
Runtime anomaly detectors for cyber-physical systems are either black-box models that miss stealthy manipulations or hand-built physical models that need expert effort per plant. The authors introduce physics-informed agents (PIA), where a large language model agent writes compact physical residuals and refines them against normal operating data. On the BATADAL water distribution benchmark, PIA with Claude Opus 5.5 reaches an F1 of 0.878 against 0.504 for the strongest of six baselines.
SkillCODER is a black-box, content-only method for auditing copyright and attributing buyers of LLM agent skills. It embeds a dormant, Artifact-Gated Capsule into a skill, with buyer-specific fingerprints encoded through Controlled-Vocabulary Error-Correcting Codes. Across three domains, six models and three agent frameworks, it reports 99.2% owner verification and 99.3% buyer Top-1 attribution accuracy.
This research presents a runtime audit for LLM agents that calls tools, aimed at indirect prompt injection in tool outputs. The audit derives an intent graph from the user's objective and tool catalog with one LLM call per task, then checks each tool call against it with a deterministic loop, median 5.7 microseconds per call. Flagged calls are escalated to a reasoning guardian that sees only the objective, plan, call history without tool outputs, and the flagged call, and the authors note end-to-end validation remains open.
ErosionBench is a benchmark that tests whether multi-agent LLM systems drift into unsafe tool actions as benign requests propagate through poisoned shared state, long-context documents and trusted peer messages. It covers 1,728 scenarios across four enterprise domains, plus 198 image-based attacks paired with text equivalents. A proposed Meta-Auditor reaches 98.5% recall at a 2.9% false-positive rate on a held-out split, and its detection is channel-invariant across image and text attacks.
Fix: The source proposes a calibrated Meta-Auditor pairing synchronous state checks with asynchronous trajectory monitoring as a reference defense.
Arxiv (cs.CR + cs.CL + cs.LG)Chatter Attack is a resource consumption attack on large language models that optimizes adversarial prompts to induce short self-doubt target phrases, which sustains long generations without the search-space explosion of earlier methods. Using inference attention loss and global entropy loss, it raises maximum response length up to 31.5x across multiple models and datasets. For black-box settings, the authors add Bayesian Prompt Optimization (BPO), which builds a discrete token kernel to explore the search space.
A layer-wise study tests how style rewriting affects the confidence of zero-shot LLM fake-news detectors. Across LLaMA-3-8B, Qwen2.5-7B and Mistral-7B on three benchmarks, rewriting articles into sensational or emotionally charged styles, without changing facts, makes the models markedly less accurate, worse than random guessing under the strongest attack, while their confidence stays the same. The authors conclude that an LLM's confidence about veracity cannot be trusted as a safety signal when an article's style may have been manipulated.
This research asks whether hidden-state probes trained only on benign tool use still detect incorrect tool actions under adversarial conditions. The authors test three models (Qwen2.5-7B-Instruct, Qwen2.5-14B-Instruct, GLM-4.6V-Flash) on four AgentDojo suites across benign, prompt-injection, system-override and tool-poisoning conditions, comparing the probes with three scalar uncertainty metrics. Under the system-override condition with Qwen2.5-14B-Instruct, all three uncertainty metrics beat the probe on AUROC and average precision in six significant paired tests after Holm correction, so the authors conclude benign-trained probes should not be assumed reliable for this purpose.
This paper builds a curated dataset of MCP vulnerabilities from CVE disclosures, filtering 119 retrieved records through February 2026 down to 24 validated cases. It reports that 87.5% are Critical or High severity and that injection and path traversal account for 66.7% of cases. The authors also release the dataset, reproducibility tooling and a maintenance workflow.
Researchers evaluated four model-internal jailbreak detectors across four LLMs and twelve jailbreak families. Thresholds calibrated on curated benign prompts frequently failed to hold their false-positive rate budgets on real-user traffic, and recalibration exposed family-level coverage gaps, including high-AUROC cases with low true-positive rates. Changing the reference set could substantially shift which attack families were detected.
Fix: The authors recommend evaluating model-internal jailbreak defenses using realized FPR on deployment-like traffic and per-family TPR at explicit low-FPR operating points, with AUROC as a complementary ranking metric rather than the primary security measure.
Arxiv (cs.CR + cs.CL + cs.LG)This systematic review screens 186 candidate sources and retains 98, identifying 284 atomic attacks and 452 atomic safeguards against agentic AI systems that extend LLMs with tools, memory, retrieval, and multi-agent coordination. Tool, MCP, and action-layer attacks make up the largest share of atomic attacks, while the prompt-injection and action-hijacking class has the broadest paper coverage. Only 278 of 660 attack-safeguard relationships are directly assessed, so safeguard coverage remains uneven.
Fix: The paper states that its findings support lifecycle-aware threat modeling, agent evaluation, tool governance, and defense-in-depth security for agentic AI systems.
Arxiv (cs.CR + cs.CL + cs.LG)This research paper evaluates three open-source scanners on 140 agent skills (80 malicious, 60 benign), including Cisco Skill Scanner with and without its LLM analyzer, finding default recall of 0.91 with 33 false positives and LLM-enabled recall of 0.85 with 40 false positives. It then proposes a framework combining static findings, safety-constrained sandbox evidence, and LLM reasoning, which detects 68 of 80 malicious skills with 0.93 precision and reduces false positives to five.
BAPLe++ studies whether backdoors implanted through prompt learning in medical foundation models (Med-FMs) transfer to other Med-FMs, measured with Clean Accuracy Difference (CAD) and Backdoor Accuracy Difference (BAD). The authors also propose DePLe, which uses the infected prompt as a fixed anchor and learns a new prompt on a small clean set to reduce backdoor accuracy. Across four Med-FMs and six chest X-ray and histopathology datasets, backdoors transferred asymmetrically and in a dataset-dependent manner, and DePLe reduced backdoor accuracy more consistently than four training-free input transformations.
Fix: DePLe: learn a new prompt on a small clean set, using the infected prompt as a fixed anchor, so the model keeps its clean task while diverging from the compromised embedding region, without retraining the backbone.
Arxiv (cs.RO + cs.CV security)Researchers propose MGE, a membership guided extraction framework that recovers training data from a pre-trained diffusion model. The method treats extraction as conditional generation, with the condition supplied by a membership inference classifier, and adds adversarial gradient and parallel branch decoding to raise success rate and efficiency. On image generation benchmarks, MGE extracts hundreds of images, which the authors say calls for more attention to privacy-preserving diffusion models.
PrivMark is a framework that suppresses soft-biometric leakage, such as age, gender, and race, from vision-language models by adding a bounded pixel-space perturbation to the image. It combines two steering vector directions in hidden-state space, one for incidental disclosure under open-ended descriptions and one for compliance with direct biometric probes. Across three white-box source models it reaches up to 85.9% biometric suppression while keeping 75% of clean OK-VQA utility, and the perturbations transfer to seven unseen VLMs, including closed-source APIs.
Gem5-Shark is a cycle-level simulation framework, built on Gem5-Aladdin's Smaug, for evaluating security threats to heterogeneous deep learning accelerators before deployment. It integrates eight threat vectors across dataset, compilation, SoC communication and accelerator execution stages, including two new compilation-stage attacks, Operator Attack (OPAtt) and Tiling Attack (TAtt). Across Minerva, LeNet-5, VGG-16 and pretrained LLMs up to 8B parameters, the attacks degrade accuracy by up to 97.7%, raise latency by up to 84× and implant backdoors with as few as 20 poisoned fine-tuning samples.
Fix: Two defenses are proposed: a compiler-stage tiling range check and a host-side finish-flag watchdog, each removing over 99% of the injected overhead on CNNs and LLMs.
OpenAlex (peer-reviewed AI security)This paper evaluates how two gradient-based untargeted evasion attacks, FGSM and PGD, affect the stability of DALEX, SHAP and Integrated Gradients explanations in intrusion and malware detection. Explanation drift is reported separately for inputs whose predicted label changes and those where it is preserved. Experiments on CICMalMem22 and CICIDS17 show measurable drift when predictions are preserved, with mean Top-5 overlap between 0.586 and 0.892, and the authors conclude that XAI is an additional attack surface.
This study evaluates indirect prompt injection against locally deployed RAG assistants, testing four attack families across five defense configurations (C1–C5) and four local LLMs on a synthetic corpus of 40 documents and 32 queries. In the Llama 3.1 8B experiment, conditional Attack Success Rate fell from 70.4% without defenses (C1) to 22.2% with deterministic post-generation filtering (C5). Context contamination remained the main residual weakness, and the authors note the filter and evaluator share some detection indicators, so post-filter ASR reflects unsafe content left in responses.
Fix: Deterministic post-generation filtering (C5), which the authors say supports independent security enforcement, reduced conditional ASR from 70.4% to 22.2% in the Llama 3.1 8B experiment and gave post-filter ASRs of 7.4–22.2% across the four models, with Answer Utility ranging from 21.9% to 62.5%.
OpenAlex (peer-reviewed AI security)GuardrailAgent is a multi-agent defense framework against jailbreak attacks on LLMs. It autonomously builds a continuously updated knowledge base of attack characteristics that supports input- and output-side detection, and a feedback module folds misclassified cases back in. Tested on over ten representative jailbreaks, it reached a 93.1% input-side detection rate, 93.9% overall detection, and a 95.8% F1 score while adapting to unseen attacks.
MemoryDojo is a benchmark for tracing how poisoned information persists through LLM agent memory and affects later agent behavior. It contains 155 multi-session long-horizon trajectories across four domains and supports five memory systems. The authors find all tested memory systems are susceptible to poisoning, with substantial variation in downstream effects, and propose a memory-aware poison optimization process.