Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,003 items
This research examines zero-shot adversarial robustness in vision-language models (VLMs), which remain vulnerable to adversarial attacks. The authors show that two existing forms of pre-trained model guidance, matching visual features and matching prediction distributions, each have limitations that can degrade zero-shot performance. They propose Textual Response-Aware Guidance (TRAG), which constrains visual representation drift using the pre-trained text encoder's responses to local input variations, and report state-of-the-art clean and robust performance across architectures and tasks.
Researchers propose CLIC, a closed-loop intent-consistency layer for LLM-driven control in Open RAN that jointly mitigates runtime prompt injection and residual virtual prompt injection (VPI) attacks. CLIC uses a formal consistency score from a small quantized language model, deployed as a Non-RT rApp for model-integrity certification and a Near-RT xApp for runtime enforcement. On the OpenAI Cellular (OAIC) platform, it reduced attack success rate from 0.92 to 0.29 and residual VPI attacks from 0.37 to 0.04, at about 27 ms of added latency.
Fix: CLIC, a closed-loop intent-consistency layer: a Non-RT rApp for model-integrity certification and a Near-RT xApp for runtime enforcement, using a formal consistency score from a small quantized language model.
Arxiv (cs.CR + cs.CL + cs.LG)This paper examines defenses against knowledge poisoning in Retrieval-Augmented Generation (RAG), which are usually scored by attack success rate (ASR). The authors introduce Aligned RAG Evaluation, a matched-query protocol that compares clean and poisoned responses with and without defense, using matched conditions analysis, LLM-judge endorsement, and retrieval dependence controls. Experiments on HotpotQA and Natural Questions show that existing defenses can introduce unexpected target-positive answers even while preventing some attacks, and that evaluation criteria can change defense rankings.
RiskyPy is an automated pipeline that generates security-oriented Python code-generation prompts calibrated to model proficiency, using functionality and vulnerability canaries. It covers 18 CWEs from the 2025 CWE Top 25 and produces 101 prompts. In experiments on Qwen3-4B and Seed-Coder-8B, 57-72% of prompts yielded functional code, while only 5-10% of generations were free of Bandit findings.
LIDARSHIELD is a Dockerized framework for assessing the adversarial robustness of LiDAR-based 3D perception models. It exposes any model through three HTTP endpoints (/info, /forward, /gradient), so connecting a new model or dataset needs only a single loader function or adapter. It supports 3D semantic segmentation and 3D object detection, integrating attacks such as PGD, Square Attack and LiDAttack.
This study examines continual learning for LLMs that select training inputs by surprise, a rule that absorbs planted falsehoods about facts the model already knows. Tested on LoRA-adapted Qwen2.5-1.5B, Qwen2.5-3B and Llama-3.2-3B students learning from streams of true and false claims, the authors find that a verifier judge removes lies only as far as it is accurate, and that the model's own prior flags contradictions with a 43% saving in judge calls. Routing contradicted claims to trusted evidence gives the lowest poison rate, but planted matching sources for 30% of lies make it worse than the judge alone.
The paper studies adversarial perturbations to the internal visual representations of world action models (WAMs), which couple visual or world representations with action generation. It finds that attack effectiveness varies sharply across visual K/V routes and that larger representation displacement does not reliably imply stronger control failure, attributing this to the action-conditioned readout. The authors introduce Tail-AV, a short-probe signal for identifying vulnerable routes, and PASA-KV, which selects action-facing K/V targets before training a fresh task-shared attack.
InferD is a low-interaction honeypot framework built to measure attacks against AI inference infrastructure, such as LLM serving stacks, inference APIs, vector databases and agent gateways. It answers every observed request from static response pools and never executes adversary input, a principle the authors call deception without inference. The framework is organized into four deception planes and a provenance plane that links planted credentials to downstream model-mediated processing.
SEAL, a method for sanitizing tool outputs against indirect prompt injection in local LLM agents, learns a soft prompt and scoring readout offline and uses a frozen task model. On AgentDojo it reduces attack success from 34.6% to 0.1% and raises joint task completion from 51.4% to 77.5%, using about 127 times fewer defense-related tokens than PISanitizer. The authors also report results on AgentDyn and a second backbone.
This study tests six large language models with 400 adversarial prompts in Yoruba and Nigerian Pidgin, scored by native-speaker annotators. Models that rejected up to 69% of English prompts rejected as few as 0% in Yoruba, largely because they failed to understand the prompts rather than being safer. Attack success rates rose for most models in the Nigerian languages, showing that English-centric safety alignment does not transfer to these languages.
Task-Graph Contextual Integrity (TG-CI) extends Contextual Integrity to multi-agent systems by modeling workflows as a Multi-Agent Task Graph (MATG) and tracking how information propagates downstream and what each recipient has already disclosed. The authors evaluate two enforcement policies, Full-Scope and Auto-Region, on MAPLE-Bench, a 144-case benchmark. In their primary configuration, TG-CI cuts mean unauthorized disclosures by 85.8-94.1% relative to a Local-History CI baseline while keeping 93.6-96.1% of its task accuracy.
This empirical study tests five adversarial jailbreaking attack families against seven LLMs, including Llama-3-8B, Mistral-7B, Qwen2.5-7B and Gemma2-9B, and four defenses. Obfuscation attacks reach a mean attack success rate of 54.3% and bypass every evaluated model with nonzero success. Model size does not predict robustness (Spearman r=-0.57, p=0.18), and self-reminder prompting cuts ASR from 44% to 10% on Mistral-7B and from 48% to 5% on Llama-3.1-8B with no reported false positives.
Fix: Self-reminder prompting is the most consistent defense in the evaluation, reducing ASR from 44% to 10% on Mistral-7B and from 48% to 5% on Llama-3.1-8B, with zero false positives and no additional inference cost.
Arxiv (cs.CR + cs.CL + cs.LG)REALM is an approach for identifying and maintaining the assets an LLM agent's task requires, including assets discovered only during execution. It represents them as Dependency Graph nodes, separates discovering an asset from authorizing its use, and exposes resource-bound Partial APIs inside a Task Container. Across 60 clean tasks and 560 injected cases in AgentDyn, it completes 40.0% and 36.4% of tasks and reduces ASR from 16.6% to 0.2% with qwen3.5-flash as the base model.
Researchers tested whether self-evolving personal agents, which learn from user feedback, become less robust to indirect prompt injection, where hidden instructions in third-party content hijack the agent. In a two-phase setup on a real personal agent, the average attack success rate rose by 14.0 to 48.8 percentage points under skill evolution and by 3.0 to 39.3 percentage points under reinforcement learning, compared with a base agent that does not evolve.
The paper proposes Proof-Carrying Messages (PCM), a protocol in which each claim between AI agents carries evidence from an append-only Merkle log, with inclusion proofs signed by an attesting signer the agent cannot control. On a benchmark of 926 messages, 712 of them lies in 16 classes, PCM accepts 10 lies, all stale evidence (1.4%, 95% CI 0.8-2.6%), while plain A2A accepts all 712.
Researchers present SynIrisAttack, a benchmark dataset for detecting synthetic iris attacks. The paper also proposes a global-local residual modeling framework for the same detection task. The work appears in the Journal of Information Security and Applications, Volume 104, published January 2027.
A comprehensive survey reviews adversarial attacks and defenses in image classification, organizing the most representative methods chronologically and by category. It analyzes the strengths and weaknesses of each method and offers perspectives on future research directions. The survey was published in ACM Computing Surveys on 2026-10-29 (DOI: 10.1145/3831673).
An empirical study in Knowledge-Based Systems (Elsevier BV, published 2026-10-29, DOI 10.1016/j.knosys.2026.117270) examines cross-dataset generalization in LLM jailbreak prompt detection. The source text provided contains no further details of the method or findings.
Researchers study backdoor poisoning of deep Q-network (DQN) traffic signal controllers and their inference-time trigger activation. They propose DUIAPD, a detector that adds discretized random perturbations to each input and measures action-wise Q-value variability against a calibrated threshold, without training a surrogate network or updating DQN parameters. On a single SUMO intersection with four actions across 18 trigger configurations, it reports true positive rates of 92.14–96.38% and false positive rates of 1.51–6.40%, with generalization to other sensors, intersections, real-time use and closed-loop benefits still to be validated.