Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,040 items
This paper proposes a robust coverless image steganography (CIS) method designed to resist black-box attacks. It builds a hierarchical coverless image dataset (CID) by computing the normalized average of Scale-Invariant Feature Transform (SIFT) keypoint descriptors, then maps hash sequences to images for hiding and extracting secret information. According to the authors, the method achieves higher capacity and stronger robustness than state-of-the-art CIS methods, and it is the first to consider robustness against black-box and mixed attacks in CIS.
The authors propose a self-supervised learning framework for soft robot proprioception, a task made hard by the complex, compliant bodies of soft robots. The method pretrains a network on abundant unannotated sensor data, then fine-tunes it with a small set of annotated samples. Validated on a high-resolution 3-D morphological reconstruction task using a publicly available dataset, the approach needs only about 1/20 of the annotated samples to outperform a fully supervised method.
The paper presents AdU-Net, a framework that pairs a dilated U-Net with nested connections and an adaptable vision transformer encoder (AVi-TE) to extract glacial lakes from Landsat 8 imagery of the Imja, Chandra and Bhaga regions. A modified spiking neural network then analyzes the extracted lake features over time to model glacial lake outburst flood (GLOF) risk. The authors state that combining spatial and temporal analysis gives a more complete view of lake dynamics and the factors behind outburst susceptibility.
Researchers built a novel adversarial dataset to test the dual thinking framework, which separates fast intuitive processing from slower logical processing, in human vision and in deep learning models. Psychophysical studies showed multiple inferences in rapid succession, and errors arose when early stopping of visual processing missed relevant information. The authors report that multimodal large language models and vision language models have improved more on intuitive processing than on logical processing, while segmentation models lack understanding of substructures.
This paper examines the obstacles that cyber supply chain risk management (C-SCRM) programs face when operations depend on the lifecycles of digital components, including hardware, firmware, software, and services. The authors combine academic literature, historical incidents, and existing standards to identify these obstacles.
The authors critically analyze 21 differential privacy (DP) deployments by top-tier companies and institutions over the past decade. They argue that applying DP in real-world scenarios has been challenging.
The Model Context Protocol published its initial 2025-06-18 revision of the specification. SDKs will adopt this version at their own pace, and the prior version may stay in use for an undetermined period. The spec defines version negotiation between clients and servers to support backwards and forwards compatibility.
This paper proposes METT, a divide-and-conquer detection method for data poisoning attacks on recommender systems. It combines causality inference over group-level and individual-level unfairness sequences, an early attack-target detection method named ideaT, and a disturbance tolerance mechanism for fake injections. On synthetic and real data, METT reduces the false alarm rate (FAR) by averages of 21% for S-attacks, 18% for profile injection attacks, 26% for reverse attacks, and 17% for optimal-injection attacks, and by 10% and 20% for hybrid promotion and demotion attacks.
Fix: METT, a divide-and-conquer detection method for data poisoning attacks on recommender systems.
IEEE Xplore (Security & AI Journals)Researchers propose the constraint transformer (CT), a method for inferring safety constraints from offline healthcare treatment records, since existing inverse constrained reinforcement learning assumes Markovian decisions and real-time interaction. CT uses causal attention over historical decisions and observations, a non-Markovian weighted-constraint layer, and a generative world model for data augmentation. In several medical scenarios, CT captured unsafe states and approximated lower mortality rates while reducing unsafe behavior probability.
Researchers introduce a prompt-guided watermarking framework for LLM outputs that works at the input level and needs no access to model parameters or decoding logits. It uses three cooperating LMs: a prompting LM that writes watermarking instructions, a marking LM that generates watermarked outputs, and a detecting LM that classifies whether a response carries a watermark. Across 25 prompting and marking LM combinations, including GPT-4o, Mistral, LLaMA3 and DeepSeek, the signals generalize across architectures and remain robust under fine-tuning, distillation and prompt-based adversarial attacks.
This paper evaluates existing online safety analysis methods, which detect unsafe LLM outputs during generation rather than after it. The authors run a pilot study showing unsafe outputs can be detected early, then release the first public benchmark covering multiple methods, models, tasks, datasets and metrics. They also test hybrid methods that combine several detectors into one safety conclusion.
Researchers introduce the certified local transferable region, a connected area of inputs where a single adversarial perturbation is mathematically guaranteed to fool a deep neural network. They propose reverse attack oracle-based search (RAOS) to estimate the maximum size of this region, using a minimal distance attack and state-of-the-art verifiers at each step. The region's size serves as a metric for the local transferability of perturbations across model structures and adversarial training scenarios.
Researchers propose a steganography scheme for large language models built on the transformer architecture. The sender encodes secret data by using matrix multiplication to modify the coverage parameters of a specific transformer block during training, and a receiver with a secret key extracts it from the stego model. The authors report experimental results on embedding capacity, undetectability, and robustness.
The paper proposes a defense against model-extraction attacks on data-driven healthcare APIs built on electronic health records. The method applies adaptive teleportation to incoming queries, guided by a bi-level optimization target and Wasserstein gradient flows, to push attack queries across decision boundaries while limiting changes to benign queries. Tested on in-hospital mortality, bleed risk and ischemic risk prediction, it suppresses cloned-model performance while keeping comparable serving utility to existing defenses.
Fix: The proposed defense is adaptive teleportation of incoming queries, generated from the formulated bi-level optimization target and following Wasserstein gradient flows (WGFs), to mislead attacking queries while preserving benign-query utility.
IEEE Xplore (Security & AI Journals)OWASP's Agentic Security Initiative (ASI) has a taxonomy called Agentic AI – Threats and Mitigations. Three developer tools, PENSAR, SPLX.AI Agentic Radar, and AI&ME, now adopt it to help teams test, defend, and build agentic systems. The ecosystem is also informing the forthcoming OWASP Top 10 for Agentic AI.
OWASP's Q2 2025 Gen AI Incident & Exploit Round-up (March to June 2025) is a semi-regular roundup of documented exploits and vulnerability research involving generative AI. Its featured entry describes a GPT-4.1 jailbreak via tool poisoning, in which attackers embedded malicious instructions in tool descriptions, causing the model to perform unauthorized actions such as data exfiltration without user awareness. The entry maps the attack to several OWASP Top 10 for LLM categories, including LLM01 Prompt Injection and LLM07 Insecure Plugin Design.
Fix: Implement strict validation and sanitization of tool descriptions. Establish permissions and access controls for tool integrations. Monitor AI behavior for anomalies during tool execution. Educate developers on secure integration practices.
OWASP GenAI SecurityGoogle DeepMind researchers evaluated whether current frontier models have the prerequisite capabilities for AI scheming, namely stealth and situational awareness. They built and open-sourced an evaluation suite and tested Gemini 2.5 Pro, GPT-4o and Claude 3.7 Sonnet as of May 2025. The most capable models passed 2 of 5 stealth challenges and 2 of 11 situational awareness challenges, which the authors read as no concerning levels of either capability.
CyberRisk Alliance (CRA), a business intelligence company serving the cybersecurity ecosystem, announced a strategic partnership with the OWASP Foundation and the OWASP Gen AI Security Project on June 26, 2025. The partnership includes joint content, events, and research initiatives aimed at application security and AI education.
This case study uses a fictional insurance company, based on real events, to examine how generative AI (GenAI) can both trigger and defend against cyberattacks. It argues that GenAI raises the stakes for both white-hat and black-hat attackers. It frames the central dilemma for IT managers and executives: adopt AI as a defense strategy, or risk being exposed to threat actors using GenAI.