Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,003 items
CAESAR is an end-to-end multi-user Mixed Reality framework for systematically evaluating cognitive attacks in collaborative search-and-rescue training. It integrates four classes of cognitive attacks in a synchronized scenario and includes a multimodal data collection pipeline. A controlled user study of the latency-based attack showed measurable effects on task performance and user behavior.
This empirical study tests whether pretrained language models can serve as lightweight auxiliary supervision for supervised vision training, without paired image-text data, prompt engineering, or contrastive objectives. The authors evaluate Explicit Language Guidance (ExLG) and Implicit Language Guidance (ImLG) across six generalization regimes, including out-of-distribution, shortcut resistance, texture and shape bias, adversarial robustness, and continual learning. Explicit guidance helps in-distribution, low-data, and continual-learning retention, while implicit guidance helps more in shortcut-sensitive and strongly adversarial settings.
This research paper presents a unified attack against statistical generative watermarks for LLMs, including KGW and its variants, GumbelSoft, DipMark, ITS and SynthID. The authors identify a score-ratio alignment between detection scores and token probability shifts, which lets a black-box attacker without secret keys or model weights produce fluent text that evades detection. Evaluated on seven watermarking schemes and three LLMs, the attack reduces detection rates to at most 0.05 while preserving text quality.
The source argues that recent claims about AI models being so dangerous at cyber attacks and so capable of escaping control that they cannot be released are overstated. It describes the situation as a "bad AI" bubble, citing industry data, though the excerpt provided does not include the data itself.
This article introduces the identity fabric, a distributed identity control plane for autonomous AI systems. It is designed to constrain delegation, preserve intent, and enforce zero trust across high-velocity agent interactions.
The authors study how changes in frontier artificial intelligence may shift the balance between attack and defense. They examine consequences for critical infrastructure and institutional trust. The paper proposes governance recommendations grounded in the moral relationships that security controls are meant to protect.
This special issue examines automated programming language translation for security, judged on four metrics: correctness, performance, safety, and idiomaticity. Eight articles cover agentic and LLM-driven translation frameworks, static-analysis-based toolchains, coverage-guided testing strategies, and safe handling of tricky low-level constructs.
This essay argues that agentic LLM systems, though they seem well suited to translating legacy software into memory-safe languages, have not yet solved translation at scale. It frames achieving good correctness, performance, and idiomaticity as an open research challenge.
This paper argues that per-action authorization, where a local guard screens each tool call in isolation, cannot enforce global policies spanning intent, purpose, provenance, consent, and delegation in autonomous LLM agents. The authors prove that relational purpose violations are 2-safety hyperproperties that a per-action monitor cannot detect without false alarms, and that none of the six surveyed guardrails implements cross-execution monitoring. Using AL-SYNTH, a SAT/SMT synthesizer, they show nine LLM judges approve the synthesized witnesses almost without exception when viewed per action, yet flag them when shown the two executions together.
The authors introduce ABench, a handcrafted benchmark for testing LLM value alignment in Chinese, built on a hierarchical taxonomy of 158 value subcategories and 2,895 open-ended questions. The questions use role-playing and logical traps to test whether models keep their stances under adversarial induction, and the authors add a weighted automated evaluation framework. Across 16 mainstream LLMs, most models scored below 60 under these attacks, indicating significant vulnerabilities.
The paper introduces the show-up attack, in which a malicious subagent manipulates its supervisor into involving it in tasks beyond its assigned scope. The authors built SUA-Arena, which implements 6 attack strategies across 188 test cases, and tested 6 frontier LLMs. All were vulnerable, with GPT-5.6-Sol as supervisor reaching an ASR of up to 0.93.
Fix: Mitigation strategies such as prompting and restricted communication fail to reduce the risk consistently. The authors conclude that multi-agent systems need explicit mechanisms for enforcing authority boundaries.
Arxiv (cs.CR + cs.CL + cs.LG)AuthorizationBench is a benchmark of 286 executable tasks that tests whether LLM agents stay within the authorization a user grants, rather than only measuring task completion or harm. Across 6,864 trajectories in an authorization-aware sandbox, agents acted out of authorization in 19.2% of cases, and agents that consulted the principal before acting showed lower rates (14.0% versus 21.6%). Higher reasoning effort left the aggregate rate nearly unchanged.
KNOX is presented as the first regulatory sandbox for generative models, built by running AI workloads inside TEEs and demonstrated on agentic workloads. For compliance, it derives an attested policy manifest per user prompt and uses an attested reference monitor to allow only actions that match it, recording every action and verdict in an attested execution graph. The authors report that KNOX meets all four requirements, detects deviations such as those caused by prompt injection, and incurs less than 1% overhead.
Researchers propose Orthogonal Boundary Unlearning (OBU), a framework for removing hazardous operational capabilities, such as cyberwarfare exploitation and pathogen enhancement, from large language models while preserving benign dual-use knowledge. OBU restricts parameter updates to the orthogonal null space of a capability subspace extracted via singular value decomposition, and is realized as Projected LoRA or Closed-Form Weight Surgery. Tested on Gemma-3-4B, Qwen3.5-4B, SmolLM3-3B, and Granite-4.2-3B, the authors report strong concept separation, substantial mitigation of in-context priming attacks, and resistance to adversarial relearning without degrading benign competence.
This paper introduces Systemic Adversarial Robustness (SAR), a measure of how many distinct attacks are needed to compromise a pool of models that share training pipelines, distillation methods and data. Small SAR values indicate a vulnerability monoculture, while large values mean failures are spread across models. The authors formulate SAR as a partial set-cover problem with provable approximation guarantees, test it on vision and large language models, and report that model pools distilled from DeepSeek-R1 and Claude Opus 4.6 show lower estimated systemic robustness than their parent pools.
Ten widely deployed LLMs were evaluated across 79 languages (English plus 78 low-resource) for translation bypass, where harmful requests refused in English produce unsafe completions when translated. Conditioned on safe English completions, mean bypass rates range from 21.9% to 44.4%, and bypass risk concentrates in certain language families. The authors also find that per-model bypass rankings are unstable, with nine of ten models changing rank across three conditioning schemes.
This study tests eight code LLMs, from 6.7B to 15B parameters, on C programs drawn from the DELTASECOMMITS, SVEN and PRECISE-BUGS datasets. The models encode which code patterns a snippet instantiates with high fidelity, but vulnerability is more accessible as a paired relational signal than as a separable categorical label. Standard classifiers on isolated samples therefore reach limited accuracy, because vulnerable and secure samples overlap in representation space, even though the models reliably rank which of two snippets is riskier.
Profile-DP is a policy-aware local differential privacy framework for releasing user profiles in personalized LLM inference, bounding the release of facts users withhold while treating public instructions, profile schemas and authorized facts as public side information. Its two finite-domain mechanisms, DP-SLOT and DP-BLOCK, are evaluated on PEEP, where DP-BLOCK leads the utility composite at all 13 structured budgets and is closest to the ideal privacy-utility point in 18 of 26 format-budget settings. DP-BLOCK discloses less, while DP-SLOT retains more authorized facts.
Researchers introduce Speaker-Activated Backdoor (SAB), an attack in which an attacker compromises a speech model during fine-tuning and uses a target speaker's natural voice as the trigger, with no access to inference-time inputs. A two-phase fine-tuning method reaches 90.2% backdoor success on Whisper-medium and 97.0% on Parakeet, with false-positive rates of 1.7% and 0%, using as little as two minutes of the target's audio. In downstream tests, injected instructions drove GPT-4o and Claude agents to carry out the harmful action in up to 87.5% of cases.
Researchers propose safety mid-training, a stage between pre-training and post-training that jointly develops safety and general reasoning by treating safety as a learnable capability shared with general tasks. The base model generates structured documents and reasoning traces from safety and utility prompts for this stage. Relative to Qwen3-1.7B, 4B, and 8B base models, it reduces average attack success rate (ASR) by 72% to 81% while improving general capability, and the safety gains persist through subsequent SFT and GRPO.