Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,003 items
This paper presents a use case for AppSec Core, a bounded application-security ontology, to make engineering-actionable requirements from scattered security authorities addressable as a typed working set. It shows two entry points: a forward path from a code change to its governing obligations, and an inverse path from a normative duty such as NIS2 to the engineering obligations it implies across domain slices.
Researchers evaluated computer-vision and motion-based defenses for detecting uncrewed surface vessels (USVs) disguised as benign craft. Adversarial hull patches suppressed detection in up to 29% of attempts but never caused benign misclassification, while physical disguise bypassed both camera and radar presence checks. An asset-relative closing-rate check flagged 94-95% of direct run-ins within 4 nautical miles by recreational-labeled vessels at 15-20 knots, versus 12% for a track-only speed check, with a 4.9% held-out false-alarm rate overall and 12.3% on recreational traffic.
Hush-Up is a stealthy availability attack against distributed LLM inference in which one malicious worker halts generation while evading attribution. The attacker profiles its local hardware error bound, then optimizes an adversarial perturbation on its output activations to maximize the end-of-sequence (EOS) token probability at the final layer. The authors report that the attack forces premature termination across diverse models and benchmarks and can evade or mislead existing verification defenses.
The paper introduces Pop Goes the Verifier, a data poisoning attack against VoicePop, a liveness detection technique in Automated Speaker Verification Systems that classifies low-frequency pop bursts from plosive phonemes using GFCC features and an SVM. The attack uses FastSpeech 2 with a modified HiFi-GAN vocoder to inject localized pop bursts at plosive locations, plus subtle Gaussian noise to reduce visibility. On ASVSpoof 2019, it achieves a moderate poisoning effect and resists signal-level inspection better than the prior SyntheticPop attack, which reduced accuracy to 14.01% with 20% poisoned samples but produced easily detectable artifacts.
The paper proposes Illusion, a proactive defense that conceals the communication topology of LLM multi-agent systems by injecting fake edges and fake messages. Low-impact candidates are chosen to preserve output-distribution equivalence and task accuracy. In experiments, edge-level F1 drops from over 0.87 to 0.62 and graph similarity from over 0.94 to 0.71, and the fake edges resist similarity-based and LLM-judge filtering, even under adaptive pruning.
Fix: The proposed mitigation is Illusion, a proactive topology obfuscation defense that injects fake edges and fake messages into LLM-MAS communication to present attackers with a deceptive topology.
Arxiv (cs.CR + cs.CL + cs.LG)This survey, published in ACM Computing Surveys (Volume 59, Issue 1, pages 1-34, January 2027), reviews vulnerabilities and defenses in audio-visual attacks, spanning audio-only systems through multimodal models. The source text provided contains only the publication citation, so no further findings or methods are described.
AgentCap is a runtime authorization layer that takes the authorization decision out of the language model and mediates each tool invocation through an MCP wrapper. It checks declared intents against policy constraints, then issues short-lived, budget-bounded capability tokens validated at the tool boundary. Across AgentDojo and InjecAgent it reduced attack success in every evaluated setting, reaching 0% with GPT-5.4 on both benchmarks and blocking all attempted unauthorized executions in controlled adversarial scenarios.
Fix: AgentCap itself is the mitigation: a runtime authorization layer that mediates every tool invocation through an MCP wrapper, validating short-lived, budget-bounded capability tokens at the tool boundary, with delegation limited to session-bound parent-child chains.
Arxiv (cs.CR + cs.CL + cs.LG)Researchers evaluated whether maritime defense systems can separate uncrewed surface vessels from benign craft using shore cameras and radar. They found adversarial hull patches suppress detection in up to 29% of attempts without causing benign misclassification, while physical disguise can bypass both camera and radar checks. An asset-relative closing-rate check flagged 94-95% of recreational-labeled direct run-in attacks within 4 nautical miles at 15-20 knots, versus 12% for a track-only speed check, with a 4.9% overall held-out false-alarm rate (12.3% on recreational traffic).
The SecPO paper addresses prompt injection, where instructions in untrusted data manipulate an AI agent. It argues that Meta-SecAlign's DPO-based training anchors to an insecure reference model and uses weak simulated attacks, and proposes a SecPO loss plus an adversarial training pipeline that synthesizes strong injections (best@240). On Qwen3.6-27B, the authors report AgentDojo ASR@800 of 8.8% versus 78.8% for Meta-SecAlign under AttackerMovesSecond, with no noticeable benign utility loss across nine benchmarks.
Fix: The proposed defense is the SecPO loss combined with adversarial training (AT) on synthesized injected inputs; the authors release code, training data and a model at the linked GitHub and Hugging Face locations.
Arxiv (cs.CR + cs.CL + cs.LG)Researchers study whether observers can infer user attributes from MCP tool-call traces alone, without access to user conversations. They introduce PrivacyTrace, a benchmark of 1,000 synthetic user profiles and 4,000 agent trajectories across four domains, built with real MCP tools from 25 servers. Four LLM attackers reach a 26.25% attack success rate for profile inference, compared with 17.97% from prior-based guessing, with tool arguments and returned results as the main evidence and cross-session trace linking amplifying the risk.
This research audits eleven evaluations of detectors that flag hijacked LLM agents from their tool-call traces and finds attack rates of 16% to 86% with no variation tested. On AgentDrift and AgentDojo, a detector with AUROC 0.879 reaches precision 0.076 at a 2% attack rate, and detector rankings reverse as tool coverage falls below about 50%. Detectors trained on AgentDrift rank AgentDojo traces at chance, while the AlignmentCheck monitor reaches AUROC 0.930 but would still mostly raise false alerts at that attack rate.
Fix: The authors turn their findings into a checklist for evaluating trace detectors, but the source text does not state any specific fix or workaround. Dropout on tool inputs is reported to largely remove the ranking reversal, which is a finding rather than a recommendation.
Arxiv (cs.CR + cs.CL + cs.LG)Researchers re-ran OLMo 2 1B's 50-billion-token mid-training stage, swapping in 7,814 matched document pairs with a coin flip deciding which copy was trained on. A single exposure raises a document's average likelihood but rarely identifies it: the reference-ratio attack reaches an AUC of 0.767 yet catches only 3.8% of members at a 1% FPR. Paired comparisons across 13 matched pairs give 80% power to detect training on a collection.
Researchers evaluated AI control monitors, which must catch a capable model sabotaging its work, using a standard dataset of honest versus backdoored code. Simple features that never execute the code outranked 20 of 24 open-weight models prompted as monitors, because the dataset leaks its labels through authorship style and problem characteristics. Swapping in Claude Sonnet 5 code showed that these features exploit the contrast between authors, and one in seven backdoors in the test set fails validity checks, distorting comparisons. The authors release the rebuilt dataset and checks.
NormDetect is a detector for machine-generated web text that anchors a learned per-section head to the normalized Binoculars zero-shot score, using the Falcon-7B scoring pair. On eight sections of the RAID benchmark with eleven attacks, it raises recall at a 0.1% calibrated false positive rate by 5.0 points over normalized Binoculars, while keeping the human false positive bound at most 1% in every section.
The paper proposes a privacy-preserving paradigm for pre-execution safety auditing of LLM agent tool calls, separating delegated audit computation from plaintext access and execution authority. Its instantiation, SealCall, combines learned audit representations, CKKS-based encrypted similarity computation, and execution control bound to audited tool calls. On 1,252 public binary candidates it reaches a mean F1 of 0.840 under grouped nested evaluation, and one configuration completes each encrypted scoring operation in 0.176 s.
ERASE is a defense against multimodal indirect prompt injection that replaces the injected region of image, audio, or video with modality-appropriate blank content while keeping the rest of the media. A student model learns from a frozen base model reading the erased view through on-policy self-distillation. On 3B audio it reaches 60.1% attacked-input accuracy versus 33.1% for ARGUS, within 0.5 points of the undefended clean-input level.
Fix: The source describes ERASE as a proposed defense method, not a fix for a specific vulnerability, and states the code will be publicly available upon acceptance.
Arxiv (cs.CR + cs.CL + cs.LG)Researchers introduce REEF, a benchmark of 9 verifier-grounded websites that pairs benign web-agent tasks with malicious objectives delivered through 5 attack types, and ANGLER, an adaptive attack framework that uses execution feedback to mutate attacks. Across 8 VLM-based web agents, ANGLER reaches a 20.36% end-to-end attack success rate versus 15.24% for a non-adaptive setting. It also cuts benign task success from 61.9% to 44.4% compared with attack-free execution.
The authors introduce SHAKTI, a Hindi benchmark covering four genres, eight LLM families, multiple prompting strategies, and adversarial transformations, for detecting AI-generated text. They also propose TRIDENT, a round-trip translation detection framework with static, dynamic, and logistic-regression variants, where TRIDENT-S achieves the best overall performance and outperforms conventional and supervised baselines. The framework shows multilingual applicability and robustness across seven languages and adversarial settings, though it is sensitive to the translation model.