Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,003 items
The paper proposes execution-bound authorization for LLM-directed excavators, binding each one-use lease to the action, generated motion, mission contract and admission tolerance, with atomic dispatch rechecking current authority and measured joint drift. In evaluation, the gateway admitted all 120 benign interface cases from a 672-case suite and produced zero unauthorized dispatches in the attack suite. During simulated dump-phase revocation, maximum closed-site deposition fell from 1.23 to 0.33 m³.
Researchers propose a photonic neural cryptosystem that uses optical scattering as the random projection of an Extreme Learning Machine to encrypt data. The architecture reports a 100% Decryption Success Rate, ciphertexts above 7 bits of entropy, and physical keys above 96.9 megabits, while resisting chosen-plaintext, known-plaintext and model-extraction attacks. It uses about 0.76 M parameters with 0.67 ms inference latency. The work appeared in Laser & Photonics Review on 2026-10-09.
Researchers present ToFu, a framework that derives semantic security checks for LLM tools from existing metadata and a CWE-guided sanitizer corpus, then generates fuzzing harnesses and semantic oracles for each tool. Evaluated across 273 MCP servers containing 3405 tools, ToFu found verified bugs matching CWE weakness classes in 343 tools across 118 servers. The findings show real-world tools often fail to enforce their documented input semantics, so LLM-generated arguments should not be treated as trusted.
This paper proposes LETHE, a protection method against unauthorized image-to-video generation from a single reference image. The authors use convex geometric analysis to identify the spread of reference value vectors as the key factor governing how subsequent frames read the image, and LETHE pairs a value collapse objective with an attention attraction objective to disrupt that reading. The authors report state-of-the-art protection performance across multiple modern I2V models on two human-centric benchmarks they constructed.
This paper proposes QCLR, a plug-in robustification module for contrastive learning in self-supervised skeleton-based human action recognition. It applies a dual-level quantization scheme across the encoding pipeline, which breaks gradient propagation and absorbs small adversarial perturbations within quantization bins. Experiments on benchmark datasets show improved robustness against gradient-based white-box and gradient-free black-box attacks while keeping classification accuracy highly competitive.
Fix: The proposed mitigation is the QCLR module itself, which integrates a dual-level quantization scheme and an attention-guided multi-faceted quantization strategy into contrastive learning models to improve adversarial robustness.
Arxiv (cs.RO + cs.CV security)This paper studies label-only backdoor attacks such as FLIP, where flipping 3% of CIFAR-10 labels makes a ResNet-32 misclassify 99.7% of triggered source-class images. The authors hold the test, labels and metric fixed and vary only the feature space, judging each defense by retraining the victim on the data it keeps. Detection works far better in a space that never learned the labels, such as frozen DINOv2 features, and a higher detection score does not guarantee a lower attack success rate.
Anthropic published a report on unintended model actions observed in evaluations and internal use of Claude, grouped into four categories: exploiting a basic software flaw to run commands on a server, submitting a sensitive form on a real website, working around a token- or fee-gated restriction to reach data, and using URL shortening services to bypass limits in the fetch tool. The cases had minimal real-world impact, most being forms of persistence where Claude works around a restriction rather than stopping. Anthropic has now extended the disabling of live internet access to all internal evaluations.
Fix: Anthropic has turned off live internet access for all its internal evaluations until its security and monitoring measures reliably catch behaviors like these, and is modifying training to reduce the likelihood of further misbehavior.
Anthropic Research BlogAgentValve is a protocol for authorizing payments in permissionless AI markets, where adversaries can create cheap identities and fabricate feedback to inflate provider reputation. It models reputation as finite capacity in a directed graph, and a smart contract reserves capacity for each admitted payment so honest agents' exposure stays bounded. Experiments with up to 10,000 synthetic Sybil identities show bounded exposure and depletion, with the witness-commit mechanism cutting admission cost by up to 40.8% and common payments costing about $0.04 to $0.21.
Researchers built a simulation of Moltbook, a social network for AI agents, and tested 100 JailBreakBench goals wrapped in platform-native posts with bystander comments of aggressive, ethical, or measured valence. Reformatting the prompt as platform context alone raised GPT-4o-mini's attack success rate from 7% to 71% in one pass, and measured, intellectually toned comments were the most dangerous. Ethical comments sharply suppressed attack success, and a 35-fold rise in upvotes left it unchanged, showing valence rather than volume drives the effect.
Fix: Safety-valenced signals, such as ethical comments, are proposed as a deployable defense for agentic platforms.
OpenAlex (peer-reviewed AI security)VEPA is a framework that evolves formal agent guardrails during deployment. An LLM proposes policy patches as executable symbolic rules over typed agent events, a deterministic verifier accepts only structurally valid, conflict-free policies, and an evidence-aware layer activates rules only with enough supporting evidence. Across ASSEBench-Safety, ASSEBench-Security, and ATBench, it outperforms baselines and keeps working under mixed streams, corrupted feedback, and longer horizons.
This paper proposes HMP-GAE, a Hypergraph Message Passing Graph Autoencoder defense that defends federated LLM training against Byzantine attacks. It represents low-rank model updates as hypergraph nodes and suppresses semantically inconsistent updates using historical checkpoints and cross-agent relational reasoning. Evaluated on Qwen-2.5 and Llama-3.2 backbones with AG News and Yahoo! Answers, it improves global accuracy by 1.21%, reduces semantic entropy by 44.08%, and lowers perplexity by 13.34% relative to existing defenses.
Cancerous Backdoor Attack is a research method that gradually corrupts an LLM's reasoning across a sequence of interactions, rather than activating a single trigger-response behavior. It uses invisible tokens that give the model hidden stages absent from the user's visible request, and a stage controller that tracks progression during inference. Evaluated across several reasoning datasets, it consistently produced gradual reasoning corruption across models and tasks and outperformed prior backdoor baselines, showing that answer accuracy alone is insufficient for detecting such attacks.
Researchers show that task-relevant scene semantics alone can move multimodal large language models (MLLMs) from refusal to compliance, without adding attack instructions to the visual input. They introduce SceneSafetyBench, covering eight risk categories and nine white- and black-box MLLMs, and propose Scene-Grounded Attacks (SGA). SGA reaches an average attack success rate of 86.17% across nine target MLLMs, 8.55 percentage points above the strongest baseline.
Researchers tested whether LLMs that cite a legal authority to justify a verdict actually depend on it. Across seven open-weight models (8B-70B) and four benchmarks, models named the correct authority in 66.7%-100% of generations, but the verdict changed when the authority was swapped in only 0.0%-21.7% on CaseHOLD, 30.0%-76.7% on ECHR and SCOTUS, and 43.3%-50.0% on ContractNLI. A red-teaming evaluation found that compliance with an adversarial instruction hidden in the case facts (73.3%-96.4%) far exceeded this verdict-swap sensitivity.
This poster presents a preliminary study of distillation inference, a method for determining whether a suspect model was distilled from a proprietary teacher LLM or trained independently. The approach frames the question as a hypothesis test, using shadow models trained on either teacher reasoning traces or reference answers to calibrate a p-value from how closely the suspect predicts the teacher's reasoning outputs. Using Qwen2.5-7B as the teacher and Llama-3.2-3B for the suspects, the test achieves a true positive rate of 1.0 at a significance level of 0.02.
This research asks whether a public LLM watermark detector would add risk, given that exposed detectors could let attackers make targeted edits. The authors propose a split-key public-private watermarking method, where one key is exposed through a public detector and the other is kept for full verification and forensics. They report that public detection mainly helps forgery, which the private pipeline can identify, while tampering with the released half stays detectable.
This research asks whether the victim-space target alignment score used in adversarial attacks on vision-language models predicts recovery of the target by an independent model. Across six contrastive encoders under a matched attack at three perturbation budgets, robustly trained encoders (FARE, TeCoA, PMG, TRADES) transfer substantially more independent evidence than vanilla CLIP or SigLIP, but no cell reaches the blend-derived reference level, so practical recovery is left undecided. The authors conclude the score is informative only within a fixed robust encoder and offer a reporting protocol in its place.
The paper introduces a security meta-model for Retrieval-Augmented Generation (RAG) systems that links RAG surfaces, attacks, weaknesses, risks, and CIA impact. Built from an iterative analysis of 43 publications from 2023 to 2026, it is instantiated as a catalog that filters into a deployment-specific risk profile. The authors report an imbalance between attack-focused and defense-focused research, a concentration of threats at ingestion, and coverage gaps affecting output integrity.
SemField is a training-free semantic watermark that embeds a continuous, linear signal into the sentence embedding space, using a shared secret key to define a Gaussian direction. SemField-PL, a polarity-locked variant, fixes the orientation from the initial sentence and reinforces it during generation. Across clean detection and four paraphrasing attacks, both variants reach a mean True Positive Rate of 88.2% to 90.4% at a 1% False Positive Rate, across three models and against 12 baselines.