Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,003 items
This research shows that instructions hidden in an image, using LSB, DCT or metadata steganography, can bypass text scanners and enter a multi-agent MCP workflow when a tool decodes them. The authors propose decode-time provenance binding, which combines HMAC-authenticated trust labels, an MCP-maintained provenance ledger and context-taint tracking. In a testbed with 24 adaptive payloads, the undefended pipeline had 100% attack success, keyword and regex defenses each permitted 62.5%, and the proposed mechanism blocked all attacks.
Fix: Decode-time provenance binding: HMAC-authenticated trust labels, an MCP-maintained provenance ledger and context-taint tracking, which label decoded content as untrusted and block critical tool calls influenced by that content. Code and data are publicly available.
Arxiv (cs.CR + cs.CL + cs.LG)SEMPRE is a fully automated protocol reverse engineering framework that replaces hand-written rules with recursive LLM reasoning grounded in dynamic program analysis, using taint-flow and loop traces. It segments messages into fields, detects nested sub-payloads, infers field types and roles, and refines boundaries in an EM-style loop until convergence. Evaluated on mainstream protocols against six baselines, it reports a field-boundary score above the baselines' 0.4 and 0.72 for semantic inference versus 0.13 for BinPRE.
Researchers tested whether LLM negotiation agents can build useful trust estimates from prior interactions in repeated multi-party, multi-issue games where one participant covertly tries to block agreement. Trust-aware agents identified the adversary within a few games, but accurate trust did not consistently yield better agreements. Explicitly guiding agents to use trust raised agreement rates by up to 23 percentage points.
This paper shows that safety filters on humanoid robots guarantee collision avoidance only when their safety parameters are configured correctly, and that no general method exists to set them across deployments. The authors present goal insertion attacks, which use normal user interactions without extra privileges, and build SIREN, an automated tool that identifies vulnerable goals for a given task and environment. SIREN finds attacks in all vulnerable scenarios in the SPARK dataset, discovers three times as many as a baseline approach, and causes a real humanoid to collide while executing tasks.
Ulysses is a multi-agent security harness that turns candidate web application vulnerability findings into verified Proofs of Exploitability by testing exploit hypotheses against independently observed execution effects. Evaluated on WebExploitGym, a benchmark of exploitable instances from 25 real-world web applications across 5 languages and 7 frameworks, it produced verified proofs for 10 of 25 real-world advisories and discovered 4 new proofs of exploitability never previously reported.
Researchers present Social-HUMINT, an LLM-augmented framework that measures how accurately a person can be profiled from public social media data and how far that profile enables social engineering. They unify SOCMINT and Cyber-HUMINT into a 38-capability taxonomy validated by domain experts (Fleiss' κ = 0.92). A multi-agent pipeline reached F1 = 97.3% for extraction and 4.73/5 for inferred attributes, and predicted participants' engagement choices above chance.
This paper couples continuous-space black-box attacks on LLMs to a continuous diffusion language model (dLLM), replacing the random latent-to-token mapping used by prior work. Character-level perplexity falls from 10^3 to 10^7 down to 3 to 5 on RareTrap, and from 20 to 28 down to 4.3 to 5.4 on ThinkTrap. The authors also report that model risk estimates can differ by an order of magnitude between the two mappings.
METR and Redwood Research staff ran a three-week pilot red-team exercise, July 21 to August 11, against OpenAI's internal misalignment monitoring system, with deep access to its codebase, monitor prompts and configuration, and interviews with about a dozen engineers. The exercise produced a 61-page internal report focused on how much agentic inference is monitored and whether agents could disable or subvert the monitor. The authors found that roughly 75–94% of eligible Responses API traffic from models at least as strong as GPT-5.5 was monitorable at the time, with 99.2% of internal Codex traffic eligible, while compaction events went unmonitored.
AgentPhishing is a benchmark that tests whether persuasive phishing emails can evade detection and get LLM email agents to comply with malicious requests. Its PhishOpt method optimizes reusable email-generation skills against individual agents, and the authors build a universal skill from persuasion-principle analysis. In an evaluation of nine agents, four had phishing-detection rates below 10% and malicious-request compliance rates above 45%.
AfterDeleteBench is a controlled behavioral audit that tests whether LLM agents still reveal information after its source memory record is deleted. In the primary setting, one query recovered a deleted synthetic secret from Qwen2.5-7B-Instruct in roughly two thirds of cases, while matched never-exposed controls showed zero recovery. Reconstruction succeeded in 62-99% of cases across Qwen, Mistral, and Phi-3 when only separately stored components remained, and provenance-aware closure eliminated observed recovery in the tested settings.
Fix: Provenance-aware closure eliminated observed recovery in the tested positive-baseline settings. An exact-literal write filter blocks reinsertion but cannot erase information already present.
Arxiv (cs.CR + cs.CL + cs.LG)The paper shows that a harmful response generated for a query can serve as a jailbreak template for diffusion large language models (dLLMs). Masking a subset of its tokens induces the dLLM to reconstruct the harmful content. The authors propose RMT-Jailbreak, which transfers existing autoregressive (AR) jailbreak attacks to dLLMs by converting AR-generated harmful responses into high-mask-ratio templates. Experiments across multiple AR jailbreak methods and target dLLMs show strong cross-paradigm transferability.
Researchers show that a harmful response to a query can serve as a jailbreak template for diffusion large language models (dLLMs). Masking a subset of its tokens causes the dLLM to reconstruct the harmful content. They propose RMT-Jailbreak, a framework that transfers existing autoregressive (AR) jailbreak attacks to dLLMs through these induced harmful responses, and report experiments across multiple AR jailbreak methods and target dLLMs showing strong cross-paradigm transferability.
The authors introduce WildFakeToolBench, a benchmark of 156 executable image-processing tools derived from 220 public web sources across 142 domains, built to test whether such tools can help evade AI-generated image detectors. On 105 images from five generators, the strongest individual tools reach attack success rates of 25.25% to 100% across six detectors, and eight-tool subsets reach worst-case coverage above 80% for every detector. The authors note that effectiveness is sparse and detector-specific, so broad coverage does not imply reliable one-shot evasion.
METR and Redwood Research staff ran a three-week pilot exercise, July 21 to August 11, with deep internal access to stress-test OpenAI's internal misalignment monitoring system, producing a 61-page internal report. The exercise found that roughly 75–94% of agentic internal Responses API traffic from models at least as strong as GPT-5.5 was eligible for monitoring at the time, and that compaction events went unmonitored for roughly 10% of eligible internal Codex traffic between May and July 2026. The authors state that OpenAI fixed the compaction gap during the exercise and that the findings describe the historical system, not its expanded current coverage.
Fix: OpenAI staff fixed the compaction monitoring gap during the exercise. The source discusses no further mitigation for the other findings.
METRResearchers present LLVMhammer, a Rowhammer attack that induces a bit-flip in Triton's compilation pipeline so that it generates a functionally identical but suboptimal GPU kernel. Because Triton caches the kernel on disk, one successful bit-flip causes persistent slowdown across later inference runs. Tested on Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Mistral-7B-Instruct-v0.3 and Deepseek-llm-7B-chat on NVIDIA RTX 4090 and AMD RX 7900 XTX GPUs, the attack raised inference latency by 4-16% and could reduce BERTScore to as low as 25%.
Omni-UBench is a benchmark that evaluates omni-modal guardrail models on safety, utility, and reliability, using over 183k samples from 35+ sources spanning text, image, audio, video, and cross-modal combinations. Tested on 15 representative OLLMs and omni-modal guardrails, it finds that modern OLLMs show strong content moderation potential even without guardrail-specific tuning. Existing guardrails still miss cross-modal queries that conceal harmful intent or use sophisticated jailbreaks, and they show safety hallucination and over-sensitivity to textual patterns.
TokenTrap is a defense that redirects suspected LLM-based scam systems toward benign, low-value computation while withholding credentials and payment information. It combines a token-task library, a router that selects among single-agent, hierarchical, and flat multi-agent workflows, and a bounded anti-guardrail strategy with token, round, time, and safety limits. Across 50-round traces it amplified token consumption by 2,487.0x, 1,616.0x, and 567.8x relative to a no-token-task baseline, and its router reached 85.67% accuracy on 300 architecture-labeled prompts.
The paper proposes a training-free fingerprinting method for LLMs that works on weights alone. It builds spectral scalar features from the singular-value distributions of GATE projection weights in gated feed-forward networks, pools them by depth into a model-level fingerprint, and compares fingerprints with linear centered kernel alignment. Experiments on representative LLM families report clear separation between related and unrelated models and stable similarity under instruction tuning, quantization, pruning and distillation.
HiMark is a multi-bit watermarking framework for LLM-generated text that embeds provenance information using hierarchical candidate control and tournament elimination during generation. Its model-agnostic detector aggregates token-level evidence to extract the watermark. In experiments, it reached 90.01% extraction accuracy under a paraphrasing attack with 32 bits embedded in a 300-token text, while maintaining text quality.
This paper introduces Enterprise Agentic RAG, a production-grade architecture built on a stateful 7-agent LangGraph workflow with Pre-Search Vector Payload Filtering across four security clearance tiers. It combines BM25 with Qdrant HNSW dense retrieval fused by Reciprocal Rank Fusion, plus an egress DLP firewall and deterministic guardrails. Reported benchmarks include 100% RBAC pre-filter precision, zero embedding leakage across privilege boundaries, a 98.7% average groundedness score, and sub-150ms latency.