Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
2,003 items
The paper proposes PARS, a reference mechanism for coordinating policy-triggered key establishment, rotation, and reconciliation among authorized Physical AI agents, to limit temporal key exposure in post-quantum Edge AI and cyber-physical systems. It also describes PARS-ESL, an optional permissioned ledger that records authenticated key-epoch state, membership, revocation, and recovery checkpoints without storing high-volume operational traffic. The authors report that ledger-assisted post-quantum key-epoch management is feasible for selected security-control workloads under stated assumptions.
EdgeDeID is a 0.6B-parameter decoder-only clinical de-identifier trained entirely on synthetic notes, with no real patient text. It reaches 0.994 micro-F1 on a 100-note synthetic benchmark, against 0.857 for the strongest of eight off-the-shelf systems, and drops to 0.48 F1 on MIMIC ICU nursing notes. Three hand-written in-context examples recover 0.08 F1 without annotated target data.
Redwood-style researchers released a paper on simulated control evaluations (SCEs), a method for testing whether AI monitors would catch a model deliberately attempting harm. Instead of executing an attacking model's tool calls, an LLM simulator fakes tool outputs and an LLM judge decides whether the attack would have succeeded. In LinuxArena, the gap between simulated and real results was often about as small as the gap between two independent real runs, and the authors say SCEs are faithful where they can check, though extrapolation to deployment involves some uncertainty and long-horizon, multi-agent attacks lack simulators.
This research examines query-time filters that screen retrieved documents to defend against RAG poisoning, showing their reported near-perfect detection depends on how poison sets were built rather than on what the filters read. Removing a single construction cue, a target-question prefix in PoisonedRAG's released code, cut the content filter RAGuard's AUROC from 0.953 to 0.758 and raised its miss rate from 0.191 to 0.763. Padding poison to clean length with word-shuffled filler drove TrustRAG's set-level miss rate from 0.000 to 0.827, while AUROC held. Detection results should be reported with the poison set's construction profile and the false-positive rate at the deployed threshold.
This study compares LoRA with full fine-tuning for prompt-safety classification, using a matched 400M encoder-decoder pair (Ettin) across three task formats and up to 15 seeds per cell. At pre-specified learning rates, full fine-tuning leads LoRA by 2-3 pp, but on the encoder the gap closes when both learning rates are chosen by the same search. Trained models beat a prompted 8B in domain yet trail it by 8-23 pp on adversarial jailbreaks and flag 27-66% of benign but harmful-looking prompts as unsafe, versus 2-3% for the 8B.
This research studies LLM-based vulnerability detection in C source code across three code representations: raw source, Abstract Syntax Trees (AST), and Control Flow Graphs (CFG). The authors propose a Representation-Aware Logical Ensemble (RALE) that combines LLMs trained on different representations using explicit logical rules and majority voting. RALE reaches a 78% F1-score, compared with up to 73% for single-representation models.
Barezzani, Coscia, Ferrari and Piuri publish a study in Computers & Security, available online 24 October 2026, on distributional leakage in diffusion models. The source text provides only the title, publication details and author list, so the research question, method and findings cannot be stated from it.
A PRISMA-guided systematic review of 72 studies from 2020 to 2025 analyses Agentic AI, reinforcement learning, LLMs and tool-augmented Generative AI for autonomous vulnerability assessment and penetration testing across IT and OT environments. It finds agentic approaches suit sequential tasks such as reconnaissance orchestration and lateral movement, while generative approaches excel at reasoning over unstructured evidence and producing traceable audit reports. The authors report that none of the coded studies was evaluated against a physical or emulated OT/IIoT target, and they name hallucination control and prompt injection as open challenges.
The paper, published in the Journal of Information Security and Applications in January 2027 (Volume 104) by Jiayu Li, Yu Li, Tanping Zhou, Wei Ke, Shihao Wang and Xiaoyuan Yang, proposes VGPD-FL, a method for group-level poisoning detection in federated learning. The source text provided is limited to publication metadata, so the method's design, evaluation and findings are not described here.
This research measures how LLM vulnerability auditors respond to static-analysis notes, using 916 PrimeVul function pairs, eleven model configurations and 16,572 audits. Fabricated assurances that a missing check exists lowered detection of real vulnerabilities by up to 19 points relative to true notes, while fabricated alarms rarely reduced accuracy. Verifying each claim in a separate call and showing the refuted claims raised accuracy under false assurances by 12 points in a pre-specified test (p=0.002), though the gain for a second model was smaller and not significant.
Fix: Pipelines should verify a tool's guarantees before the model relies on them, and studies should measure deference in a direction-aware way. Verifying each claim in a separate call and showing the refuted claims improved accuracy under false assurances in the pre-specified test.
Arxiv (cs.CR + cs.CL + cs.LG)NTBench, a new benchmark, tests whether LLMs and agents resist near-target prompt injection, where injected instructions produce answers that look legitimate but are incorrect or harmful. It builds attack targets from legitimate reference outcomes across ten language datasets and two agent benchmarks, evaluating seven LLMs. Under three attack executions per sample, both incorrect-target and harmful-target tracks raised attack success over the fixed-target baseline for every model, by 9.3 to 35.8 points and 1.7 to 17.3 points on language tasks respectively.
Researchers present a lightweight inference-time framework for localizing and steering safety-sensitive representations in frozen vision-language models, without retraining. On Qwen3-VL-4B, steering along a learned safety direction in early vision layers gives bidirectional refusal control, with unsafe rejection spanning 6%-96%. The authors report that deeper layers stay separable along that direction yet respond little to steering, and that InternVL2.5-4B shows similar depth-dependent geometry.
KnotScryer detects insecure compositions of individually clean agent skills by modeling skills as typed capability flows and building a Skill Capability-Flow Graph of cross-skill information flow. Applied to 13,598 clean skills from ClawHub, it identified 612 insecure compositions, raising the exploitation rate under GLM-5.1 from 16.5%-18.2% to 85.6% while cutting the search space by 85.3%. Of the detected chains, 13.7% consisted entirely of skills a single-skill scanner labeled safe.
AgentTrace is an active framework that identifies the foundation model behind a black-box LLM agent from observable execution traces. It injects structurally ambiguous probes that trigger tokenization-sensitive serialization failures at rigid tool interfaces, then models the sequence of repair operations with lightweight Markov chains. The method reaches 96.0% attribution accuracy under adversarial identity spoofing and separates intra-family variants that share tokenizers, such as Llama-3 8B and 70B.
Researchers show that in edge-cloud split inference of multimodal LLMs, an honest-but-curious server can infer age, gender and race from the intermediate features it receives, even when raw images stay local. They propose a label-budgeted recovery curve to measure feature-level privacy, finding that Safe-LLaVA, which filters model responses, still lets a server recover 75-85% of unprotected leakage from transmitted features with 20 labeled samples per class.
Researchers measured how hints attributed to a professor, a friend, or a dog shift answers from three open-source model families (Qwen, Gemma, SmolLM) on MMLU questions. They found the hint produces a roughly constant mean shift in the hinted answer's log-odds, largely independent of source, and the shift persists even when the source is declared inaccurate or explicitly distrusted.
SynMark is a multi-bit watermarking method for LLM-generated code that replaces surface-token routing with learned semantic coordinates, so identifier renaming does not desynchronize message routing. It uses a syntax-aware adaptive bias during generation and reliability-weighted aggregation during extraction. Under 100% identifier renaming it retains 82.27% BitAcc on CodeLlama and 83.77% on DeepSeek-Coder, and across three code LMs it improves clean BitAcc and Pass@1 by up to 4.53 and 11.18 percentage points over the strongest watermarked baselines.
This paper evaluates a multi-layered defense against prompt injection, data poisoning, adversarial input manipulation and supply-chain tampering in generative-AI and AIOps systems. Using 172 attack instances, equal across threat classes, it compares an unsecured baseline, an input-gateway configuration and a full defense combining data screening, provenance checks, adversarial validation, policy enforcement, monitoring and incident response. The authors report that isolated controls miss attacks exploiting trusted context, polluted training material or operational automation, while the layered approach reduces attack success and improves detection, containment and recovery.
Fix: The paper recommends a layered security pattern combining data screening, provenance checks, adversarial validation, policy enforcement, continuous monitoring and incident response, with governance, auditability and controlled automation for generative-AI and AIOps components deployed in production.
OpenAlex (peer-reviewed AI security)This paper introduces a paired response-final divergence framework that measures jailbreak-induced representational displacement between same-goal direct and jailbreak responses, using hidden states at the final generated token. Applied to 15 open-source LLMs across four alignment strategies and four attack types, it finds that Difference-in-Means refusal direction suppression shows the largest cross-model association with jailbreak amplification. Divergence–ΔASR correlation is attack-type-specific, strongest for authority framing (r = 0.836) and context manipulation (r = 0.712), and weak for roleplay attacks.
This paper shows that approximate unlearning in LLMs can leave residual traces in pre- and post-unlearning model snapshots, which can be used to recover the supposedly removed training data. It proposes TRACE, which reconstructs unlearned text from the two snapshots without prior knowledge of the data, using embedding-level parameter differences and a perplexity-gap contrastive score. Experiments on TOFU, MUSE, and WMDP across multiple LLMs and six unlearning methods report high-fidelity reconstruction.