Research
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
Academic papers, new techniques, benchmarks, and theoretical findings in AI/LLM security.
61 items
Physicist proposes a candidate nonperturbative definition of the Standard Model on a lattice, a grid representing spacetime, in an October preprint prepared with the aid of GPT-6 Astra. The author had a strategy but could not prove it worked in all required cases, and Astra helped build the construction and led development of the detailed proof.
Researchers tested four JEPA-based chest X-ray encoders, trained on 78,468 NIH ChestX-ray14 images, under two black-box adversarial attacks (SPSA, Boundary Attack), three image corruptions, and a label-scarcity test. The perturbations caused severe, model-dependent damage, with encoder-level Cosine Similarity dropping as low as 0.33 and AUROC falling by as much as 0.12 for I-JEPA under the Boundary Test. Adversarial fine-tuning, which trained the predictor on clean and perturbed images together, raised embedding consistency above 0.97 in nearly every case and improved AUROC for most models, with the largest gain of +0.10 on I-JEPA.
Fix: Adversarial fine-tuning, which trains the small predictor on clean and perturbed images together, raised embedding consistency above 0.97 in nearly every case and improved AUROC for most models.
Arxiv (cs.RO + cs.CV security)Researchers conducted the first systematic study of privacy-preserving training for multimodal document key information extraction (KIE) models, comparing differential privacy (DP), federated learning (FL), and DP federated learning (DP-FL) across six benchmark datasets. They found that DP training causes substantial performance loss, which can be largely recovered through pretrained document foundation models, DP-suited hyperparameter tuning, and task-specific pretraining, retaining 77–93% of non-private baseline performance at ε ∈ {8, 20}. Non-private KIE models were shown to be highly vulnerable to membership inference attacks, which FedAVG and DP substantially mitigate.
Fix: The source recommends training with differential privacy, federated learning, or DP federated learning, using pretrained document foundation models, DP-suited hyperparameter tuning, and task-specific pretraining aligned to the private data distribution, and it provides practical guidelines for privacy-utility trade-offs.
OpenAlex (peer-reviewed AI security)A systematic literature review categorizes privacy-preserving deep learning research from 2017 to May 2024, distinguishing between inference and training and between single and multiple data owners. The authors report that execution times are steadily decreasing, but defending models against privacy attacks still often reduces utility, such as accuracy.
This paper evaluates image-specific and image-agnostic adversarial attacks on YOLOv3 and YOLOv4 detectors paired with SORT, Deep SORT, Strong SORT, and OC-SORT trackers in the CARLA simulator. Under image-specific attacks, HOTA for YOLOv3-based trackers falls from 53.45-56.19% to 14.85-17.03%, with MOTA reaching -772.60% in the worst case, while YOLOv4-based trackers decline to 25.06-26.86%. A single Universal Adversarial Perturbation can also reduce tracking accuracy without per-frame perturbations.
Researchers tested whether small language model (SLM) agents can reliably write the security artifacts that enforcement defenses such as CaMeL, Progent, IPIGuard and DRIFT depend on. Across 15 open-weight models, the artifacts written by the agent's own model admitted attacks on 14 (Progent), 13 (IPIGuard) and 11 (DRIFT) of the models, and CaMeL's programs failed to run in up to 68% of episodes. The proposed CapLite derives its policy from tool schemas and admits writes only when the deciding arguments appear in the user's request, with no attack succeeding in 13,935 attacked AgentDojo episodes.
This paper estimates the share of a document corpus produced by a watermarked language model, not just whether one document is watermarked, for Gumbel-max and green-list (KGW) watermarks under unknown edits. It derives a power envelope over retention, entropy and in-place substitution budgets, giving finite-sample bounds on the share. On OPT-1.3B and Qwen3-1.7B text, the exact-null bound at a share of 0.2 is about [0.14,0.38], and the bounds are fragile to LLM paraphrasing and to small null-calibration errors at large corpus sizes.
This research proposes a codebook-based watermarking scheme that maps an n-bit payload to a longer u-bit codeword and embeds it during generation through token-level logit biasing. Experiments show near-perfect recovery for 16-24 bit payloads on 200-token generations and strong performance on 127-token generations for base models, though results are weaker on some instruction-tuned models with very short texts. The Levenshtein variant improves resilience to deletions and light paraphrasing, while insertions and stronger rewrites remain challenging.
MobileProbe is a red-teaming framework that tests mobile GUI agents against environmental injection attacks, where malicious instructions embedded in on-screen UI redirect the agent's actions. It discovers attacker-controllable content, localizes vulnerable decision points along the agent's trajectory, and iteratively refines context-aware payloads through a surrogate-filtered evolutionary loop. Across 83 long-horizon attack instances in 12 mobile applications, it reports higher attack success rates than human-written, zero-shot LLM and prior automated baselines on three agent architectures.
OpenAI describes how automated red-teaming was used to train GPT-6 Astra and Sol to resist jailbreaks in high-stakes domains such as cyber, biology and violent activity. An attacker model iteratively refines prompts against defender models GPT-5.4 and GPT-5.5, and the resulting synthetic jailbreaks serve as training data. On a fixed cyber jailbreak dataset, GPT-6 Astra answered 91.5% safely versus 59% for GPT-5.6 Sol, measured without production safeguards.
Researchers propose Less is Safer (LIS), a 2.4K-sample dataset for LLM safety alignment, built by selecting prompts via Sparse Autoencoder (SAE) activations and by multi-agent adversarial synthesis of harmless prompts that still trigger unnecessary refusals. On four safety and five over-refusal benchmarks, LIS outperforms much larger datasets such as TRIDENT-Core (26K), TRIDENT-Edge (19K) and WildJailbreak (262K). Ablations show that safety data alone and over-refusal data alone each fall short of optimal results.
PolicySniff is a side-channel attack that infers an LLM agent's approval-policy setting, such as whether it asks before a consequential tool call, by timing only the attacker's own GPU kernels on a shared GPU. On an eager sparse mixture-of-experts runtime, a frozen and blinded study identified the deployed setting at 94.17% balanced accuracy on its primary estimand, reaching 88.82% where both settings yield the same decision and 91.98% where they differ.
Fix: The source recommends that deployments keeping their approval setting private measure its distinguishability on the serving runtime and enforce authorization outside the model.
Arxiv (cs.CR + cs.CL + cs.LG)Researchers introduce Intent-Aware Security, a framework that treats an AI agent's trusted intent as a security object, defined through Intent Preservation and Intent Drift and operationalized as an Intent Profile. On an 887-sample benchmark across 35 agents, Intent Profile-based protection raises four-class macro-F1 from 45% to 79%, with Out-of-Scope F1 rising from 8% to 76%. On more than 10,000 executed AgentDojo trajectories, profile-based post-execution detection reaches 0.949 AUROC, versus 0.909 for AlignmentCheck.
Researchers study whether private information retained in the fixed-size recurrent state of latent-state LLMs can leak. Using partial-state probing and GPU power observation on released hybrid implementations, they find that a small state observation predicts private facts and response rules, that this information survives public dialogue, and that complete-read power can recover private facts under a fixed execution schedule. An unprivileged account on a partitioned GPU also inferred a user's age from whole-device power during public requests.
Researchers asked whether an attacker can identify the content of images hidden by deep image hiding models without knowing which model produced them. They built a dataset of stego images from 11 hiding models and trained a single decoder, based on HiNet, that extracts secret content across unseen models, with humans and machines able to recognize it. The attack also reveals content hidden by AIRPNet, a privacy-oriented image restoration method.
Researchers ask whether suppressing knowledge during pretraining makes unwanted capabilities in open-weight language models harder to recover through fine-tuning. Gradient ascent interleaved with pretraining on a filtered corpus raised the fine-tuning cost of medical knowledge: at 1B parameters, matching the held-out PubMed loss of an unfiltered model required over 20 times as many PubMed fine-tuning tokens as filtering alone. The authors report a favorable trade-off between fine-tuning difficulty and held-out loss on retained data, and release code at github.com/MEGlr/DETER.