Beyond Single-Pair Attacks: Disrupting Vision-Language Pre-Training Models With Dual-Semantic Frequency Stealth

Summary

Researchers propose DSFG-Attack, a transferable adversarial attack against Vision-Language Pre-training (VLP) models that disrupts cross-modal alignment by injecting non-aligned semantics into image-text pairs and restricting perturbations to high-frequency domains for stealth. On Flickr30K image-text retrieval, with TCL as the source model and CLIP_ViT as the target, it raises the black-box attack success rate by an average of 5.08%. The adversarial examples also transfer to captioning and to Multimodal Large Language Models such as GPT-4o and Qwen 2.

Topics

• Adversarial machine learning

Related items

• InfoAnchored Rank-Adversarial Fine-Tuning Pretrained Visual Place Recognition ModelsSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoSay the Same, Act Differently: Diagnosing and Steering Text-Action Divergence in Reasoning VLAsSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoDetect, Repair and Audit: A Hierarchical Multi-Agent Defense Against Adversarial Attacks in Energy ForecastingSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoSANAI: Secure ECC Distributed Keyed Parity Embedded in Weights for Protecting Neural Networks against Bit-Flip AttacksSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoTowards Fine-grained and Robust Detection of Human-AI Co-authorshipSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)