Tokens in Disarray: A Unified Adversarial Attack Against Multi-Layered Safeguards in LLMs and VLMs

Summary

Researchers present the Token Disarray Attack (TDA), a unified adversarial method against LLMs and VLMs protected by multi-layered safeguards. TDA uses LLM-generated token permutations of harmful inputs, refined by a genetic algorithm, and reports an average attack success rate above 70%. Against commercial models it reaches up to 53% on individual models, with notable cross-model transferability.

Topics

• Adversarial machine learning

Related items

• InfoImproving Fuzzy Robustness in LLM Safety with Automated Red TeamingSimilar attack · OpenAI Alignment Research Blog
• InfoBait: Mitigating Harmful Agentic Reinforcement Learning via Probing Reward FunctionSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoImage watermarking robustness under MLLM-guided semantic editing: Benchmark, defense, and diagnosisSimilar attack · Elsevier Security Journals
• LowYour Robot's Safety Functions Already Work. What If the Input Lies?Similar attack · VicOne Blog
• InfoSpeech Tokenizers are Vulnerable: Transferable Semantic Attack and Robust TokenizerSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)