Speech Tokenizers are Vulnerable: Transferable Semantic Attack and Robust Tokenizer

Summary

This paper shows that speech tokenizers, the front ends that encode audio for ASR systems and large audio-language models, are fragile at the semantic level. The authors propose T-SemAttack, a transferable attack that uses an ensemble of semantic encoders to disrupt speech content while preserving perceptual quality, and report strong transfer to 9 open-source and black-box ASR systems and 4 LALMs. They also introduce ROSETok, a robust tokenizer built with noisy training and robust semantic distillation.

Mitigation

ROSETok, a robust speech tokenizer that combines noisy training with robust semantic distillation, is proposed to improve reconstruction fidelity and downstream SpeechLLM robustness.

Related items

• InfoImage watermarking robustness under MLLM-guided semantic editing: Benchmark, defense, and diagnosisSimilar attack · Elsevier Security Journals
• LowYour Robot's Safety Functions Already Work. What If the Input Lies?Similar attack · VicOne Blog
• InfoAdversarial Retraining for Machine Learning-Based Man-in-the-Middle Attack Detection in Smart Grid Industrial Control SystemsSimilar attack · OpenAlex (peer-reviewed AI security)
• LowMeasuring Robustness and Efficiency in a Connectome-Constrained Fly Visual System Model: Linking an Adversarial and a Perceptual IllusionSimilar attack · Arxiv (cs.RO + cs.CV security)
• LowMeasuring Robustness and Efficiency in a Connectome-Constrained Fly Visual System Model: Linking an Adversarial and a Perceptual IllusionSimilar attack · Arxiv (cs.RO + cs.CV security)