Skip to content
InfoResearchPeer-reviewed

Dataset Distillation Sweeper: Backdoor Defense for Multimodal Learning

Published
Record updated
View JSON

Summary

Researchers present Dataset Distillation Sweeper (DDaS), a defense against backdoor attacks on multimodal pre-trained models that are downloaded from third-party platforms and fine-tuned on downstream datasets. DDaS runs cyclic dataset distillation with the pre-trained model to separate backdoor datasets from clean ones, without invasive changes to the model backbone. The authors report attack success rate reductions of over 97% (Image-to-Text) and 93% (Text-to-Image) in image-text retrieval, and 99% in Visual Question Answering, with accuracy loss under 1% and within 2% respectively.

Mitigation

DDaS itself is the proposed mitigation: it uses the multimodal pre-trained models to perform cyclic dataset distillation to distinguish backdoor datasets from clean datasets and obtain a clean model, without invasive modifications to the model backbone. Code is at https://github.com/grey1989/DDaS.