Differentially Private Zeroth-Order Methods for Scalable Large Language Model Fine-Tuning

Summary

This paper studies differentially private (DP) zeroth-order methods for fine-tuning large language models, which approximate gradients to avoid the scalability bottleneck of DP-SGD. It proposes two methods: DP-ZOSO, which dynamically schedules key hyperparameters, and DP-ZOPO, which applies data-free stagewise pruning to reduce trainable parameters without extra privacy budget. The authors report theoretical analysis and empirical results on encoder-only and decoder-only language models across diverse tasks.