A Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

This paper surveys Direct Preference Optimization (DPO), a method for aligning large language models (AI systems trained on massive amounts of text) with human values and preferences without using reinforcement learning (a training approach that rewards desired behaviors). The survey reviews the theoretical foundations, different versions of DPO, available datasets of human preferences, and real-world applications, while also identifying current limitations and suggesting directions for future research.