A Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

Summary

This survey reviews Direct Preference Optimization (DPO), an RL-free alternative to Reinforcement Learning from Human Feedback (RLHF) for aligning LLM policy models with human preferences. It covers theoretical analyses, variants, relevant preference datasets, and applications. The authors categorize recent DPO studies by key research questions and propose future research directions.