Deep Model Fusion: A Survey

Summary

This survey reviews deep model fusion, which merges parameters or predictions from multiple deep learning models, including large language models and foundation models, to offset individual model biases and errors. It groups existing methods into four categories: weight averaging, mode connectivity, alignment, and ensemble learning. The authors also analyze open challenges, such as high computational cost and interference between heterogeneous models, and outline future research directions.