Deep Model Fusion: A Survey

Deep model fusion is a technique that combines parameters or predictions from multiple deep learning models into one unified system to improve performance by reducing individual model biases and errors. The survey categorizes four main fusion approaches: weight average (averaging model parameters), mode connectivity (connecting models through optimized paths), alignment (matching corresponding units between models), and ensemble learning (combining model outputs during inference). However, applying this technique to large-scale models like LLMs (large language models, which are AI systems trained on massive amounts of text) faces challenges including high computational cost and interference between different types of models.