LoRASculpt: Harmonious Low-Rank Adaptation for Multimodal Large Language Models

Summary

LoRASculpt+ is a framework for adapting Multimodal Large Language Models to downstream tasks while preserving general knowledge. It uses sparse LoRA updates, an asymmetric adaptation strategy that assigns different sparsity and capacity to the LLM and connector modules, and regularization terms that steer updates away from critical pretrained regions. The authors report that the method improves generalization and downstream performance across tasks, model scales and backbones, even at high sparsity, and mitigates catastrophic forgetting.