Spectral Unlearning: Transformer Structure-Preserving Updates for Language Models

Summary

Spectral unlearning is an update framework that combines standard LLM unlearning objectives with matrix-structured optimization, so the update respects the matrix structure of transformer weights rather than treating each scalar coordinate independently. The authors show the spectral update is the steepest-descent direction for transformers built from linear layers with normalized inputs and outputs. Across memorization, privacy and utility, summarized by their harmonic mean, it consistently outperforms a broad range of baselines and closes most of the gap to a retain-trained reference.