DETER: Suppression During Pretraining for Tamper Resistance

Summary

Researchers ask whether suppressing knowledge during pretraining makes unwanted capabilities in open-weight language models harder to recover through fine-tuning. Gradient ascent interleaved with pretraining on a filtered corpus raised the fine-tuning cost of medical knowledge: at 1B parameters, matching the held-out PubMed loss of an unfiltered model required over 20 times as many PubMed fine-tuning tokens as filtering alone. The authors report a favorable trade-off between fine-tuning difficulty and held-out loss on retained data, and release code at github.com/MEGlr/DETER.