Characterizing Recoverable Expert-Utilization Signatures in Fine-Tuned Mixture-of-Experts LLMs

Summary

Researchers report structural memorization in fine-tuned Mixture-of-Experts LLMs, where a model retains the internal expert routing pathway used for its training data as well as the data itself. They show these routing pathways can be recovered through unconditional generation using only routing and output logits, without knowledge of the fine-tuning data or procedure. The authors also propose composite metrics combining reference model tail likelihood with complexity-aware normalization to improve detection of memorized routing structures, which traditional membership inference metrics miss.

Related items

• InfoFrom Training Exposure to Membership Evidence: A Controlled Study of OLMo 2Similar attack · Arxiv (cs.CR + cs.CL + cs.LG)
• InfoBeyond public access in LLM pre-training dataSimilar attack · OpenAlex (peer-reviewed AI security)
• InfoDifficulty Calibration Across Time for Membership Inference in Diffusion ModelsSimilar attack · Arxiv (cs.RO + cs.CV security)
• InfoDiscrete Unintentional Adversarial Perturbation Detection for DQN Traffic Signal ControlSimilar attack · OpenAlex (peer-reviewed AI security)
• InfoTrustworthy Distributed Machine Learning in Cloud-Edge-IoT Systems: A Heterogeneity-Aware Survey of Security, Privacy, and RobustnessSimilar attack · Arxiv (cs.CR + cs.CL + cs.LG)