From Zipf's Law to Neural Scaling through Heaps' Law and Hilberg's Hypothesis

Summary

This paper derives the neural scaling law, which describes how a foundation model's cross entropy rate changes with training tokens, parameters, and compute, from Zipf's law on token distributions. The authors chain the derivation through Heaps' law on vocabulary growth and Hilberg's hypothesis on entropy scaling. They illustrate the steps with a toy example, the Santa Fe process, which satisfies all four statistical laws.