Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods

Summary

This paper analyzes Chain-of-Thought (CoT) prompting from a statistical perspective, using a multi-step latent variable model to show that the CoT estimator approximates a Bayesian estimator that infers the posterior from examples in the prompt. The authors prove the statistical error splits into a prompting error, which decreases exponentially as the number of prompt examples grows, and a pretraining error of the pretrained LLM. They also construct a transformer model class and establish its generalization error under the PAC-Bayes framework.