Neuron Statistics: Notes on the Tensor Programs Master Theorem

TLDR. Tensor programs are a mathematical language for describing computations in wide neural networks. Their master theorem says that, as the width grows, averages over the neurons become predictable: they converge to expectations in a much simpler scalar random process. This turns the analysis of a high-dimensional random network into a tractable probability calculation and provides a rigorous foundation for studying its infinite-width behavior, including Gaussian-process limits Lee et al…