DMFT, Two Point Correlations of Resolvents, and Applications to Machine Learning Theory

Machine learning algorithms evolve the parameters of a model in a high dimensional and disordered loss landscape. To characterize the effects of random initialization of model parameters, randomly sampled training data, and the effect of SGD noise, it often is useful to invoke ideas from random matrix theory and the physics of disordered systems. In this seminar, I describe a general idea, known as dynamical mean field theory (DMFT) which describes the evolution of a disordered dynamical system in infinite dimensions. I will briefly describe simple examples of interest to theoretical neuroscientists and machine learning theorists. For linear dynamical systems, I will show that this method characterizes the typical case trajectory in terms of two point correlations of resolvent matrices evaluated at different frequencies. This bispectral object can account for puzzling effects such as late time divergence of gradient descent at the interpolation threshold (when parameters = dataset size) despite the Jacobian of the dynamics having real and non-positive eigenvalues. I will then describe a novel two point correlation result for general free products of the form M = O B O^T A for O sampled from the Haar measure. I will use this result to characterize the exact asymptotics of the performance of a linear transformer trained to perform in-context linear regression on “generic” (randomly rotated) covariance matrices.