Universality Phenomena in Machine Learning, and Their Applications
Presidential Lectures are free public colloquia centered on four main themes: Biology, Physics, Mathematics and Computer Science, and Neuroscience and Autism Science. These curated, high-level scientific talks feature leading scientists and mathematicians and are intended to foster discourse and drive discovery among the broader NYC-area research community. We invite those interested in the topic to join us for this weekly lecture series.
A canonical task in machine learning is to fit a model (from a certain class) to a dataset. In many settings there is little theoretical understanding of the algorithms used for this task, since they involve nonconvex optimization.
We have empirically observed that in many settings the models fitted to real-life datasets display randomlike properties — the model parameters behave like random numbers for various tests. This is somewhat reminiscent of the ‘universality’ phenomenon in mathematics and physics, whereby matrices in a host of settings turn out to display properties similar to those of the Gaussian ensemble.
In this talk, Sanjeev Arora will describe how these randomlike properties can be used to gain a new understanding in some settings — for example, they can offer insights into linear algebraic properties of word meanings in natural languages, and reversibility properties of fully connected deep nets. In some cases, they can lead us to provably efficient algorithms, such as algorithms for making inferences in a topic model.