Universality Phenomena in Machine Learning, and Their Applications
Presidential Lectures are a series of free public colloquia spotlighting groundbreaking research across four themes: neuroscience and autism science, physics, biology, and mathematics and computer science. These curated, high-level scientific talks feature leading scientists and mathematicians and are designed to foster discussion and drive discovery within the New York City research community. We invite those interested in these topics to join us for this weekly lecture series.
A canonical task in machine learning is to fit a model (from a certain class) to a dataset. In many settings there is little theoretical understanding of the algorithms used for this task, since they involve nonconvex optimization.
We have empirically observed that in many settings the models fitted to real-life datasets display randomlike properties — the model parameters behave like random numbers for various tests. This is somewhat reminiscent of the ‘universality’ phenomenon in mathematics and physics, whereby matrices in a host of settings turn out to display properties similar to those of the Gaussian ensemble.
In this talk, Sanjeev Arora will describe how these randomlike properties can be used to gain a new understanding in some settings — for example, they can offer insights into linear algebraic properties of word meanings in natural languages, and reversibility properties of fully connected deep nets. In some cases, they can lead us to provably efficient algorithms, such as algorithms for making inferences in a topic model.