chevron-down--small
Simons Foundation chevron-down--small
Mathematics & Physical Sciences

NSF-Simons Research Collaborations on the Mathematical and Scientific Foundations of Deep Learning

Policies and
Procedures
Contact Info

The NSF-Simons Research Collaborations on the MoDL initiative awarded two collaborations designed to support research activities focused on a particular set of topics involving some of the most challenging questions in the general area of mathematical and scientific foundations of deep learning.

Collaboration on the Theoretical Foundations of Deep Learning
Peter Bartlett — Director, University of California, Berkeley
      Bin Yu — Co-Investigator, University of California, Berkeley
Emmanuel Abbé — PI, Ecole polytechnique fédérale de Lausanne
Mikhail Belkin — PI, University of California, San Diego
Amit Daniely — PI, Hebrew University
Andrea Montanari — PI, Stanford University
Alexander Rakhlin — PI, Massachusetts Institute of Technology
      Elchanan Mossel — Co-Investigator, Massachusetts Institute of Technology
      Nike Sun — Co-Investigator, Massachusetts Institute of Technology
Roman Vershynin — PI, University of California, Irvine
Nathan Srebro — PI, Toyota Technological Institute at Chicago

The success of deep learning has had a major impact across industry, commerce, science and society. But there are many aspects of this technology that are very different from classical methodology and that are poorly understood. Gaining a theoretical understanding will be crucial for overcoming its drawbacks. The Collaboration on the Theoretical Foundations of Deep Learning aims to address these challenges: understanding the mathematical mechanisms that underpin the practical success of deep learning, using this understanding to elucidate the limitations of current methods and extending them beyond the domains where they are currently applicable, and initiating the study of the array of mathematical problems that emerge. The team has planned a range of mechanisms to facilitate collaboration, including teleconference and in-person research meetings, a centrally organized postdoc program and a program for visits between institutions by postdocs and graduate students. The collaboration will have broad impacts through its education, human resource development and broadening participation programs, in particular through training a diverse cohort of graduate students and postdocs using an approach that emphasizes strong mentorship, flexibility and breadth of collaboration opportunities; through an annual summer school that will deliver curriculum in the theoretical foundations of deep learning to a diverse group of graduate students, postdocs and junior faculty; and through targeting broader participation in the collaboration’s research workshops and summer schools.

The collaboration’s research agenda is built on the following hypotheses: that over-parametrization allows efficient optimization; that interpolation with implicit regularization enables generalization; and that depth confers representational richness through compositionality. It aims to formulate and rigorously study these hypotheses as general mathematical phenomena, with the objective of understanding deep learning, extending its applicability and developing new methods. Beyond enabling the development of improved deep learning methods based on principled design techniques, understanding the mathematical mechanisms that underlie the success of deep learning will also have repercussions on statistics and mathematics, including a new point of view of classical statistical methods, such as reproducing kernel Hilbert spaces and decision forests, and new research directions in nonlinear matrix theory and in understanding random landscapes. In addition, the research workshops that the collaboration will organize will be open to the public and will serve the broader research community in addressing these key challenges.

Collaborative Research: Transferable, Hierarchical, Expressive, Optimal, Robust, Interpretable NETworks (THEORINET)
René Vidal — PI/Director, Johns Hopkins University
      Mauro Maggioni — Co-Investigator, Johns Hopkins University
      Joshua Vogelstein — Co-Investigator, Johns Hopkins University
      Soledad Villar — Co-Investigator, Johns Hopkins University
Gitta Kutyniok — PI, Ludwig-Maximilians-Universität München
Guillermo Sapiro — PI, Duke University
      Ingrid Daubechies — Co-Investigator, Duke University
      Rong Ge — Co-Investigator, Duke University
Alejandro Ribeiro — PI, University of Pennsylvania
      Edgar Dobriban — Co-Investigator, University of Pennsylvania
      Robert Ghrist — Co-Investigator, University of Pennsylvania
      George Pappas — Co-Investigator, University of Pennsylvania
Yi Ma — PI, University of California, Berkeley
      S. Shankar Sastry — Co-Investigator, University of California, Berkeley
      Jacob Steinhardt — Co-Investigator, University of California, Berkeley
Emmanuel Candès — PI, Stanford University

Recent advances in deep learning have led to many disruptive technologies: from automatic speech recognition systems to automated supermarkets, to self-driving cars. However, the complex and large-scale nature of deep networks makes them hard to analyze and, therefore, they are mostly used as black boxes without formal guarantees on their performance. Moreover, the design of deep networks remains an art and is largely driven by empirical performance on a data set. As deep learning systems are increasingly employed in our daily lives, it becomes critical to understand if their predictions satisfy certain desired properties. The goal of this collaboration is to develop a mathematical, statistical and computational framework that helps explain the success of current network architectures, understand its pitfalls and guide the design of novel architectures with guaranteed confidence, robustness, interpretability, optimality and transferability. This project will train a diverse STEM workforce with data science skills that are essential for the global competitiveness of the U.S. economy by creating new undergraduate and graduate programs in the foundations of data science and organizing a series of collaborative research events, including semester research programs and summer schools on the foundations of deep learning. This project will also impact women and underrepresented minorities by involving undergraduates in the foundations of data science.

Deep networks have led to dramatic improvements in the performance of pattern recognition systems. However, the mathematical reasons for this success remain elusive. This project brings together a multidisciplinary team of mathematicians, statisticians, theoretical computer scientists and electrical engineers to develop the mathematical and scientific foundations of deep learning. The project is divided into four main thrusts. The analysis thrust will use principles from approximation theory, information theory, statistical inference and robust control to analyze properties of deep networks such as expressivity, interpretability, confidence, fairness and robustness. The learning thrust will use principles from dynamical systems, non-convex and stochastic optimization, statistical learning theory, adaptive control, and high-dimensional statistics to design and analyze learning algorithms with guaranteed convergence, optimality and generalization properties. The design thrust will use principles from algebra, geometry, topology, graph theory and optimization to design and learn network architectures that capture algebraic, geometric and graph structures in both the data and the task. The transferability thrust will use principles from multiscale analysis and modeling, reinforcement learning and Markov decision processes to design and study data representations that are suitable for learning from and transferring to multiple tasks.

Read More
Contact Info

Letters of Intent and Full Proposals must be submitted via the NSF as outlined in the program solicitation. Awards will be made jointly with the Simons Foundation. All application materials should be submitted to NSF. NSF will share all submitted materials with the Simons Foundation.

Half of the proposal budget must be prepared following Simons Foundation’s grant policies, which includes the foundation’s indirect costs policy (20 percent). For further guidance, please see the foundation’s grant award policies on our website.

Applicants will be required to submit a detailed, five-year budget and justification for the Simons Foundation portion of the award. For budgets with subcontracts, a subcontract budget and budget justification for each subcontract must be included. These budgets must be submitted using the Simons Foundation budget template.

Recommended for Funding proposals will be resubmitted by the PIs to the Simons Foundation, in accordance with instructions given by the Simons Foundation Program Officer.

Contact Info

Heitsch, Christine — PI/Director
Lu, Hang — Co-I/Associate Director
NSF-Simons Southeast Center for Mathematics and Biology
Georgia Institute of Technology

This award will establish the NSF-Simons Southeast Center for Mathematics and Biology, a regional center with national impact whose defining mission is connecting mathematical theory with biosystems data. The core center activity is catalyzing new research collaborations at the math-bio interface with a collective focus on understanding emergent properties at critical genome-to-phenome junctures. This is complemented by a strong interdisciplinary training component, with an emphasis on enabling trainees to initiate new, transdisciplinary collaborations as they progress in their research careers, supplemented by educational outreach and convening research activities. In these ways, SCMB is addressing the challenge of building research capacity at the math-bio interface by driving discoveries in mathematics and biology that propel both fields forward into new territory.

The center recognizes that diverse sources of data require diverse theoretical approaches and that novel data structures acquired from biological systems spur mathematical growth. Moreover, this center has identified its fundamental challenge as developing math-bio research collaborations that advance the frontiers of both disciplines. By actively matching experimentalists and mathematicians, the center is catalyzing new math-bio collaborations who will mentor interdisciplinary trainees with an emphasis on building interactional expertise. Educational outreach will build workforce capacity by priming the math-bio pipeline and convening activities will establish a national center of gravity in the southeast for the mathematics of complex biosystems.

 

Murray, Andrew — PI/Chair and Center Director
NSF-Simons Center for Mathematical and Statistical Analysis of Biology
Harvard University

The NSF-Simons Center for Mathematical and Statistical Analysis of Biology at Harvard University aims to use mathematics and statistics to understand biology well enough for scientists to make accurate predictions of how organisms, from simple bacteria and yeasts to mammals, respond to changes in their genes and their environments. The center’s research will focus on three areas: how cells and organisms make decisions that change their behavior and their fate; how the sophisticated structures, from outer coating of viruses and the machinery that segregates a cell’s chromosomes to the structure of embryos and organs, can assemble without blueprints or explicit instructions; and how organisms change their structures and activities to deal with changes in their environments over timescales that range from minutes to millennia. In each of these areas, the center will bring together mathematicians and biologists who want to work together to reach a deep understanding of biology and mathematics, with the hope that the mathematicians will help to create deeper and more general understanding of biological problems, and the biologists will stimulate the mathematicians into inventing new tools in mathematics, statistics and computation that can be applied to problems outside as well as inside biology. The hope is that these close interactions will produce a new generation of scientists who see themselves as being mathematicians and biologists equally and thus will be equipped to tackle a variety of problems in both subjects that cannot yet be solved.

 

Nie, Qing – PI/Director
NSF-Simons Center for Multiscale Cell Fate Research
University of California, Irvine

A new NSF-Simons Center for Multiscale Cell Fate Research will be established at the University of California, Irvine (UCI), to provide a stimulating and empowering intellectual and physical environment for innovative team research at the interface between mathematics and biology. A team of mathematical scientists and biologists at UCI will develop novel mathematical, computational and statistical tools to analyze cell fate through a multiscale lens. The center will carry out a coherent program for community building, interdisciplinary training and workforce development, and diversity enhancement to expand the mathematics-biology interface and to promote the convergence of mathematical and biological sciences. The center will produce cohorts of interconnected young researchers and will nationally seed the next generation’s laboratories. The knowledge gained will create new, multiscale mathematics for analyzing big data and modeling complex systems, with broader applications to regenerative medicine, embryonic development and birth defects.

The NSF-Simons Center for Multiscale Cell Fate Research at UCI will establish new understanding of mechanisms and principles of cell fate control through investigation of emerging behavior of cells across scales. Three concerted efforts will be made to enhance overall capacity of research and training at the interface between mathematics and biology: 1) expanding mathematical sciences proximal to the biology interface, 2) connecting mathematical scientists across the spectrum and 3) fostering mathematical scientists’ ability to connect directly to experiments.

 

Carthew, Richard W. — PI/Center Director
NSF-Simons Center for Quantitative Biology
Northwestern University

This NSF-Simons Center for Quantitative Biology at Northwestern University will catalyze quantitative approaches at the intersection of the mathematical sciences and developmental biology. These will further our understanding of the biological mechanisms and principles associated with growth and development. The center’s focus is on four interrelated research project areas selected to elucidate different aspects of growth and development. Each project integrates common and shared hypothesis-driven mathematical modeling, data-driven mathematical analysis, as well as advanced imaging, genomics and metabolomics tools to explore biological processes across time, space and extrinsic variables. At the core of the mathematical approach is an emphasis on the dynamics associated with the developmental program, how this sequence of events maintains robustness in the presence of noise, and an intimate connection between mathematical modeling and analysis and state-of-the-art experimental investigations. The center will engage students and investigators across this multidisciplinary spectrum. Graduate, undergraduate and postdoctoral trainees will be embedded in shared mathematical and biological environments, intimately learning how other groups work, think and communicate. The center will build interdisciplinary capacity through a visiting scholars program, a named Fellows program, a pilot project program, a middle-school-science-club learning module, workshops and training courses. It will advance knowledge in biology and the mathematical sciences, generate new conceptual models explaining development and growth, and develop new investigators that are experienced in both fields.

The center will transform our understanding of organismal growth and development through quantitative approaches uniquely combining three fundamental mathematical areas: dynamical systems theory, stochastic processes and dimension reduction. New statistical methodologies and methods for combining multiple types of mathematical approaches will be tested and validated. New biological datasets, conceptual models and mathematical models that will transform our current understanding of growth and development will be made available to the broader research. Furthermore, it will promote interdisciplinary education and workforce training at the intersection of mathematical sciences and biology, scientific outreach to the underserved K-12 education community, multiple strategies to foster recruitment of mathematical scientists into biology, and interdisciplinary training for center trainees and visiting scholars.

Subscribe to MPS announcements and other foundation updates