Mathematics of Deep Learning Seminar: Stephane Mallat

Title: Harmonic Analysis View of Deep Network Concentration

Abstract: What mathematics can explain how deep convolutional network avoid the curse of dimensionality ? This talk will emphasize the impact of convolutional structures, leading to low-dimensional harmonic analysis. Numerical experiments indicate that deep networks concentrate data distributions. Beginning from two layer networks, we will analyze the role of known group transformations and multiscale sparse representations to explain this concentration. To simplify the mathematics, I will introduce highly structured multiscale networks, where spatial filters are not learned, but still achieve ResNet performances. Concentration can then be related to invariances and renormalization group mechanisms, which opens many questions that I will discuss.

This talk is based on a joint work with Florentin Guth and John Zarka: https://arxiv.org/abs/2012.10424