Mathematics of Deep Learning Seminar: Julien Mairal

Title: Trainable Algorithms for Inverse Imaging Problems

Abstract: Classical inverse imaging problems are often formulated as the minimization of a cost function consisting of finding a signal that fits data observations, and that is compatible with a priori knowledge about the solution. This approach requires both a good physical model of the data acquisition process and a good prior. On the other hand, when supervised data are available (e.g., pairs of corrupted/clean signals), it is also classical to use data-driven machine learning approaches, often a deep neural networks, which are often seen as black boxes models that are huge-dimensional and hard to interpret. In this talk, we present a hybrid strategy, which we call “trainable algorithms”, that retains the interpretability of classical inverse problem formulations, while allowing us to train model parameters end to end.

Our first example is the problem of super-resolution from a burst of raw low-resolution (LR) images acquired by a prosumer or smartphone camera. The goal is to exploit image misalignments and aliasing artefacts (which contain useful high-frequency information) in order to increase the number of available samples from the underlying high-resolution (HR) signal. The problem is difficult as it requires (i) accurately aligning images with subpixel accuracy, (ii) dealing with noisy raw data produced by the sensor, and (iii) designing an appropriate image prior. In this work, we demonstrate state-of-the-art results on synthetic benchmarks and on real raw data produced by various digital cameras and smartphones.

Our second example focuses on more traditional restoration tasks from a single image in sparse coding models while also leveraging non-local self-similarity priors, which have been shown to be powerful for image restoration problems. The first interesting conclusion is the ability of our models to perform on par with state-of-the-art convolutional neural networks with orders of magnitude less parameters. The second notable fact is the ability to leverage the model interpretability to improve the efficiency of our models for blind denoising.

In these two examples, the benefits of trainable algorithms were the ability to mix traditional inverse problem formulations with deep learning principles, leading to robust and parameter-efficient trainable architectures

This is a joint work with Bruno Lecouat and Jean Ponce.