Frugality in machine learning: Sparsity, a value for the future?

Sparse vectors and sparse matrices play a transerve role in signal and image processing: they have led to succesful approaches efficiently addressing tasks as diverse as data compression, fast transforms, signal denoising and source separation, or more generally inverse problems. To what extent can the potential of sparsity be also leveraged to achieve more frugal (deep) learning techniques? Through an overview of recent explorations around this theme, I will compare and contrast classical sparse regularization for inverse problems with its natural extensions that aim at learning neural networks with sparse connections. During our journey, I will notably highlight the role of rescaling-invariances of modern deep parameterizations, which come with their curses and blessings.

Frugality in machine learning: Sparsity, a value for the future?

Sparse vectors and sparse matrices play a transerve role in signal and image processing: they have led to succesful approaches efficiently addressing tasks as diverse as data compression, fast transforms, signal denoising and source separation, or more generally inverse problems. To what extent can the potential of sparsity be also leveraged to achieve more frugal (deep) learning techniques? Through an overview of recent explorations around this theme, I will compare and contrast classical sparse regularization for inverse problems with its natural extensions that aim at learning neural networks with sparse connections. During our journey, I will notably highlight the role of rescaling-invariances of modern deep parameterizations, which come with their curses and blessings.

‘Conservation Laws for Gradient Flows’

Understanding the geometric properties of gradient descent dynamics is
a key ingredient in deciphering the recent success of very large
machine learning models. A striking observation is that trained
over-parameterized models retain some properties of the optimization
initialization. This “implicit bias” is believed to be responsible for
some favorable properties of the trained models and could explain
their good generalization properties. In this work, we expose the
definitions and properties of “conservation laws”, that define
quantities conserved during gradient flows of a given machine learning
model, such as a ReLU network, with any training data and any loss.
After explaining how to find the maximal number of independent
conservation laws via Lie algebra computations, we provide algorithms
to compute a family of polynomial laws, as well as to compute the
number of (not necessarily polynomial) conservation laws. We obtain
that on a number of architecture there are no more laws than the known
ones, and we identify new laws for certain flows with momentum and/or
non-Euclidean geometries.
Joint work with Sibylle Marcotte and Gabriel Peyré.

https://sfb1481.rwth-aachen.de/symposium24/programme

https://jraf-2023.sciencesconf.org/

http://perso.ens-lyon.fr/nelly.pustelnik/DIPOpt/

https://www.ens-lyon.fr/evenement/recherche/quelle-ia-pour-2030-le-projet-ailys