Greedy Methods for Neural Network and Application to a Semi-Lagrangian Scheme
Please login to view abstract download link
Neural networks are increasingly used for the numerical approximation of solutions to partial differential equations. These techniques are particularly interesting in high-dimensional settings, where they may mitigate the curse of dimensionality. Indeed, from a theoretical perspective, it is known that when the solution belongs to a suitable function space, such as a Barron space, two-layer neural networks can achieve approximation errors of order at least $\mathcal{O}(\sqrt{m})$ where $m$ is the width of the hidden layer, independently of the spatial dimension. However, the effective training of such networks remains challenging, and the theoretical analysis of the convergence of the underlying optimization algorithms is still largely open, mainly due to the non-convex nature of the problem. In this talk, we discuss greedy methods for training neural networks applied to the solution of PDEs. The approximation is constructed iteratively by successively adding simple networks—or even single neurons—trained to correct the residual error. The update is then performed either via an orthogonal projection of the solution onto the span of the chosen networks, or through a step in the direction of the newly added network. One of the main interest of this approach is that it allows the training process to be cast within the well-established framework of greedy algorithms for abstract dictionaries, thereby enabling the use of existing convergence results. Under appropriate assumptions on the regularity of the solution and on the sampling, one can then establish convergence rates of order $\mathcal{O}(\sqrt{m})$, where $m$ is the number of iterations of the greedy algorithm. We apply these method to a semi-lagragian scheme for a transport problem, where at each time-step the solution is approximated by a neural network.
