Theoretical Foundations

Lead: Stuart and Anandkumar

Formulation

The core idea is to cast learning about closure models as (possibly stochastic) inverse problems. This takes us beyond the state-of-the-art that primarily views the problem predominantly through the lens of supervised learning. It enables the integration of disparate data sources, leads naturally to uncertainty quantification and physically interpretable models. The approach also enables solution using gradient-free ensemble methods where model derivatives are not available. Furthermore, the inverse problem perspective, combined with that of data-assimilation, can be used to address missing data and lift semi-supervised learning problems into the realm of supervised learning. The implementation of this core idea enables mixed-precision and mixed-resolution representations.

This idea leads to two forms of inverse problem: direct and indirect inverse problems. The direct type of inverse problem occurs when we directly observe inputs and outputs for the desired parameterized object. The indirect type of inverse problem occurs when the outputs are related to the inputs via a (possibly unknown) mapping. We treat these as a missing data problem.

Innovations in supervised learning and optimization

We represent closure models as neural operators in space and recurrent neural operators in time. This enables us to extend the notion of supervised learning between spaces of probability measures. We systematically exploit known physics and structure.

Parameter calibration remains a challenge in inverse problems. Two competing and powerful methodologies are stochastic gradient descent (SGD) and ensemble Kalman methods (EnK). We are exploring methods to combine EnK methods with SGD for a tractable approach to uncertainty quantification (UQ) for the training of neural networks. We stabilize neural operator training by developing deterministic scale-free initialization methods. We then develop approaches to dynamically train Fourier neural operators starting with a small number of low frequency modes and gradually including more modes as training progresses.

Sources of data and experimental design

Data arises from three primary sources: (i) numerically by solving a lower scale problem for phenomena with clear scale separation, (ii) numerically using pre-existing closure models based on high fidelity simulations and (iii) experimental and observational data. Typically (i) leads to a direct inverse problem while (ii) and (iii) to an indirect inverse problem. We develop methods that can combine both direct and indirect sources of data.

While advances in experimental techniques and computational power will lead to increasing amount of data, data remains sparse in many fields of interest. Further, the distribution from which data should be drawn for the resulting closure to be applicable is unclear. We are developing ensemble Kalman based optimal design to address these issues.

Multi-precision and multi-resolution computation

We take a broad view of multi-precision computation to include: (i) multi-precision arithmetic; (ii) multilevel discretization methods; (iii) multiple ensemble levels. We also consider
interactions between these, such as between (ii) and (iii) in multi-level Monte Carlo. Furthermore, we develop new mixed-precision strategies for neural operators.

Variable resolution is widely used in CS&E, including adaptive finite elements, multi-grid methods and recent work using multilevel discretization. Indeed, one may view closure models as an approach to a mixed resolution strategy. Therefore, we exploit the data-driven closure models as an approach to multi-resolution computation.