A data-driven framework for the stochastic reconstruction of small-scale features with application to climate data sets

https://doi.org/10.1016/j.jcp.2021.110484Get rights and content

Highlights

  • Stochastic data-driven reconstruction of small-scale features in turbulent flows.

  • The machine learning framework is based on conditionally Gaussian statistics.

  • Small scales are naturally split into a deterministic and stochastic component.

  • Demonstration with reanalysis data for vorticity over Europe.

Abstract

Turbulent fluid flows in atmospheric and oceanic sciences are characterized by strongly transient features with spatial inhomogeneity, spanning a wide range of spatial and temporal scales. While large-scale dynamics are often well approximated by closure schemes there is still a need to efficiently represent the corresponding small-scale features, when it comes to the risk analysis for extreme events. We introduce a data-driven framework for the stochastic reconstruction of the small spatial scales in terms of the large ones. The framework employs a spherical wavelet decomposition to partition field quantities, obtained from reanalysis data into non-overlapping spectral components. Using these time-series we formulate, for each spatial location, a machine-learning scheme that naturally ‘splits’ the small-scales into a predictable part, which can be effectively parametrized in terms of the large-scales time-series, and a stochastic residual, which cannot be uniquely determined using the large-scale information. The later is represented using a conditionally Gaussian process, a choice that allows us to overcome the need for a vast amount of training data, which for climate applications, is naturally limited to a single realization for each spatial location. Using a second round of machine-learning we parametrize, for each location, the covariance of the stochastic component in terms of the large scales. We employ the machine-learned statistics to parsimoniously reconstruct random realizations of the small scales. We demonstrate the approach on reanalysis data involving vorticity over Western Europe and we show that the reconstructed random samples for the small scales result in excellent agreement to the spatial spectrum, single-point probability density functions, and temporal spectral content.

Introduction

Large-scale environmental flows are characterized by a wide range of spatial scales with strong dynamical coupling and important spatial inhomogeneities. While in several important problems in geophysical fluid dynamics one can observe or estimate accurately the large-scale behavior, the reconstruction of high-resolution features given the large-scale information remains a challenge. Classical examples in ocean models are the Gent-McWilliams parameterization of the effects of ocean mesoscale features on heat and salt transport, [14], and the parameterizations of the effects of buoyancy induced convection on deepwater formation, [45]. Several other efforts have focused on the description of small-scale characteristics in terms of large scales in the context of geophysical turbulence. In [17], [16], [18] a stochastic superparametrization framework was developed where the effect of the small scales was parameterized in terms of large-scale quantities using linear stochastic differential equations excited by white noise. The coefficients of the linear stochastic differential equations were tuned based on high-resolution simulations. To account for spatial inhomogeneities a decomposition of the flow field into a large scale component and a time-uncorrelated component is adopted in [32], [33], [34], leading to a random version of the Reynolds transport theorem. In this series of papers the emphasis was primarily given on deriving a closed set of equations governing the large scales, rather than a detailed representation of the small scales.

In engineering turbulence several efforts have also been focused on representing the dependence of the small scale features on the larger scales, [10], [28], [48], [11]. In particular, many works have considered the problem of reconstructing the flow field using sparse measurements or large-scale features of the flow. While these approaches have shown promise for low-Re laminar flows, they are less effective for high-Re complex flows, [9]. This is not surprising given the strongly turbulent, non-stationary, multi-scale character of these flows, which inevitably induces fundamental barriers on how much energy present in the small scales can be parameterized from the large scales in a deterministic way. Specifically, the nonlinear interactions between small and large scales are typically characterized by strong instabilities which, in turn, lead to loss of predictability of the small scales, [46], [4], [31]. Therefore, independently of what parametrization method is employed, one has to compromise with the fact that any deterministic parametrization of the small scales will be limited by the loss of predictability.

The scope of this work is the formulation of a stochastic representation framework that will be able to generate realizations of the small scale features conditioned on a given realization for the large scale dynamics. This is particularly important for the quantification of risk for certain events that can take place in small spatial domains, i.e. smaller than the large scale dynamics. This topic, known as statistical downscaling, has been the focus of numerous studies over the last decades. These include ideas based on linear regression and analog techniques [19], [26]. For example in the context of statistical downscaling for precipitation, an estimator is obtained using an optimized linear combination of the local circulation features [20], [23]. Beyond these linear schemes, there is a number of machine learning techniques that have been utilized for statistical downscaling. These efforts began with artificial neural networks [47], [37] and subsequently relied on alternative machine learning approaches, such as support vector machines [41], random forests [22], [30], nearest neighbor [12] or genetic programming [35]. More recently, deep learning ideas have also been employed for statistical downsampling. These include cases aiming for recovering high-resolution fields from low-resolution inputs with a generalized stacked super resolution convolutional neural network [44], autoencoder architectures [43], long short-term memory networks [29] and ideas based on blended architectures [27]. A comprehensive assessment of different deep learning techniques for statistical downscaling for precipitation and temperature is presented in [2].

While deep learning ideas have proven to be successful in several cases, they are not always able to adequately parametrize all the energy of the small scale features. In particular, when the downscaling goes to sufficiently fine scales, there is typically an energy component that behaves stochastically, i.e. it cannot be parametrized in a deterministic manner from the low-resolution input. This requires the adoption of a stochastic representation for the downscaling. Recent efforts have focused on this issue using adversarial deep learning [39], a promising approach for representing stochastic phenomena by machine learning the full probability density function. However, adversarial deep learning requires a vast amount of training data, which inevitably leads to the assumption of spatial homogeneity, i.e. machine learn a single neural network using as training data the information over different spatial regions. The obstacle in this case is the spatially-inhomogeneous character of climate dynamics which does not allow us to utilize data from one region in order to train a data-driven parametrization scheme for another region. Therefore, we have to rely for training data on a single realization (reanalysis data) for each spatial location.

To overcome these limitations, we introduce a new framework, the Stochastic Machine-Learning (SMaL) parametrization. The SMaL parameterization framework consists of a spatial wavelet decomposition at different scales, a machine-learning parameterization of the predictable part of the small scales, as well as a machine-learning parametrization of the stochastic part of the small scales. The latter is represented using a conditionally Gaussian statistical structure in order to minimize the need for training data. The assumption of Gaussian statistics for the small scale fluctuations conditioned on the large scales does not imply Gaussian statistics for the fluctuations. Such conditional Gaussian models have been developed and analyzed previously in the context of filtering and data-assimilation for special models, [6], [7], [5], [24]. Here we focus on machine-learning the mean and covariance of the conditionally Gaussian models for the small scales in terms of the large scale features. This is a non-trivial task as we have to rely on a single realization (reanalysis data) in order to machine-learn statistical quantities such as the conditional covariance, while we also have to respect the spatially inhomogeneous character of the climate dynamics, i.e. we cannot use data from one spatial location to train in another location.

First we employ a wavelet decomposition to partition the training flow fields into large and small-scale features. The choice of a localized representation is a critical aspect, as it allows us to exploit the local character of the interactions between scales. In this way, it leads to more efficient parametrization of the dynamics governing the small scales and better predictability skill. Next, we train a temporal convolution network (TCN), which has as input the large-scale information and provides as output the small scales. Because of the turbulent character of the dynamics, only a portion of the overall energy contained in the small scales can be parametrized and reproduced correctly by this TCN. In this way, this first stage of machine-learning naturally “splits” the small scales into i) a predictable component, which can be effectively parametrized by the large-scale features through the TCN scheme, and ii) a stochastic remainder, which cannot be parametrized by the data-driven scheme. The challenge now is how to compute the statistical characteristics, e.g. variance, of this stochastic remainder since we have only one realization available. To overcome this obstacle we introduce a local-in-time-averaging operator, as well as a method to select the averaging window, which provides the local-in-time variance of the stochastic residual. Using a second round of data-driven modeling with a TCN we parametrize the local-in-time variance of the fluctuations in terms of the large-scale features. As a final step for the training phase, we estimate the temporal correlation matrix of the stochastic fluctuations using the correlation coefficient matrix for the small scales, estimated from the full time series of the training data, to obtain a complete estimate of the statistics of the small-scale behavior. This is done taking into account the spatial inhomogeneous character of the dynamics.

The last component of the framework involves the reconstruction of random realizations, consistent with the large scales. This is done by superimposing the predictable part of the small scales (obtained from the first TCN) with random realizations of the stochastic part. The random realizations are generated by utilizing the predicted temporal statistics (obtained from the second TCN) and the assumption of conditionally Gaussian statistics. We demonstrate the SMaL parameterization framework on reanalysis data for atmospheric quantities of interest such as vorticity over Western Europe.

Section snippets

Problem formulation

The goal of this work is the development of a data-driven downscaling framework where small-scale features will be machine-learned in terms of the large-scale characteristics. In this section we provide an overview of our approach, SMaL, through a reanalysis data set involving atmospheric flows.

Temporal convolution networks

Before we describe details of the SMaL parametrization we introduce the machine-learning architecture at the heart of our work - the temporal convolution networks (TCN). This architecture has been recently shown to outperform its popular alternative, the recurrent neural networks (RNNs), in speech and language related tasks, [42], [1]. For the modeling problems considered in this work, we have observed from numerical experiments that the performance is indistinguishable between a TCN and a

Training of SMaL parametrization

With the model variables and the scale-differentiating representation (i.e. the wavelet functions), we can proceed with the detailed description of the SMaL parametrization framework. We model the small-scale wavelet coefficients as a non-stationary Gaussian process conditioned on the large-scale wavelet coefficients. This process has as mean the predictable part obtained from a deterministic machine-learning scheme that takes as input the large-scale wavelet coefficients. The covariance matrix

Samples-generation and validation

The next step involves drawing samples according to the modeled mean and cross-covariance matrix function. In particular, based on the assumption of Gaussian statistics for the stochastic part, we will obtain realizations for the small-scale coefficients that are conditioned on their past, as well as the obtained statistics from the large scales.

Conclusions

We have introduced a data-driven strategy, the Stochastic Machine-Learning (SMaL) framework, for explicitly parameterizing small-scale atmospheric features in terms of the corresponding large-scale observations, taking into account the spatially inhomogeneous character of the problem and the fact that there is only a single realization available for each spatial location that can be used for training. Our framework represents the large- and small-scale features using a versatile wavelet basis

CRediT authorship contribution statement

Zhong Yi Wan: Investigation, Methodology, Software, Writing – original draft. Boyko Dodov: Conceptualization, Data curation, Investigation, Methodology, Writing – review & editing. Christian Lessig: Data curation, Investigation, Methodology, Software, Writing – review & editing. Henk Dijkstra: Investigation, Methodology, Writing – review & editing. Themistoklis P. Sapsis: Conceptualization, Funding acquisition, Investigation, Methodology, Software, Supervision, Writing – original draft.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The authors appreciate several stimulating discussions with Dr. Zoltan Toth, NOAA. Funding by AIR Worldwide and Verisk Analytics is gratefully acknowledged.

References (48)

  • I. Grooms et al.

    Stochastic superparameterization in a quasigeostrophic model of the Antarctic circumpolar current

    Ocean Model.

    (2015)
  • M. Milano et al.

    Neural network modeling for near wall turbulent flow

    J. Comput. Phys.

    (2002)
  • S. Bai et al.

    An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling

    (2018)
  • J. Baño-Medina et al.

    Configuration and intercomparison of deep learning neural models for statistical downscaling

    Geosci. Model Dev.

    (2020)
  • P. Berrisford et al.

    The ERA-Interim archive Version 2.0

    (2011)
  • G.J. Boer

    Predictability as a function of scale

    Atmos.-Ocean

    (2003)
  • N. Chen, Y. Li, Efficient nonlinear filtering, smoothing, forward and backward sampling algorithms for partially...
  • N. Chen et al.

    Filtering nonlinear turbulent dynamical systems through conditional Gaussian statistics

    Mon. Weather Rev.

    (2016)
  • N. Chen et al.

    Conditional Gaussian systems for multiscale nonlinear stochastic systems: prediction, state estimation and uncertainty quantification

    Entropy

    (2018)
  • C.C. Da Silva et al.

    A local spectral exterior calculus for the sphere and application to the shallow water equation

  • N.B. Erichson et al.

    Shallow learning for fluid flow reconstruction with limited sensors and limited data

    (Feb. 2020)
  • R. Everson et al.

    Karhunen–Loève procedure for gappy data

    J. Opt. Soc. Am. A

    (Aug. 1995)
  • K. Fukami et al.

    Super-resolution reconstruction of turbulent flows with machine learning

    J. Fluid Mech.

    (Jul. 2019)
  • S. Gangopadhyay et al.

    Statistical downscaling using K-nearest neighbors

    Water Resour. Res.

    (2005)
  • W.A. Gardner et al.

    Cyclostationarity: half a century of research

    (2006)
  • P.R. Gent et al.

    Isopycnal mixing in ocean circulation models

    J. Phys. Oceanogr.

    (1990)
  • I. Goodfellow et al.

    Deep Learning

    (2016)
  • I. Grooms et al.

    Efficient stochastic superparameterization for geophysical turbulence

    Proc. Natl. Acad. Sci. USA

    (2013)
  • I. Grooms et al.

    Stochastic superparameterization in a one-dimensional model for wave turbulence

    Commun. Math. Sci.

    (2014)
  • J.M. Gutiérrez et al.

    An intercomparison of a large ensemble of statistical downscaling methods over Europe: results from the VALUE perfect predictor cross-validation experiment

    Int. J. Climatol.

    (2019)
  • A. Hannachi et al.

    Empirical Orthogonal Functions and Related Techniques in Atmospheric Science: A Review

    (2007)
  • K. He et al.

    Deep residual learning for image recognition

  • C. Hutengs et al.

    Downscaling land surface temperatures at regional scales with random forest regression

    Remote Sens. Environ.

    (2016)
  • D.I. Jeong et al.

    Multisite statistical downscaling model for daily precipitation combined by multivariate multiple linear regression and stochastic weather generator

    Clim. Change

    (2012)
  • Cited by (7)

    • Multi-path deep learning framework on discrete pressure points to predict velocity field of pump-jet propulsor

      2022, Applied Ocean Research
      Citation Excerpt :

      It can provide a new method for predicting the performance of PJP. At the current stage, the combination of fluid mechanics and deep learning methods are mainly the solution of turbulence model parameters (Ling and Templeton, 2015; Parish and Duraisamy, 2016; Gamahara and Hattori, 2017; McConkey et al. 2021 etc.), the super-resolution reconstruction of the flow field (Fukami et al. 2019, Wan et al. 2021, etc.), and the flow field prediction of simple models, such as airfoils (Zhu et al., 2019; Sekar et al., 2019; Peng et al., 2020, etc.) and cylinders (Lee and You, (2019); Li et al., 2020, etc.). There are few studies on the prediction of the flow field with complex structure.

    • Symbiotic Ocean Modeling Using Physics-Controlled Echo State Networks

      2023, Journal of Advances in Modeling Earth Systems
    View all citing articles on Scopus
    View full text