The effect of calibration data length on the performance of a conceptual hydrological model versus LSTM and GRU: A case study for six basins from the CAMELS dataset

Highlights

• LSTM and GRU approach GR4H at a notable rate, considering their number of parameters.

• GRU models benefit more efficiently from additional calibration data than LSTM.

• Compared to LSTM, the calibration of GRU is more robust in the case of limited data.

Abstract

We systematically explore the effect of calibration data length on the performance of a conceptual hydrological model, GR4H, in comparison to two Artificial Neural Network (ANN) architectures: Long Short-Term Memory Networks (LSTM) and Gated Recurrent Units (GRU), which have just recently been introduced to the field of hydrology. We implemented a case study for six river basins across the contiguous United States, with 25 years of meteorological and discharge data. Nine years were reserved for independent validation; two years were used as a warm-up period, one year for each of the calibration and validation periods, respectively; from the remaining 14 years, we sampled increasing amounts of data for model calibration, and found pronounced differences in model performance. While GR4H required less data to converge, LSTM and GRU caught up at a remarkable rate, considering their number of parameters. Also, LSTM and GRU exhibited the higher calibration instability in comparison to GR4H. These findings confirm the potential of modern deep-learning architectures in rainfall-runoff modelling, but also highlight the noticeable differences between them in regard to the effect of calibration data length.

Introduction

According to Mount et al. (2016), a hydrological model is a functional relationship in which the model output (y) is determined by its structure (f), inputs (x), parameters (p), and the residual error ε between the modelled variable and its true (or observed) value:

$$y=f\left(x,p\right)+\epsilon$$

Following Mount et al. (2016), conceptual hydrological models put a strong emphasis on structure as a result of knowledge-driven model reduction, of how we formalize our concept of a hydrological system. In contrast, data-driven methods trust in the explanatory power of input data itself as they “simultaneously ‘discover’ the model structure and optimize its parameters [and minimize] the role of a priori hypotheses” (Mount et al., 2016). As a consequence, the number of parameters that need to be optimized (or calibrated) is relatively small for conceptual hydrological models (typically in the order of ${10}^0$ – ${10}^2$), and relatively high (in the order of ${10}^3$ – ${10}^5$) for data-driven models, such as Artificial Neural Networks (ANN).

Numerous studies have been devoted to different aspects of hydrological model calibration, such as the choice of the objective function Fowler et al. (2018), or the use of auxiliary information (Nijzink et al., 2018). Yet, the majority of hydrological models are still calibrated against discharge time series, using square error-based objective functions (Kisi et al., 2013; Arsenault et al., 2018), and their validity requires examination on independent data (Refsgaard et al., 2005). To that end, split-sampling techniques have been suggested by various authors as a basis for calibration-validation experiments, as diagnostic tools to assess the reliability of models and the robustness of their parameterization (e.g., Coron et al., 2012; Kisi et al., 2012; Thirel et al., 2015; Motavita et al., 2019).

In the light of limited data availability for calibration and validation, such techniques also provide a means to study the effect of the amount (or length) of calibration data on the reliability of models. A fair number of studies investigated this effect for conceptual hydrological models (e.g., Sorooshian et al., 1983; Yapo et al., 1996; Boughton, 2007; Perrin et al., 2007; Li et al., 2010; Arsenault et al., 2018; Motavita et al., 2019). For data-driven ANN models, Anctil et al. (2004), Cigizoglu and Kisi (2005), and Toth and Brath (2007) investigated the effect of calibration data in the context of runoff forecasting, i.e. on the skill of runoff prediction at lead times of several days, using observed runoff and meteorological forcing as predictors. Rainfall-runoff modelling, however, aims to predict runoff from meteorological variables, only. This is a different problem, and we found only one study – conducted by Gauch et al. (2019) – that investigates the effect of calibration data length on the performance of continental-scale data-driven rainfall-runoff models (namely, Extreme Gradient Boosting and Long Short-Term Memory Networks). However, the study does not compare this effect for conceptual hydrological models versus various data-driven models at basin scale, which is a gap that we want to address with the present study.

Thus, the objective of this study is to investigate the effect of calibration data length on the ability of a conceptual hydrological model (GR4H) and two data-driven ANN models to predict runoff from meteorological forcing, i.e. precipitation, temperature, and potential evaporation – at an hourly resolution. Hence, we do not limit our analysis to a single ANN architecture. Instead, we include two modern deep-learning architectures, Long Short-Term Memory Networks (LSTM) and Gated Recurrent Units (GRU), which have just recently been introduced to the field of hydrological modelling (Marçais and de Dreuzy, 2017; Kratzert et al., 2018; Shen, 2018; Xu and Niu, 2018; Reichstein et al., 2019). Specifically, Kratzert et al. (2018, 2019) have identified LSTM as superior to traditional ANN architectures and a conceptual hydrological model. Therefore, we specifically examine to what extent this superiority could depend on the amount of data that is available for model calibration.

Furthermore, our experimental design (Sect. 2.2) provides us a possibility to investigate the effect of model instability, i.e., the sensitivity of model performance on the validation period. This instability reflects both the variability between calibration sub-periods of the same length, and the variability in-between sub-periods of different lengths.

Altogether, this study aims to provide a new perspective on the role of calibration data length, and to serve as a guideline for establishing corresponding calibration-validation experiments. We start with an outline of the experimental setup as well as the underlying data for the meteorological forcing and the observed discharge (Sect. 2); the different models are described in Sect. 3, followed by details on model calibration (Sect. 3.3). We then provide the results of our experiments in Sect. 4 and conclude with Sect. 5.