Abstract
The use of neural networks in hydrology has been frequently undermined by limitations regarding the quantification of uncertainty in predictions. Many authors have proposed different methodologies to overcome these limitations, such as running Monte Carlo simulations, Bayesian approximations, and bootstrapping training samples, which come with computational limitations of their own, and two-step approaches, among others. One less-frequently explored alternative is to repurpose the dropout scheme during inference. Dropout is commonly used during training to avoid overfitting. However, it may also be activated during the testing period to effortlessly provide an ensemble of multiple “sister” predictions. This study explores the predictive uncertainty in hydrological models based on neural networks by comparing a multiparameter ensemble to a dropout ensemble. The dropout ensemble shows more reliable coverage of prediction intervals, while the multiparameter ensemble results in sharper prediction intervals. Moreover, for neural network structures with optimal lookback series, both ensemble strategies result in similar average interval scores. The dropout ensemble, however, benefits from requiring only a single calibration run, i.e., a single set of parameters. In addition, it delivers important insight for engineering design and decision-making with no increase in computational cost. Therefore, the dropout ensemble can be easily included in uncertainty analysis routines and even be combined with multiparameter or multimodel alternatives.
This is a preview of subscription content, log in via an institution to check access.
adapted from Le et al. (2019)
adapted from Srivastava et al. (2014)
Similar content being viewed by others
Data availability
The models developed in this study can be provided by the authors upon request. The datasets used were the (i) GPM IMERG Final Precipitation v06 1 day and (ii) ETo-Brazil v3. Both are stored online and available at their respective repositories: (i) https://disc.gsfc.nasa.gov/datasets/GPM_3IMERGDF_06/summary and (ii) https://data.mendeley.com/datasets/sstjw74ryh/3.
References
Althoff D, Dias SHB, Filgueiras R, Rodrigues LN (2020a) ETo-Brazil: a daily gridded reference evapotranspiration data set for Brazil—repository. Mendeley Data V3. https://doi.org/10.17632/sstjw74ryh.3
Althoff D, Dias SHB, Filgueiras R, Rodrigues LN (2020b) ETo-Brazil: a daily gridded reference evapotranspiration data set for Brazil (2000–2018). Water Resour Res. https://doi.org/10.1029/2020WR027562
Alvares CA, Stape JL, Sentelhas PC et al (2013) Köppen’s climate classification map for Brazil. Meteorol Z 22:711–728. https://doi.org/10.1127/0941-2948/2013/0507
Antonopoulos VZ, Gianniou SK, Antonopoulos AV (2016) Artificial neural networks and empirical equations to estimate daily evaporation: application to Lake Vegoritis, Greece. Hydrol Sci J 61:2590–2599. https://doi.org/10.1080/02626667.2016.1142667
ASCE Task Committee on Application of Artificial Neural Networks in Hydrology (2000) Artificial neural networks in hydrology. I: preliminary concepts. J Hydrol Eng 5:115–123. https://doi.org/10.1061/(ASCE)1084-0699(2000)5:2(115)
ASCE Task Committee on Application of Artificial Neural Networks in Hydrology (2000) Artificial neural networks in hydrology. II: hydrologic applications. J Hydrol Eng 5:124–137. https://doi.org/10.1061/(ASCE)1084-0699(2000)5:2(124)
Blöschl G, Bierkens MFP, Chambel A et al (2019) Twenty-three unsolved problems in hydrology (UPH)—a community perspective. Hydrol Sci J 64:1141–1158. https://doi.org/10.1080/02626667.2019.1620507
Chang L-C, Chang F-J, Chiang Y-M (2004) A two-step-ahead recurrent neural network for stream-flow forecasting. Hydrol Process 18:81–92. https://doi.org/10.1002/hyp.1313
Chen P-A, Chang L-C, Chang F-J (2013) Reinforced recurrent neural networks for multi-step-ahead flood forecasts. J Hydrol 497:71–79. https://doi.org/10.1016/j.jhydrol.2013.05.038
Coron L, Delaigue O, Thirel G et al (2020) airGR: suite of GR hydrological models for precipitation-runoff modelling. Version 1.4.3.65URL https://CRAN.R-project.org/package=airGR
Coron L, Thirel G, Delaigue O et al (2017) The suite of lumped GR hydrological models in an R package. Environ Model Softw 94:166–171. https://doi.org/10.1016/j.envsoft.2017.05.002
Di Baldassarre G, Montanari A (2009) Uncertainty in river discharge observations: a quantitative analysis. Hydrol Earth Syst Sci 13:913–921
Efstratiadis A, Koutsoyiannis D (2010) One decade of multi-objective calibration approaches in hydrological modelling: a review. Hydrol Sci J 55:58–78. https://doi.org/10.1080/02626660903526292
Falbel D, Allaire JJ, Chollet F et al (2020) keras: R interface to “Keras.” Version 2.3.0.0URL https://CRAN.R-project.org/package=keras
Gadelha AN, Coelho VHR, Xavier AC et al (2019) Grid box-level evaluation of IMERG over Brazil at various space and time scales. Atmos Res 218:231–244. https://doi.org/10.1016/j.atmosres.2018.12.001
Gal Y, Ghahramani Z (2016) Dropout as a bayesian approximation: representing model uncertainty in deep learning. In: Proceedings of machine learning research. New York, USA, pp 1050–1059
Garrick M, Cunnane C, Nash JE (1978) A criterion of efficiency for rainfall-runoff models. J Hydrol 36:375–381. https://doi.org/10.1016/0022-1694(78)90155-5
Gneiting T, Raftery AE (2007) Strictly proper scoring rules, prediction, and estimation. J Am Stat Assoc 102:359–378. https://doi.org/10.1198/016214506000001437
Gupta HV, Kling H, Yilmaz KK, Martinez GF (2009) Decomposition of the mean squared error and NSE performance criteria: implications for improving hydrological modelling. J Hydrol 377:80–91. https://doi.org/10.1016/j.jhydrol.2009.08.003
Her Y, Yoo S-H, Cho J et al (2019) Uncertainty in hydrological analysis of climate change: multi-parameter vs. multi-GCM ensemble predictions. Sci Rep 9:4974. https://doi.org/10.1038/s41598-019-41334-7
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9:1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Hochreiter S, Schmidhuber J (1997) LSTM can solve hard long time lag problems. In: Mozer MC, Jordan MI, Petsche T (eds) Advances in neural information processing systems, vol 9. MIT Press, Cambridge, pp 473–479
Huffman GJ, Bolvin DT, Braithwaite D et al (2019a) Integrated multi-satellitE retrievals for GPM (IMERG). Natioanl Aeronautics and Space Administration
Huffman GJ, Stocker EF, Bolvin DT et al (2019b) GPM IMERG final precipitation L3 1 day 0.1 degree x 0.1 degree V06. Goddard Earth Sciences Data and Information Services Center (GES DISC), Greenbelt, MD
Jalbert J, Mathevet T, Favre A-C (2011) Temporal uncertainty estimation of discharges from rating curves using a variographic analysis. J Hydrol 397:83–92. https://doi.org/10.1016/j.jhydrol.2010.11.031
Khan MS, Coulibaly P (2006) Bayesian neural network for rainfall-runoff modeling. Water Resour Res. https://doi.org/10.1029/2005WR003971
Khosravi A, Nahavandi S, Creighton D (2013) Quantifying uncertainties of neural network-based electricity price forecasts. Appl Energy 112:120–129. https://doi.org/10.1016/j.apenergy.2013.05.075
Kingston GB, Lambert MF, Maier HR (2005) Bayesian training of artificial neural networks used for water resources modeling. Water Resour Res. https://doi.org/10.1029/2005WR004152
Kratzert F, Herrnegger M, Klotz D et al (2019) Neural hydrology—interpreting LSTMs in hydrology. In: Samek W, Montavon G, Vedaldi A et al (eds) Explainable AI: interpreting, explaining and visualizing deep learning. Springer International Publishing, Cham, pp 347–362
Krause P, Boyle DP, Bäse F (2005) Comparison of different efficiency criteria for hydrological model assessment. Adv Geosci 5:89–97. https://doi.org/10.5194/adgeo-5-89-2005
Kumar DN, Raju KS, Sathish T (2004) River flow forecasting using recurrent neural networks. Water Resour Manag 18:143–161. https://doi.org/10.1023/B:WARM.0000024727.94701.12
Le X-H, Ho HV, Lee G, Jung S (2019) Application of long short-term memory (LSTM) neural network for flood forecasting. Water 11:1387. https://doi.org/10.3390/w11071387
Legates DR, McCabe GJ (1999) Evaluating the use of “goodness-of-fit” measures in hydrologic and hydroclimatic model validation. Water Resour Res 35:233–241. https://doi.org/10.1029/1998WR900018
Maier HR, Dandy GC (2000) Neural networks for the prediction and forecasting of water resources variables: a review of modelling issues and applications. Environ Model Softw 15:101–124. https://doi.org/10.1016/S1364-8152(99)00007-9
MapBiomas (2020) Projeto de Mapeamento Anual da Cobertura e Uso do Solo do Brasil [In English: Brazilian annual land use and land cover mapping project]. In: MapBiomas V41. http://mapbiomas.org/
McGlynn BL, Blöschl G, Borga M et al (2013) A data acquisition framework for runoff prediction in ungauged basins. In: Blöschl G, Sivapalan M, Wagener T et al (eds) Runoff prediction in ungauged basins. Cambridge University Press, Cambridge, pp 29–52
Nash JE, Sutcliffe JV (1970) River flow forecasting through conceptual models part I—a discussion of principles. J Hydrol 10:282–290. https://doi.org/10.1016/0022-1694(70)90255-6
Papacharalampous G, Tyralis H (2020) Hydrological time series forecasting using simple combinations: big data testing and investigations on one-year ahead river flow predictability. J Hydrol 590:125205. https://doi.org/10.1016/j.jhydrol.2020.125205
Papacharalampous G, Tyralis H, Koutsoyiannis D (2019a) Comparison of stochastic and machine learning methods for multi-step ahead forecasting of hydrological processes. Stoch Environ Res Risk Assess 33:481–514. https://doi.org/10.1007/s00477-018-1638-6
Papacharalampous G, Tyralis H, Koutsoyiannis D, Montanari A (2020) Quantification of predictive uncertainty in hydrological modelling by harnessing the wisdom of the crowd: a large-sample experiment at monthly timescale. Adv Water Resour 136:103470. https://doi.org/10.1016/j.advwatres.2019.103470
Papacharalampous G, Tyralis H, Langousis A et al (2019b) Probabilistic hydrological post-processing at scale: why and how to apply machine-learning quantile regression algorithms. Water 11:2126. https://doi.org/10.3390/w11102126
Papacharalampous GA, Tyralis H (2018) Evaluation of random forests and Prophet for daily streamflow forecasting. Adv Geosci 45:201–208. https://doi.org/10.5194/adgeo-45-201-2018
Perrin C, Michel C, Andréassian V (2003) Improvement of a parsimonious model for streamflow simulation. J Hydrol 279:275–289. https://doi.org/10.1016/S0022-1694(03)00225-7
Pham LT, Luo L, Finley AO (2020) Evaluation of random forest for short-term daily streamflow forecast in rainfall and snowmelt driven watersheds. Hydrol Earth Syst Sci Discuss. https://doi.org/10.5194/hess-2020-305
Pushpalatha R, Perrin C, Le Moine N et al (2011) A downward structural sensitivity analysis of hydrological models to improve low-flow simulation. J Hydrol 411:66–76. https://doi.org/10.1016/j.jhydrol.2011.09.034
R Core Team (2020) R: a language and environment for statistical computing. R Foundation for Statistical Computing, Vienna
Ritter A, Muñoz-Carpena R (2013) Performance evaluation of hydrological models: statistical significance for reducing subjectivity in goodness-of-fit assessments. J Hydrol 480:33–45. https://doi.org/10.1016/j.jhydrol.2012.12.004
Sahoo BB, Jha R, Singh A, Kumar D (2019) Long short-term memory (LSTM) recurrent neural network for low-flow hydrological time series forecasting. Acta Geophys. https://doi.org/10.1007/s11600-019-00330-1
Srivastava N, Hinton G, Krizhevsky A et al (2014) Dropout: a simple way to prevent neural networks from overfitting. J Mach Learn Res 15:1929–1958
Tegegne G, Park DK, Kim Y-O (2017) Comparison of hydrological models for the assessment of water resources in a data-scarce region, the Upper Blue Nile River Basin. J Hydrol Reg Stud 14:49–66. https://doi.org/10.1016/j.ejrh.2017.10.002
Tomkins KM (2014) Uncertainty in streamflow rating curves: methods, controls and consequences: uncertainty in streamflow rating curves. Hydrol Process 28:464–481. https://doi.org/10.1002/hyp.9567
Wei S, Zuo D, Song J (2012) Improving prediction accuracy of river discharge time series using a wavelet-NAR artificial neural network. J Hydroinform 14:974–991. https://doi.org/10.2166/hydro.2012.143
Wen Y, AlHakeem D, Mandal P et al (2020) Performance evaluation of probabilistic methods based on bootstrap and quantile regression to quantify PV power point forecast uncertainty. IEEE Trans Neural Netw Learn Syst 31:1134–1144. https://doi.org/10.1109/TNNLS.2019.2918795
Worland SC, Steinschneider S, Asquith W et al (2019) Prediction and inference of flow duration curves using multioutput neural networks. Water Resour Res 55:6850–6868. https://doi.org/10.1029/2018WR024463
Yapo PO, Gupta HV, Sorooshian S (1998) Multi-objective global optimization for hydrologic models. J Hydrol 204:83–97. https://doi.org/10.1016/S0022-1694(97)00107-8
Yaseen ZM, El-Shafie A, Afan HA et al (2016) RBFNN versus FFNN for daily river flow forecasting at Johor River, Malaysia. Neural Comput Appl 27:1533–1542. https://doi.org/10.1007/s00521-015-1952-6
Zealand CM, Burn DH, Simonovic SP (1999) Short term streamflow forecasting using artificial neural networks. J Hydrol 214:32–48. https://doi.org/10.1016/S0022-1694(98)00242-X
Zeiler MD (2012) ADADELTA: an adaptive learning rate method. ArXiv12125701 Cs
Zhang C, Chu J, Fu G (2013) Sobol′’s sensitivity analysis for a distributed hydrological model of Yichun River Basin, China. J Hydrol 480:58–68. https://doi.org/10.1016/j.jhydrol.2012.12.005
Zhang X, Liang F, Srinivasan R, Liew MV (2009) Estimating uncertainty of streamflow simulation using Bayesian neural networks. Water Resour Res. https://doi.org/10.1029/2008WR007030
Zhang X, Zhao K (2012) Bayesian neural networks for uncertainty analysis of hydrologic modeling: a comparison of two schemes. Water Resour Manag 26:2365–2382. https://doi.org/10.1007/s11269-012-0021-5
Zhu S, Luo X, Yuan X, Xu Z (2020) An improved long short-term memory network for streamflow forecasting in the upper Yangtze River. Stoch Environ Res Risk Assess. https://doi.org/10.1007/s00477-020-01766-4
Acknowledgements
This study was financed in part by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES—In English: Coordination of Improvement of Higher Education Personnel) – Finance code 001, and by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPQ – In English: National Council for Scientific and Technological Development) – Grant number 142273/2019-8.
Author information
Authors and Affiliations
Contributions
Conception: DA and HCB; data collection, curation, and modeling: DA; analysis and interpretation, writing, and reviewing; DA, LNR, and HCB; supervision: LNR.
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Clinical trials registration
Does not apply.
Ethics approval
Animal research: Does not apply. Consent to participate: Does not apply. Consent to publish: Does not apply. All authors have consented to participate and publish the manuscript.
Plant reproducibility
Does not apply.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Althoff, D., Rodrigues, L.N. & Bazame, H.C. Uncertainty quantification for hydrological models based on neural networks: the dropout ensemble. Stoch Environ Res Risk Assess 35, 1051–1067 (2021). https://doi.org/10.1007/s00477-021-01980-8
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00477-021-01980-8