How certain are our uncertainty bounds? Accounting for sample variability in Monte Carlo-based uncertainty estimates

https://doi.org/10.1016/j.envsoft.2020.104931Get rights and content

Highlights

  • Due to sampling variability, prediction interval estimates from Monte Carlo simulations can be highly uncertain.

  • A general approach is provided for adjusting prediction interval widths to account for the sample size used in their construction.

  • The method is applicable for any form of probability density function and can be particularly useful when the model is expensive to run.

Abstract

It is common for model-based simulations to be reported using prediction interval estimates that characterize the lack of precision associated with the simulated values. When based on Monte-Carlo sampling to approximate the relevant probability density function(s), such estimates can significantly underestimate the width of the prediction intervals, unless the sample size is sufficiently large. Using theoretical arguments supported by numerical experiments, we discuss the nature and severity of this problem, and demonstrate how better estimates of prediction intervals can be achieved by adjusting the interval width to account for the size of the sample used in its construction. Our method is generally applicable regardless of the form of the underlying probability density function, and can be particularly useful when the model is expensive to run and large samples are not available. We illustrate its use via a simple example involving conceptual modeling of the rainfall-runoff response of a catchment.

Introduction

It is common for simulations or predictions made using dynamical environmental system models (DESMs) to be presented in the form of “best” estimates, accompanied by 95% (or 90% or 99%, etc.) prediction intervals (PIs; see Fig. 1a). The PIs are typically indicated by reporting the positions of the 2.5% and 97.5% quantiles of the probability density function (PDF) characterizing the lack of precision (uncertainty) associated with the model-simulated value (Cho et al., 2016; Hassan et al., 2009; Hirsch et al., 2015; Inam et al., 2017; Roy et al, 2017, 2018; Yang, 2011). Throughout this paper, we use the term “prediction interval” to refer to prediction uncertainty, and the term “confidence interval” to represent uncertainty in some estimated quantity.

In many cases, the reported PIs are estimated via the use of Monte-Carlo sampling to approximate the form of the relevant PDF. Random samples are typically drawn from either the input space (Nikishova et al., 2017) or parameter space (Wagener and Kollat, 2007), or both, and used to generate an ensemble of model simulations from which the PIs are then calculated. The advantage of Monte-Carlo-based methods is that they help us to better understand model behaviors, sensitivities, and uncertainties (Wagener and Kollat, 2007). More detailed methods, also premised on Monte-Carlo sampling, have been proposed (Ajami et al., 2007; Stedinger et al., 2008; Yang et al., 2018).

Monte Carlo-based methods for estimating quantiles have been examined by the non-hydrology literature (e.g. Linnet, 2000; Sun and Lahiri, 2006; Bulter et al., 2017; and references therein), where it has been shown that if the cumulative distribution function (CDF) is differentiable and has a positive derivative at a population quantile, the sample quantile will be asymptotically normal. Further, for this case, the centered and scaled sample quantile Znn(Fn1(p)F1(p)), will converge to a standard normal distribution with zero mean and a variance of p(1p)/f2(F1(p)) in the limiting case (n) where f is the PDF (Sun and Lahiri, 2006); here Fn1 is the sample quantile estimator, F1 is the theoretical quantile, p is the non-exceedance probability, and n is the sample size.

One of the crucial considerations in Monte-Carlo sampling is the size of the sample. In Section 3, we show that for smaller sample sizes, these methods will typically underestimate the width of the PI due to unavoidable considerations of sampling variability. Through theoretical arguments supported by numerical experiments, we investigate and demonstrate the nature and severity of this problem, and its relationship to sample size. We also demonstrate how a better (more representative) estimate of the PI can be achieved by adjusting its width to account for the size of the sample used (Section 4). In Section 5, we briefly illustrate the application of this approach to streamflow uncertainty estimation via hydrological modeling of a catchment.

Section snippets

Computing the quantiles of a probability distribution function

Quantiles of a density function fX(x) are points along the variable axis that divide the range of the PDF into contiguous intervals having equal probability mass (Fig. 1b). A quantile is defined as the value xz such that the cumulative density function (CDF) FX(xz)=xzfX(x)dx=z100, meaning that z % of the total probability mass of fX(x) lies to the left of xz on the variable axis (i.e., in the region xxz for an unbounded distribution), where z is the non-exceedance probability (NEP). So,

Uncertainty associated with sample-based estimates of the quantiles

It is important to note that since many possible equally likely sets Xi={x1i,x2i,,xNi} can be randomly generated (i represents realizations) via Monte-Carlo sampling, there can be many possible estimates FˆX(xi) of the CDF, and hence, many possible corresponding estimates of the quantiles {xˆ2.5i,xˆ97.5i}. In other words, any estimate of quantile xzi is a statistic having its own sampling distribution fXZ(xˆz), where the statistical properties of the distribution depend on both the form of the

Accounting for uncertainty in the estimated quantiles

Given that the PIs estimated using a single Monte-Carlo realization are subject to sampling variability, it would make sense to take this factor into account when reporting the precision associated with a model simulated variable X (e.g., simulated streamflow). One way to acknowledge the imprecision in the estimate (a ‘hat’ is used to denote an estimate) of quantile xz is to compute the quantile yz(xˆz) associated with the quantile of interest (i.e., to take into consideration the sampling

Illustration of application to streamflow estimation via hydrological modeling

We illustrate the use of adjusted prediction intervals for the case of streamflow estimation using the conceptual catchment model HyMod (Boyle et al., 2000) and the Leaf River (Mississippi) data, which has been extensively used in several previous studies (e.g. Brazil and Hudlow, 1981; Gong et al., 2013; Moradkhani et al., 2005; Sorooshian et al., 1983). The model has five adjustable parameters, which can vary within ranges specified by the user. Given the hydrologic model M(x,θ) generating the

Discussion

We have examined the effects of sampling variability on the estimates of prediction intervals computed for an uncertain quantity (e.g., streamflow), when the underlying theoretical PDF is not known, and is instead approximated via Monte-Carlo sampling. In particular, we have investigated the implications of sample sizes, such as those commonly used by modelers, that are not large enough to adequately represent (be properly informative about) the underlying form of the parent PDF.

Our analysis

Conclusions

The effects of sampling variability can significantly affect the estimation of prediction intervals, with significant implications to hydrologic applications, especially when using small Monte-Carlo sample sizes. In this study, we propose and demonstrate a method for adjusting the width of the prediction intervals to compensate for small sample sizes. The method is easy to implement and effectively accounts for the unavoidable effects of sampling variability. By proper adjustment of the

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgements

The second author acknowledges partial support by the Australian Centre of Excellence for Climate System Science (CE110001028). Data and codes used in this study are available upon request from the authors.

References (20)

There are more references available in the full text version of this article.

Cited by (6)

View full text