Obtaining accurate estimates of machine learning model uncertainties on newly predicted data is essential for understanding the accuracy of the model and whether its predictions can be trusted. A common approach to such uncertainty quantification is to estimate the variance from an ensemble of models, which are often generated by the generally applicable bootstrap method. In this work, we demonstrate that the direct bootstrap ensemble standard deviation is not an accurate estimate of uncertainty but that it can be simply calibrated to dramatically improve its accuracy. We demonstrate the effectiveness of this calibration method for both synthetic data and numerous physical datasets from the field of Materials Science and Engineering. The approach is motivated by applications in physical and biological science but is quite general and should be applicable for uncertainty quantification in a wide range of machine learning regression models.

在新预测的数据上获得机器学习模型不确定性的准确估计对于理解模型的准确性以及其预测是否可信至关重要。这种不确定性量化的常用方法是从一组模型中估计方差，这些模型通常由普遍适用的引导方法生成。在这项工作中，我们证明了直接引导集成标准偏差不是对不确定性的准确估计，但可以简单地对其进行校准以显着提高其准确性。我们证明了这种校准方法对来自材料科学与工程领域的合成数据和大量物理数据集的有效性。