Abstract

A key issue in machine learning research is the lack of reproducibility. We illustrate what role hyperparameter search plays in this problem and how regular hyperparameter search methods can lead to a large variance in outcomes due to nondeterministic model training during hyperparameter optimization. The variation in outcomes poses a problem both for reproducibility of the hyperparameter search itself and comparisons of different methods each optimized using hyperparameter search. In addition, the fact that hyperparameter search may result in nonoptimal hyperparameter settings may affect other studies, since hyperparameter settings are often copied from previously published research. To remedy this issue, we define the mean prediction error across model training runs as the objective for the hyperparameter search. We then propose a hypothesis testing procedure that makes inference on the mean performance of each hyperparameter setting and results in an equivalence class of hyperparameter settings that are not distinguishable in performance. We further embed this procedure into a group sequential testing framework to increase efficiency in terms of the average number of model training replicates required. Empirical results on machine learning benchmarks show that at equal computation the proposed method reduces the variation in hyperparameter search outcomes by up to 90% while resulting in equal or lower mean prediction errors when compared to standard random search and Bayesian optimization. Moreover, the sequential testing framework successfully reduces computation while preserving performance of the method. The code to reproduce the results is available online and in the supplementary materials.

摘要

机器学习研究的一个关键问题是缺乏可重复性。我们说明了超参数搜索在这个问题中的作用，以及由于超参数优化过程中的非确定性模型训练，常规超参数搜索方法如何导致结果的巨大差异。结果的变化给超参数搜索本身的可重复性和使用超参数搜索优化的不同方法的比较带来了问题。此外，超参数搜索可能导致非最优超参数设置这一事实可能会影响其他研究，因为超参数设置通常是从先前发表的研究中复制而来的。为了解决这个问题，我们将模型训练运行的平均预测误差定义为超参数搜索的目标。然后，我们提出了一个假设检验程序，该程序对每个超参数设置的平均性能进行推断，并导致在性能上不可区分的等价类超参数设置。我们进一步将此过程嵌入到组顺序测试框架中，以提高所需的模型训练重复次数的平均效率。机器学习基准的经验结果表明，在同等计算下，与标准随机搜索和贝叶斯优化相比，所提出的方法可将超参数搜索结果的变化减少多达 90%，同时产生相等或更低的平均预测误差。此外，顺序测试框架成功地减少了计算量，同时保持了方法的性能。