Research challenges encountered across science, engineering, and economics can frequently be formulated as optimization tasks. In chemistry and materials science, recent growth in laboratory digitization and automation has sparked interest in optimization-guided autonomous discovery and closed-loop experimentation. Experiment planning strategies based on off-the-shelf optimization algorithms can be employed in fully autonomous research platforms to achieve desired experimentation goals with the minimum number of trials. However, the experiment planning strategy that is most suitable to a scientific discovery task is a priori unknown while rigorous comparisons of different strategies are highly time and resource demanding. As optimization algorithms are typically benchmarked on low-dimensional synthetic functions, it is unclear how their performance would translate to noisy, higher-dimensional experimental tasks encountered in chemistry and materials science. We introduce Olympus, a software package that provides a consistent and easy-to-use framework for benchmarking optimization algorithms against realistic experiments emulated via probabilistic deep-learning models. Olympus includes a collection of experimentally derived benchmark sets from chemistry and materials science and a suite of experiment planning strategies that can be easily accessed via a user-friendly Python interface. Furthermore, Olympus facilitates the integration, testing, and sharing of custom algorithms and user-defined datasets. In brief, Olympus mitigates the barriers associated with benchmarking optimization algorithms on realistic experimental scenarios, promoting data sharing and the creation of a standard framework for evaluating the performance of experiment planning strategies.

在科学、工程和经济学领域遇到的研究挑战经常可以表述为优化任务。在化学和材料科学领域，最近实验室数字化和自动化的发展激发了对优化引导的自主发现和闭环实验的兴趣。可以在完全自主的研究平台中采用基于现成优化算法的实验规划策略，以最少的试验次数实现预期的实验目标。然而，最适合科学发现任务的实验计划策略是先验的。未知，而对不同策略的严格比较需要大量时间和资源。由于优化算法通常以低维合成函数为基准，目前尚不清楚它们的性能如何转化为化学和材料科学中遇到的嘈杂、高维实验任务。我们介绍了Olympus，这是一个软件包，它提供了一个一致且易于使用的框架，用于针对通过概率深度学习模型模拟的现实实验对优化算法进行基准测试。Olympus包括一系列来自化学和材料科学的实验派生基准集和一套实验规划策略，可通过用户友好的 Python 界面轻松访问。此外，奥林巴斯促进了自定义算法和用户定义数据集的集成、测试和共享。简而言之，奥林巴斯减轻了与现实实验场景中的基准优化算法相关的障碍，促进数据共享和创建用于评估实验规划策略性能的标准框架。