Machine learning methods are widely used in drug discovery and toxicity prediction. While showing overall good performance in cross-validation studies, their predictive power (often) drops in cases where the query samples have drifted from the training data’s descriptor space. Thus, the assumption for applying machine learning algorithms, that training and test data stem from the same distribution, might not always be fulfilled. In this work, conformal prediction is used to assess the calibration of the models. Deviations from the expected error may indicate that training and test data originate from different distributions. Exemplified on the Tox21 datasets, composed of chronologically released Tox21Train, Tox21Test and Tox21Score subsets, we observed that while internally valid models could be trained using cross-validation on Tox21Train, predictions on the external Tox21Score data resulted in higher error rates than expected. To improve the prediction on the external sets, a strategy exchanging the calibration set with more recent data, such as Tox21Test, has successfully been introduced. We conclude that conformal prediction can be used to diagnose data drifts and other issues related to model calibration. The proposed improvement strategy—exchanging the calibration data only—is convenient as it does not require retraining of the underlying model.

中文翻译：

---

评估具有体外预测的毒理学体外模型中的标定

机器学习方法广泛用于药物发现和毒性预测。尽管在交叉验证研究中显示出总体良好的性能，但在查询样本偏离训练数据的描述符空间的情况下，它们的预测能力（通常）会下降。因此，应用机器学习算法的假设（训练和测试数据源于相同的分布）可能不会始终得到满足。在这项工作中，共形预测用于评估模型的校准。与预期误差的偏离可能表明训练和测试数据源自不同的分布。以Tox21数据集为例，该数据集由按时间顺序发布的Tox21Train，Tox21Test和Tox21Score子集组成，我们观察到，虽然可以使用对Tox21Train的交叉验证来训练内部有效的模型，对外部Tox21Score数据的预测导致错误率高于预期。为了改善对外部数据集的预测，已成功引入了将校准数据集与最新数据交换的策略，例如Tox21Test。我们得出的结论是，保形预测可以用于诊断数据漂移和其他与模型校准有关的问题。提出的改进策略（仅交换校准数据）很方便，因为它不需要重新训练基础模型。我们得出的结论是，保形预测可以用于诊断数据漂移和其他与模型校准有关的问题。提出的改进策略（仅交换校准数据）很方便，因为它不需要重新训练基础模型。我们得出的结论是，保形预测可以用于诊断数据漂移和其他与模型校准有关的问题。提出的改进策略（仅交换校准数据）很方便，因为它不需要重新训练基础模型。