OBJECTIVE The goal of this study is to assess machine learning for predicting procedure performance from operator and procedure characteristics. BACKGROUND Procedures are vital for the performance and safety of high-risk industries. Current procedure design guidelines are insufficient because they rely on subjective assessments and qualitative analyses that struggle to integrate and quantify the diversity of factors that influence procedure performance. METHOD We used data from a 25-participant study with four procedures, conducted on a high-fidelity oil extraction simulation to develop logistic regression (LR), random forest (RF), and decision tree (DT) algorithms that predict procedure step performance from operator, step, readability, and natural language processing-based features. Features were filtered using the Boruta approach. The algorithms were trained and optimized with a repeated 10-fold cross-validation. After training, inference was performed using variable importance and partial dependence plots. RESULTS The RF, DT, and LR algorithms with all features had an area under the receiver operating characteristic curve (AUC) of 0.78, 0.77, and 0.75, respectively, and significantly outperformed the LR with only operator features (LROP), with an AUC of 0.61. The most important features were experience, familiarity, total words, and character-based metrics. The partial dependence plots showed that steps with fewer words, abbreviations, and characters were correlated with correct step performance. CONCLUSION Machine learning algorithms are a promising approach for predicting step-level procedure performance, with acknowledged limitations on interpolating to nonobserved data, and may help guide procedure design after validation with additional data on further tasks. APPLICATION After validation, the inferences from these models can be used to generate procedure design alternatives.

中文翻译：

---

根据操作员和文本特征预测程序步骤性能：机器学习驱动程序设计的关键第一步

目的 本研究的目的是评估机器学习根据操作员和程序特征预测程序性能的能力。背景技术程序对于高风险行业的绩效和安全至关重要。当前的程序设计指南是不够的，因为它们依赖于主观评估和定性分析，难以整合和量化影响程序性能的因素的多样性。方法 我们使用来自 25 名参与者的四个程序研究的数据，在高保真石油开采模拟中进行，以开发逻辑回归 (LR)、随机森林 (RF) 和决策树 (DT) 算法，用于预测程序步骤性能基于运算符、步骤、可读性和自然语言处理的功能。使用 Boruta 方法过滤特征。通过重复 10 倍交叉验证对算法进行训练和优化。训练后，使用变量重要性和部分依赖图进行推理。结果 具有所有特征的 RF、DT 和 LR 算法的接收者操作特征曲线下面积 (AUC) 分别为 0.78、0.77 和 0.75，并且显着优于仅具有操作员特征的 LR (LROP)。 0.61。最重要的特征是经验、熟悉程度、总单词数和基于性格的指标。部分依赖图表明，单词、缩写和字符较少的步骤与正确的步骤表现相关。结论 机器学习算法是预测步骤级程序性能的一种有前途的方法，在插值到非观察数据方面具有公认的局限性，并且可以在使用有关进一步任务的附加数据进行验证后帮助指导程序设计。应用 验证后，这些模型的推论可用于生成程序设计替代方案。