OBJECTIVES Machine learning (ML) and natural language processing have great potential to improve efficiency and accuracy in diagnosis, treatment recommendations, predictive interventions, and scarce resource allocation within psychiatry. Researchers often conceptualize such an approach as operating in isolation without much need for human involvement, yet it remains crucial to harness human-in-the-loop practices when developing and implementing such techniques as their absence may be catastrophic. We advocate for building ML-based technologies that collaborate with experts within psychiatry in all stages of implementation and use to increase model performance while simultaneously increasing the practicality, robustness, and reliability of the process. METHODS We showcase pitfalls of the traditional ML framework and explain how it can be improved with human-in-the-loop techniques. Specifically, we applied active learning strategies to the automatic scoring of a story recall task and compared the results to a traditional approach. RESULTS Human-in-the-loop methodologies supplied a greater understanding of where the model was least confident or had knowledge gaps during training. As compared to the traditional framework, less than half of the training data were needed to reach a given accuracy. CONCLUSIONS Human-in-the-loop ML is an approach to data collection and model creation that harnesses active learning to select the most critical data needed to increase a model's accuracy and generalizability more efficiently than classic random sampling would otherwise allow. Such techniques may additionally operate as safeguards from spurious predictions and can aid in decreasing disparities that artificial intelligence systems otherwise propagate.

中文翻译：

---

通过人在环方法提高人工智能在精神病学应用中的适用性。

目标机器学习 (ML) 和自然语言处理在提高精神病学诊断、治疗建议、预测性干预和稀缺资源分配方面的效率和准确性方面具有巨大潜力。研究人员经常将这种方法概念化为孤立地操作而无需太多人的参与，但在开发和实施此类技术时利用人在环实践仍然至关重要，因为它们的缺席可能是灾难性的。我们提倡构建基于 ML 的技术，在实施的所有阶段与精神病学专家合作，并用于提高模型性能，同时提高过程的实用性、稳健性和可靠性。方法 我们展示了传统 ML 框架的缺陷，并解释了如何使用 Human-in-the-loop 技术对其进行改进。具体来说，我们将主动学习策略应用于故事回忆任务的自动评分，并将结果与​​传统方法进行比较。结果 Human-in-the-loop 方法更深入地了解了模型在训练期间最不自信或存在知识差距的地方。与传统框架相比，达到给定准确度所需的训练数据不到一半。结论 Human-in-the-loop ML 是一种数据收集和模型创建的方法，它利用主动学习来选择最关键的数据，以比经典随机抽样更有效地提高模型的准确性和泛化性。