Background and Hypothesis Despite decades of “proof of concept” findings supporting the use of Natural Language Processing (NLP) in psychosis research, clinical implementation has been slow. One obstacle reflects the lack of comprehensive psychometric evaluation of these measures. There is overwhelming evidence that criterion and content validity can be achieved for many purposes, particularly using machine learning procedures. However, there has been very little evaluation of test-retest reliability, divergent validity (sufficient to address concerns of a “generalized deficit”), and potential biases from demographics and other individual differences. Study Design This article highlights these concerns in development of an NLP measure for tracking clinically rated paranoia from video “selfies” recorded from smartphone devices. Patients with schizophrenia or bipolar disorder were recruited and tracked over a week-long epoch. A small NLP-based feature set from 499 language samples were modeled on clinically rated paranoia using regularized regression. Study Results While test–retest reliability was high, criterion, and convergent/divergent validity were only achieved when considering moderating variables, notably whether a patient was away from home, around strangers, or alone at the time of the recording. Moreover, there were systematic racial and sex biases in the model, in part, reflecting whether patients submitted videos when they were away from home, around strangers, or alone. Conclusions Advancing NLP measures for psychosis will require deliberate consideration of test-retest reliability, divergent validity, systematic biases and the potential role of moderators. In our example, a comprehensive psychometric evaluation revealed clear strengths and weaknesses that can be systematically addressed in future research.

中文翻译：

---

自然语言处理和精神病：关于综合心理测量评估的需要

背景和假设尽管数十年的“概念验证”研究结果支持在精神病研究中使用自然语言处理（NLP），但临床实施进展缓慢。其中一个障碍反映了缺乏对这些措施的全面心理测量评估。有压倒性的证据表明，可以出于多种目的实现标准和内容有效性，特别是使用机器学习程序。然而，对重测可靠性、差异有效性（足以解决“普遍缺陷”的担忧）以及人口统计和其他个体差异带来的潜在偏差的评估却很少。研究设计本文强调了开发 NLP 测量方法中的这些问题，用于跟踪智能手机设备录制的视频“自拍照”中临床评定的偏执狂。招募了精神分裂症或双相情感障碍患者，并对其进行了为期一周的跟踪。使用正则化回归，根据临床评定的偏执狂对来自 499 个语言样本的基于 NLP 的小型特征集进行建模。研究结果虽然重测信度很高，但只有在考虑调节变量时才能实现标准效度和收敛/发散效度，特别是在记录时患者是否离开家、在陌生人身边或独自一人。此外，模型中存在系统性的种族和性别偏见，部分反映了患者是否在离开家、在陌生人身边或独自一人时提交视频。结论 推进 NLP 治疗精神病的措施需要认真考虑重测可靠性、发散有效性、系统偏差和调节者的潜在作用。在我们的示例中，全面的心理测量评估揭示了明显的优点和缺点，可以在未来的研究中系统地解决。