Though there is widespread recognition of the importance of implementation research, evaluators often face intense logistical, budgetary, and methodological challenges in their efforts to assess intervention implementation in the field. This article proposes a set of natural language processing techniques called semantic similarity as an innovative and scalable method of measuring implementation constructs. Semantic similarity methods are an automated approach to quantifying the similarity between texts. By applying semantic similarity to transcripts of intervention sessions, researchers can use the method to determine whether an intervention was delivered with adherence to a structured protocol, and the extent to which an intervention was replicated with consistency across sessions, sites, and studies. This article provides an overview of semantic similarity methods, describes their application within the context of educational evaluations, and provides a proof of concept using an experimental study of the impact of a standardized teacher coaching intervention.

尽管实施研究的重要性得到了广泛认可，但评估人员在评估该领域干预实施的努力时经常面临激烈的后勤、预算和方法挑战。本文提出了一组称为语义相似性的自然语言处理技术，作为衡量实现构造的创新且可扩展的方法。语义相似性方法是一种量化文本之间相似性的自动化方法。通过将语义相似性应用于干预会话的转录本，研究人员可以使用该方法来确定干预是否在遵守结构化协议的情况下进行，以及干预在会话、站点和研究之间以一致性复制的程度。