ABSTRACT

As we take advantage of new technologies that allow us to streamline the coding process of large qualitative datasets, we must consider whether human cognitive bias may introduce statistical bias in the process. Our research group analyzes large sets of student responses by developing computer models that are trained using human-coded responses and a suite of machine-learning techniques. Once a model is initially trained, it may be insufficiently accurate. Increasing the number of human-coded responses typically enhances these models to an acceptable level of accuracy. Alternatively, instead of human coding responses, we can rapidly increase the number of coded responses by verifying computer-predicted codes for each response. However, having access to this information may bias human coders. We designed the present study to test for differences in level of agreement with computer-predicted codes in terms of magnitude and direction during computer model calibration if information about computer-predicted codes is available. Our results indicate human coding bias despite being disciplinary experts who were aware of the possibility of cognitive bias creating statistical bias and that magnitude and direction of that bias varies across experts.

摘要

当我们利用新技术简化大型定性数据集的编码过程时，我们必须考虑人类认知偏差是否会在此过程中引入统计偏差。我们的研究小组通过开发使用人工编码响应和一套机器学习技术训练的计算机模型来分析大量学生的响应。一旦模型最初被训练，它可能不够准确。增加人工编码响应的数量通常会将这些模型提高到可接受的准确度水平。或者，我们可以通过验证每个响应的计算机预测代码来快速增加编码响应的数量，而不是人工编码响应。但是，访问此信息可能会使人类编码人员产生偏见。如果有关计算机预测代码的信息可用，我们设计本研究以测试计算机模型校准过程中与计算机预测代码在幅度和方向方面的一致性水平差异。我们的结果表明人类编码偏差，尽管他们是学科专家，他们意识到认知偏差产生统计偏差的可能性，并且这种偏差的大小和方向因专家而异。