ABSTRACT Political communication has become one of the central arenas of innovation in the application of automated analysis approaches to ever-growing quantities of digitized texts. However, although researchers routinely and conveniently resort to certain forms of human coding to validate the results derived from automated procedures, in practice the actual “quality assurance” of such a “gold standard” often goes unchecked. Contemporary practices of validation via manual annotations are far from being acknowledged as best practices in the literature, and the reporting and interpretation of validation procedures differ greatly. We systematically assess the connection between the quality of human judgment in manual annotations and the relative performance evaluations of automated procedures against true standards by relying on large-scale Monte Carlo simulations. The results from the simulations confirm that there is a substantially greater risk of a researcher reaching an incorrect conclusion regarding the performance of automated procedures when the quality of manual annotations used for validation is not properly ensured. Our contribution should therefore be regarded as a call for the systematic application of high-quality manual validation materials in any political communication study, drawing on automated text analysis procedures.

中文翻译：

---

在我们信任的验证中？不完美的人工注释作为黄金标准对自动内容分析验证质量的影响

摘要 政治传播已成为将自动化分析方法应用于数量不断增长的数字化文本的创新中心领域之一。然而，尽管研究人员经常方便地采用某些形式的人工编码来验证从自动化程序得出的结果，但在实践中，这种“黄金标准”的实际“质量保证”往往不受检查。通过手动注释进行验证的当代实践远未在文献中被认为是最佳实践，并且验证程序的报告和解释差异很大。我们依靠大规模蒙特卡罗模拟系统地评估了人工注释中人工判断的质量与自动程序相对于真实标准的相对性能评估之间的联系。模拟结果证实，当用于验证的手动注释的质量没有得到适当保证时，研究人员对自动化程序的性能得出错误结论的风险要大得多。因此，我们的贡献应被视为呼吁在任何政治传播研究中系统应用高质量的手动验证材料，利用自动文本分析程序。模拟结果证实，当用于验证的手动注释的质量没有得到适当保证时，研究人员对自动化程序的性能得出错误结论的风险要大得多。因此，我们的贡献应被视为呼吁在任何政治传播研究中系统应用高质量的手动验证材料，利用自动文本分析程序。模拟结果证实，当用于验证的手动注释的质量没有得到适当保证时，研究人员对自动化程序的性能得出错误结论的风险要大得多。因此，我们的贡献应被视为呼吁在任何政治传播研究中系统应用高质量的手动验证材料，利用自动文本分析程序。