ABSTRACT

Wordscores is a popular computational text analysis method with numerous applications in communication research. Wordscores claims to scale documents on specified dimensions without requiring researchers to read or even understand the language of the input text. We investigate whether Wordscores delivers this claim by scaling the Euromanifestos of 117 political parties across 23 countries on 4 salient dimensions of political conflict. We assess validity by comparing the Wordscores estimates to expert surveys and other judgmental measures, and by examining the Wordscores’s estimates ability to predict party membership in the European Parliament groups. We find that the Wordscores estimates correlate poorly with expert and judgmental measures of party positions, while the latter outperform Wordscores in the predictive validity test. We conclude that Wordscores does not live up to its original claim of a “quick and easy” language blind method, and urge researchers to demonstrate the validity of the method in their domain of interest before any empirical analysis.

摘要

Wordscores是一种流行的计算文本分析方法，在通讯研究中具有许多应用。Wordscores声称可以按指定的尺寸缩放文档，而无需研究人员阅读或理解输入文本的语言。我们调查Wordscores是否通过在政治冲突的4个显着维度上扩展23个国家/地区的117个政党的《欧洲宣言》来实现这一主张。我们通过将Wordscores的估算值与专家调查和其他判断性措施进行比较，并检查Wordscores的估算能力来预测欧洲议会团体中的政党成员身份，从而评估有效性。我们发现单词分数估计与当事人立场的专家和判断指标之间的相关性很差，而后者在预测效度测试中的表现优于Wordscores。我们得出的结论是，Wordscores不符合其最初的“快速而简单”的语言盲法主张，并敦促研究人员在进行任何实证分析之前，在其感兴趣的领域证明该方法的有效性。