Social bots – partially or fully automated accounts on social media platforms – have not only been widely discussed, but have also entered political, media and research agendas. However, bot detection is not an exact science. Quantitative estimates of bot prevalence vary considerably and comparative research is rare. We show that findings on the prevalence and activity of bots on Twitter depend strongly on the methods used to identify automated accounts. We search for bots in political discourses on Twitter, using three different bot detection methods: Botometer, Tweetbotornot and “heavy automation”. We drew a sample of 122,884 unique user Twitter accounts that had produced 263,821 tweets contributing to five political discourses in five Western democracies. While all three bot detection methods classified accounts as bots in all our cases, the comparison shows that the three approaches produce very different results. We discuss why neither manual validation nor triangulation resolves the basic problems, and conclude that social scientists studying the influence of social bots on (political) communication and discourse dynamics should be careful with easy-to-use methods, and consider interdisciplinary research.

社交机器人——社交媒体平台上部分或完全自动化的账户——不仅被广泛讨论，而且还进入了政治、媒体和研究议程。然而，机器人检测并不是一门精确的科学。机器人流行率的定量估计差异很大，比较研究很少见。我们表明，关于 Twitter 上机器人的流行和活动的调查结果在很大程度上取决于用于识别自动帐户的方法。我们在 Twitter 上的政治话语中搜索机器人，使用三种不同的机器人检测方法：Botometer、Tweetbotornot 和“重度自动化”。我们抽取了 122,884 个独立用户 Twitter 帐户的样本，这些帐户产生了 263,821 条推文，为五个西方民主国家的五个政治话语做出了贡献。虽然在我们所有的案例中，所有三种机器人检测方法都将帐户归类为机器人，比较表明，这三种方法产生了截然不同的结果。我们讨论了为什么手动验证和三角测量都不能解决基本问题，并得出结论，研究社交机器人对（政治）交流和话语动态的影响的社会科学家应该谨慎使用易于使用的方法，并考虑跨学科研究。