当前位置: X-MOL 学术arXiv.cs.CY › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Algorithm Auditing at a Large-Scale: Insights from Search Engine Audits
arXiv - CS - Computers and Society Pub Date : 2021-06-10 , DOI: arxiv-2106.05831
Roberto Ulloa, Mykola Makhortykh, Aleksandra Urman

Algorithm audits have increased in recent years due to a growing need to independently assess the performance of automatically curated services that process, filter and rank the large and dynamic amount of information available on the internet. Among several methodologies to perform such audits, virtual agents stand out because they offer the possibility of performing systematic experiments simulating human behaviour without the associated costs of recruiting participants. Motivated by the importance of research transparency and replicability of results, this paper focuses on the challenges of such an approach, and it provides methodological details, recommendations, lessons learned and limitations that researchers should take into consideration when setting up experiments with virtual agents. We demonstrate the successful performance of our research infrastructure in multiple data collections with diverse experimental designs, and point to different changes and strategies that improved the quality of the method. We conclude that virtual agents are a promising venue for monitoring the performance of algorithms during longer periods of time, and we hope that this paper serves as a base to widen the research in this direction.

中文翻译:

大规模算法审计:来自搜索引擎审计的见解

近年来,由于越来越需要独立评估自动策划服务的性能,这些服务处理、过滤和排名互联网上大量动态信息,因此算法审计有所增加。在执行此类审计的几种方法中,虚拟代理脱颖而出,因为它们提供了执行模拟人类行为的系统实验的可能性,而无需招募参与者的相关成本。受研究透明度和结果可重复性的重要性的启发,本文重点关注这种方法的挑战,并提供了研究人员在设置虚拟代理实验时应考虑的方法细节、建议、经验教训和局限性。我们展示了我们的研究基础设施在具有不同实验设计的多个数据集合中的成功表现,并指出了提高方法质量的不同变化和策略。我们得出的结论是,虚拟代理是在较长时间内监控算法性能的一个有前途的场所,我们希望本文作为扩展这一方向研究的基础。
更新日期:2021-06-11
down
wechat
bug