当前位置: X-MOL 学术J. Am. Med. Inform. Assoc. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
An evaluation of two commercial deep learning-based information retrieval systems for COVID-19 literature
Journal of the American Medical Informatics Association ( IF 6.4 ) Pub Date : 2020-11-17 , DOI: 10.1093/jamia/ocaa271
Sarvesh Soni 1 , Kirk Roberts 1
Affiliation  

Abstract
The COVID-19 pandemic has resulted in a tremendous need for access to the latest scientific information, leading to both corpora for COVID-19 literature and search engines to query such data. While most search engine research is performed in academia with rigorous evaluation, major commercial companies dominate the web search market. Thus, it is expected that commercial pandemic-specific search engines will gain much higher traction than academic alternatives, leading to questions about the empirical performance of these tools. This paper seeks to empirically evaluate two commercial search engines for COVID-19 (Google and Amazon) in comparison with academic prototypes evaluated in the TREC-COVID task. We performed several steps to reduce bias in the manual judgments to ensure a fair comparison of all systems. We find the commercial search engines sizably underperformed those evaluated under TREC-COVID. This has implications for trust in popular health search engines and developing biomedical search engines for future health crises.


中文翻译:

评估两个基于商业深度学习的COVID-19文献​​信息检索系统

摘要
COVID-19大流行导致对获取最新科学信息的巨大需求,导致COVID-19文献​​的语料库和搜索引擎可以查询此类数据。尽管大多数搜索引擎研究都是在学术界进行严格评估的,但主要的商业公司主导着网络搜索市场。因此,可以预期的是,针对大流行的商业搜索引擎将比学术选择具有更高的吸引力,从而引发对这些工具的经验性能的质疑。本文旨在与TREC-COVID任务中评估的学术原型相比,以经验评估两个COVID-19的商业搜索引擎(Google和Amazon)。我们执行了几个步骤来减少手动判断中的偏差,以确保所有系统的公平比较。我们发现,商业搜索引擎的表现远不及TREC-COVID评估的那些。这对信任大众健康搜索引擎以及开发生物医学搜索引擎以应对未来的健康危机具有重要意义。
更新日期:2021-01-16
down
wechat
bug