Purpose

How to provide the most useful papers for searchers is a key issue for academic search engines. A lot of research has been carried out to address this problem. However, when evaluating the effectiveness of an academic search engine, most of the previous investigations assume that the only concern of the user is the relevancy of the paper to the query. The authors believe that the usefulness of a paper is determined not only by its relevance to the query but also by other aspects including its publication age and impact in the research community. This is vital, especially when a large number of papers are relevant to the query.

Design/methodology/approach

This paper proposes a group of metrics to measure the usefulness of a ranked list of papers. When defining these metrics, three factors, including relevance, publication age and impact, are considered at the same time. To accommodate this, the authors propose a framework to rank papers by a combination of their relevance, publication age and impact scores.

Findings

The framework is evaluated with the ACL (Association for Computational Linguistics Anthology Network) dataset. It demonstrates that the proposed ranking algorithm is effective for improving usefulness when two or three aspects of academic papers are considered at the same time, while the relevance of the retrieved papers is slightly down compared with the relevance-only retrieval.

Originality/value

To the best of the authors’ knowledge, the proposed multi-objective academic search framework is the first of its kind that is proposed and evaluated with a group of new evaluation metrics.

中文翻译：

---

一种确定论文在学术搜索中有用性的多目标方法

目的

如何为搜索者提供最有用的论文是学术搜索引擎的一个关键问题。已经进行了大量研究来解决这个问题。然而，在评估学术搜索引擎的有效性时，之前的大多数调查都假设用户唯一关心的是论文与查询的相关性。作者认为，一篇论文的有用性不仅取决于其与查询的相关性，还取决于其他方面，包括其发表时间和在研究界的影响。这是至关重要的，尤其是当大量论文与查询相关时。

设计/方法/方法

本文提出了一组指标来衡量论文排名列表的有用性。在定义这些指标时，同时考虑了三个因素，包括相关性、出版年龄和影响。为了适应这一点，作者提出了一个框架，根据论文的相关性、发表时间和影响力评分对论文进行排名。

发现

该框架使用 ACL（计算语言学选集网络协会）数据集进行评估。这表明当同时考虑学术论文的两个或三个方面时，所提出的排序算法可以有效提高有用性，而检索到的论文的相关性与仅相关性检索相比略有下降。

原创性/价值

据作者所知，所提出的多目标学术搜索框架是同类中第一个提出并使用一组新评估指标进行评估的框架。