The design of algorithms that generate personalized ranked item lists is a central topic of research in the field of recommender systems. In the past few years, in particular, approaches based on deep learning (neural) techniques have become dominant in the literature. For all of them, substantial progress over the state-of-the-art is claimed. However, indications exist of certain problems in today’s research practice, e.g., with respect to the choice and optimization of the baselines used for comparison, raising questions about the published claims. To obtain a better understanding of the actual progress, we have compared recent results in the area of neural recommendation approaches based on collaborative filtering against a consistent set of existing simple baselines. The worrying outcome of the analysis of these recent works—all were published at prestigious scientific conferences between 2015 and 2018—is that 11 of the 12 reproducible neural approaches can be outperformed by conceptually simple methods, e.g., based on the nearest-neighbor heuristic or linear models. None of the computationally complex neural methods was actually consistently better than already existing learning-based techniques, e.g., using matrix factorization or linear models. In our analysis, we discuss common issues in today’s research practice, which, despite the many papers that are published on the topic, have apparently led the field to a certain level of stagnation. 1

中文翻译：

---

推荐系统研究的可重复性和进展令人不安的分析

生成个性化排名项目列表的算法设计是推荐系统领域研究的中心课题。特别是在过去几年中，基于深度学习（神经）技术的方法在文献中占据主导地位。对于所有这些，都声称在最先进的技术上取得了实质性进展。然而，在当今的研究实践中存在某些问题的迹象，例如，关于用于比较的基线的选择和优化，对已发表的声明提出了质疑。为了更好地了解实际进展，我们将基于协同过滤的神经推荐方法领域的最新结果与一组一致的现有简单基线进行了比较。对这些近期工作的分析令人担忧的结果——所有这些工作都发表在 2015 年至 2018 年间的著名科学会议上——是 12 种可重复的神经方法中有 11 种可以通过概念上简单的方法来超越，例如，基于最近邻启发式或线性模型。没有一种计算复杂的神经方法实际上始终比现有的基于学习的技术更好，例如，使用矩阵分解或线性模型。在我们的分析中，我们讨论了当今研究实践中的常见问题，尽管发表了许多关于该主题的论文，但显然导致该领域处于一定程度的停滞。基于最近邻启发式或线性模型。没有一种计算复杂的神经方法实际上始终比现有的基于学习的技术更好，例如，使用矩阵分解或线性模型。在我们的分析中，我们讨论了当今研究实践中的常见问题，尽管发表了许多关于该主题的论文，但显然导致该领域处于一定程度的停滞。基于最近邻启发式或线性模型。没有一种计算复杂的神经方法实际上始终比现有的基于学习的技术更好，例如，使用矩阵分解或线性模型。在我们的分析中，我们讨论了当今研究实践中的常见问题，尽管发表了许多关于该主题的论文，但显然导致该领域处于一定程度的停滞。1