The development of scalable, representative, and widely adopted benchmarks for graph data systems have been a question for which answers has been sought for decades. We conduct an in-depth study of the existing literature on benchmarks for graph data management and processing, covering 20 different benchmarks developed during the last 15 years. We categorize the benchmarks into three areas focusing on benchmarks for graph processing systems, graph database benchmarks, and bigdata benchmarks with graph processing workloads. This systematic approach allows us to identify multiple issues existing in this area, including i) few benchmarks exist which can produce high workload scenarios, ii) no significant work done on benchmarking graph stream processing as well as graph based machine learning, iii) benchmarks tend to use conventional metrics despite new meaningful metrics have been around for years, iv) increasing number of big data benchmarks appear with graph processing workloads. Following these observations, we conclude the survey by describing key challenges for future research on graph data systems benchmarking.

中文翻译：

---

对图形数据管理和处理系统进行基准测试：一项调查

几十年来，图形数据系统的可扩展、有代表性和广泛采用的基准的开发一直是一个寻求答案的问题。我们对有关图数据管理和处理基准的现有文献进行了深入研究，涵盖了过去 15 年中开发的 20 个不同的基准。我们将基准分为三个领域，重点是图处理系统的基准、图数据库基准和具有图处理工作负载的大数据基准。这种系统的方法使我们能够识别该领域存在的多个问题，包括 i) 很少存在可以产生高工作负载场景的基准，ii) 在基准图流处理和基于图的机器学习方面没有做重大工作，iii) 尽管新的有意义的指标已经存在多年，但基准往往使用传统指标，iv) 越来越多的大数据基准出现在图形处理工作负载中。根据这些观察，我们通过描述未来图数据系统基准测试研究的关键挑战来结束调查。