The tremendous impact of deep learning algorithms over a wide range of application domains has encouraged a surge of neural network (NN) accelerator research. Facilitating the NN accelerator design calls for guidance from an evolving benchmark suite that incorporates emerging NN models. Nevertheless, existing NN benchmarks are not suitable for guiding NN accelerator designs. These benchmarks are either selected for general-purpose processors without considering unique characteristics of NN accelerators or lack quantitative analysis to guarantee their completeness during the benchmark construction, update, and customization. In light of the shortcomings of prior benchmarks, we propose a novel benchmarking methodology for NN accelerators with a quantitative analysis of application performance features and a comprehensive awareness of software-hardware co-design. Specifically, we decouple the benchmarking process into three stages: First, we characterize the NN workloads with quantitative metrics and select the representative applications for the benchmark suite to ensure diversity and completeness. Second, we refine the selected applications according to the customized model compression techniques provided by specific software-hardware co-design. Finally, we evaluate a variety of accelerator designs on the generated benchmark suite. To demonstrate the effectiveness of our benchmarking methodology, we conduct a case study of composing an NN benchmark from the TensorFlow Model Zoo and compress these selected models with various model compression techniques. Finally, we evaluate compressed models on various architectures, including GPU, Neurocube, DianNao, and Cambricon-X.

中文翻译：

---

NNBench-X

深度学习算法对广泛应用领域的巨大影响促进了神经网络 (NN) 加速器研究的激增。促进 NN 加速器设计需要来自不断发展的基准套件的指导，该套件包含新兴的 NN 模型。然而，现有的 NN 基准并不适合指导 NN 加速器设计。这些基准要么是为通用处理器选择的，而不考虑 NN 加速器的独特特性，要么在基准构建、更新和定制过程中缺乏定量分析来保证它们的完整性。鉴于先前基准的缺点，我们为 NN 加速器提出了一种新颖的基准测试方法，对应用程序性能特征进行定量分析，并全面了解软硬件协同设计。具体来说，我们将基准测试过程解耦为三个阶段：首先，我们用量化指标来描述 NN 工作负载，并为基准套件选择具有代表性的应用程序，以确保多样性和完整性。其次，我们根据特定软硬件协同设计提供的定制模型压缩技术对选定的应用程序进行细化。最后，我们在生成的基准套件上评估各种加速器设计。为了证明我们的基准测试方法的有效性，我们进行了一个案例研究，从 TensorFlow Model Zoo 中编写一个 NN 基准，并使用各种模型压缩技术压缩这些选定的模型。最后，我们评估了各种架构上的压缩模型，包括 GPU、Neurocube、DianNao 和 Cambricon-X。