Deep Learning (DL) has achieved remarkable progress over the last decade on various tasks such as image recognition, speech recognition, and natural language processing. In general, three main crucial aspects fueled this progress: the increasing availability of large amount of digitized data, the increasing availability of affordable parallel and powerful computing resources (e.g., GPU) and the growing number of open source deep learning frameworks that facilitate and ease the development process of deep learning architectures. In practice, the increasing popularity of deep learning frameworks calls for benchmarking studies that can effectively evaluate and understand the performance characteristics of these systems. In this paper, we conduct an extensive experimental evaluation and analysis of six popular deep learning frameworks, namely, TensorFlow, MXNet, PyTorch, Theano, Chainer, and Keras, using three types of DL architectures Convolutional Neural Networks (CNN), Faster Region-based Convolutional Neural Networks (Faster R-CNN), and Long Short Term Memory (LSTM). Our experimental evaluation considers different aspects for its comparison including accuracy, training time, convergence and resource consumption patterns. Our experiments have been conducted on both CPU and GPU environments using different datasets. We report and analyze the performance characteristics of the studied frameworks. In addition, we report a set of insights and important lessons that we have learned from conducting our experiments.

在过去的十年中，深度学习（DL）在各种任务（例如图像识别，语音识别和自然语言处理）上取得了显着进步。一般而言，三个主要关键方面推动了这一进展：大量数字化数据的可用性不断提高，可负担的并行和强大的计算资源（例如GPU）的可用性不断提高，以及促进和简化了开源深度学习框架的数量不断增长深度学习架构的开发过程。在实践中，深度学习框架的日益普及要求进行基准测试，以有效评估和理解这些系统的性能特征。在本文中，我们对六个流行的深度学习框架进行了广泛的实验评估和分析，即TensorFlow，MXNet，PyTorch，Theano，Chainer和Keras，使用三种类型的DL架构卷积神经网络（CNN）的，更快的基于区域的卷积神经网络（更快的R-CNN）和长短期记忆（LSTM）。我们的实验评估从不同方面进行比较，包括准确性，训练时间，收敛性和资源消耗模式。我们的实验是在CPU和GPU环境下使用不同的数据集进行的。我们报告和分析所研究框架的性能特征。此外，我们报告了从进行实验中学到的一系列见解和重要教训。