This paper presents a novel “Distributed Deep Learning Framework” for a heterogeneous multi-GPU cluster that can effectively improve overall resource utilization without sacrificing training accuracy. Specifically, we employ a hybrid aggregation approach using a parameter-server and all-reduce schemes in order to address potential performance degradation problems in running deep learning applications on a heterogeneous computing system. In addition, we design and implement an asynchronous large mini-batch training mechanism to maintain training accuracy for asynchronous data-paralleled deep learning processing with enhanced collective communication capability based on MPI. We successfully implement our proposed framework on TensorFlow and perform extensive experiments in both of homogeneous and heterogeneous computing systems. Evaluation results show that our proposed framework can improve computing performance by decreasing I/O bottlenecks, and effectively increasing the resource utilization in the heterogeneous multi-GPU cluster.

本文针对异质性提出了一种新颖的“分布式深度学习框架”多GPU群集，可以在不牺牲训练准确性的情况下有效提高整体资源利用率。具体来说，我们采用使用参数服务器和全约简方案的混合聚合方法，以解决在异构计算系统上运行深度学习应用程序时潜在的性能下降问题。此外，我们设计并实现了一个异步大型微型批处理训练机制，以维持基于MPI的增强的集体通信能力的异步数据并行深度学习处理的训练精度。我们成功地在TensorFlow上实现了我们提出的框架，并在同构和异构计算系统中进行了广泛的实验。