The past decade has witnessed great progress in automatic speech recognition (ASR) due to advances in deep learning. The improvements in performance can be attributed to both improved models and large-scale training data. The key to training such models is the employment of efficient distributed learning techniques. In this article, we provide an overview of distributed training techniques for deep neural network (DNN) acoustic models used for ASR. Starting with the fundamentals of data parallel stochastic gradient descent (SGD) and ASR acoustic modeling, we investigate various distributed training strategies and their realizations in high-performance computing (HPC) environments with an emphasis on striking a balance between communication and computation. Experiments are carried out on a popular public benchmark to study the convergence, speedup, and recognition performance of the investigated strategies.

中文翻译：

---

自动语音识别深度神经网络声学模型的分布式训练：当前训练策略的比较


过去十年，由于深度学习的进步，自动语音识别（ASR）取得了巨大进步。性能的提高可归因于模型的改进和大规模训练数据。训练此类模型的关键是采用高效的分布式学习技术。在本文中，我们概述了用于 ASR 的深度神经网络 (DNN) 声学模型的分布式训练技术。从数据并行随机梯度下降 (SGD) 和 ASR 声学建模的基础知识开始，我们研究了各种分布式训练策略及其在高性能计算 (HPC) 环境中的实现，重点是在通信和计算之间取得平衡。在流行的公共基准上进行了实验，以研究所研究策略的收敛性、加速性和识别性能。