Deep neural networks (DNNs) have gained tremendous popularity in recent years due to their ability to achieve superhuman accuracy in a wide variety of machine learning tasks. However, the compute and memory requirements of DNNs have grown rapidly, creating a need for energy-efficient hardware. Resistive crossbars have attracted significant interest in the design of the next generation of DNN accelerators due to their ability to natively execute massively parallel vector-matrix multiplications within dense memory arrays. However, crossbar-based computations face a major challenge due to device and circuit-level nonidealities, which manifest as errors in the vector-matrix multiplications and eventually degrade DNN accuracy. To address this challenge, there is a need for tools that can model the functional impact of nonidealities on DNN training and inference. Existing efforts toward this goal are either limited to inference or are too slow to be used for large-scale DNN training. We propose TxSim, a fast and customizable modeling framework to functionally evaluate DNN training on crossbar-based hardware considering the impact of nonidealities. The key features of TxSim that differentiate it from prior efforts are: 1) it comprehensively models nonidealities during all training operations (forward propagation, backward propagation, and weight update) and 2) it achieves computational efficiency by mapping crossbar evaluations to well-optimized Basic Linear Algebra Subprograms (BLAS) routines and incorporates speedup techniques to further reduce simulation time with minimal impact on accuracy. TxSim achieves $6\times $ – $108\times $ improvement in simulation speed over prior works, and thereby makes it feasible to evaluate the training of large-scale DNNs on crossbars. Our experiments using TxSim reveal that the accuracy degradation in DNN training due to nonidealities can be substantial (3%–36.4%) for large-scale DNNs and data sets, underscoring the need for further research in mitigation techniques. We also analyze the impact of various device and circuit-level parameters and the associated nonidealities to provide key insights that can guide the design of crossbar-based DNN training accelerators.

中文翻译：

---

TxSim：电阻交叉开关系统上的深度神经网络建模培训

近年来，由于深度神经网络（DNN）在各种机器学习任务中都能实现超人精度，因此已经获得了极大的普及。但是，DNN的计算和内存需求增长迅速，因此需要高能效的硬件。电阻式交叉开关在下一代DNN加速器的设计中引起了极大的兴趣，因为它们能够在密集存储阵列中本地执行大规模并行矢量矩阵乘法。但是，基于交叉开关的计算由于设备和电路级的非理想性而面临着重大挑战，这在向量矩阵乘法中表现为错误，最终会降低DNN的准确性。为了应对这一挑战，需要一种可以对非理想性对DNN训练和推理的功能影响进行建模的工具。为实现此目标而进行的现有努力要么仅限于推理，要么太慢而无法用于大规模DNN训练。我们提出TxSim，这是一种快速且可自定义的建模框架，用于在考虑非理想性影响的情况下对基于交叉开关的硬件进行DNN培训进行功能评估。TxSim与以前的工作不同的关键特征是：1）在所有训练操作（前向传播，后向传播和权重更新）中对非理想性进行综合建模，以及2）通过将交叉开关评估映射到经过充分优化的基础上来实现计算效率线性代数子程序（BLAS）例程，并结合了加速技术，以进一步减少仿真时间，而对精度的影响最小。 $ 6 \次$ – $ 108 \次$ 与以前的工作相比，仿真速度有所提高，从而使评估交叉开关上大规模DNN的训练变得可行。我们使用TxSim进行的实验表明，对于大型DNN和数据集，由于非理想性导致DNN训练的准确性可能会大幅下降（3％–36.4％），从而强调了对缓解技术进行进一步研究的必要性。我们还分析了各种设备和电路级参数以及相关的非理想性的影响，以提供重要的见解，可以指导基于交叉开关的DNN训练加速器的设计。