Recent advances in deep learning have been driven by everincreasing model sizes, with networks growing to millions or even billions of parameters. Such enormous models call for fast and energy-efficient hardware accelerators. We study the potential of Analog AI accelerators based on Non-Volatile Memory, in particular Phase Change Memory, for iso-accurate inference of natural language processing applications. We demonstrate a path to iso-accuracy for the GLUE benchmark on BERT (Bidirectional Encoder Representations from Transformers), by combining noise-aware training to combat inherent PCM drift and noise sources, together with reduced-precision digital attention-block computation down to INT6.

中文翻译：

---

在具有模拟存储设备的基于变压器的深度神经网络上实现等精度

不断增长的模型规模推动了深度学习的最新进展，网络规模增长到数百万甚至数十亿。如此庞大的模型需要快速且节能的硬件加速器。我们研究了基于非易失性存储器（尤其是相变存储器）的模拟AI加速器对自然语言处理应用程序进行等精度推断的潜力。我们通过结合噪声感知训练来对抗固有的PCM漂移和噪声源以及降低精度的数字注意块计算（直至INT6），证明了BERT（来自变压器的双向编码器表示）上GLUE基准测试的等精度途径。 。