Hardware for deep neural network (DNN) inference often suffers from insufficient on-chip memory, thus requiring accesses to separate memory-only chips. Such off-chip memory accesses incur considerable costs in terms of energy and execution time. Fitting entire DNNs in on-chip memory is challenging due, in particular, to the physical size of the technology. Here, we report a DNN inference system—termed Illusion—that consists of networked computing chips, each of which contains a certain minimal amount of local on-chip memory and mechanisms for quick wakeup and shutdown. An eight-chip Illusion system hardware achieves energy and execution times within 3.5% and 2.5%, respectively, of an ideal single chip with no off-chip memory. Illusion is flexible and configurable, achieving near-ideal energy and execution times for a wide variety of DNN types and sizes. Our approach is tailored for on-chip non-volatile memory with resilience to permanent write failures, but is applicable to several memory technologies. Detailed simulations also show that our hardware results could be scaled to 64-chip Illusion systems.

用于深度神经网络 (DNN) 推理的硬件通常存在片上内存不足的问题，因此需要访问单独的仅内存芯片。这种片外存储器访问在能量和执行时间方面产生了相当大的成本。由于技术的物理尺寸，将整个 DNN 安装在片上存储器中具有挑战性。在这里，我们报告了一个称为 Illusion 的 DNN 推理系统，它由网络计算芯片组成，每个芯片都包含一定数量的本地片上内存和快速唤醒和关闭机制。八芯片 Illusion 系统硬件的能耗和执行时间分别比没有片外存储器的理想单芯片低 3.5% 和 2.5%。幻觉是灵活和可配置的，为各种 DNN 类型和大小实现近乎理想的能量和执行时间。我们的方法是为片上非易失性存储器量身定制的，具有对永久写入故障的恢复能力，但适用于多种存储器技术。详细的模拟还表明，我们的硬件结果可以扩展到 64 芯片的 Illusion 系统。