Improving the performance of deep neural networks (DNNs) is important to both the compiler and neural architecture search (NAS) communities. Compilers apply program transformations in order to exploit hardware parallelism and memory hierarchy. However, legality concerns mean they fail to exploit the natural robustness of neural networks. In contrast, NAS techniques mutate networks by operations such as the grouping or bottlenecking of convolutions, exploiting the resilience of DNNs. In this work, we express such neural architecture operations as program transformations whose legality depends on a notion of representational capacity. This allows them to be combined with existing transformations into a unified optimization framework. This unification allows us to express existing NAS operations as combinations of simpler transformations. Crucially, it allows us to generate and explore new tensor convolutions. We prototyped the combined framework in TVM and were able to find optimizations across different DNNs, that significantly reduce inference time - over 3$\times$ in the majority of cases. Furthermore, our scheme dramatically reduces NAS search time. Code is available at~\href{https://github.com/jack-willturner/nas-as-program-transformation-exploration}{this https url}.

中文翻译：

---

神经架构搜索作为程序转换探索

改善深度神经网络（DNN）的性能对于编译器和神经体系结构搜索（NAS）社区都很重要。编译器应用程序转换以利用硬件并行性和内存层次结构。但是，合法性问题意味着他们无法利用神经网络的自然鲁棒性。相反，NAS技术利用卷积的分组或瓶颈之类的操作来改变网络，从而利用DNN的弹性。在这项工作中，我们将这种神经体系结构操作表示为程序转换，其合法性取决于表示能力的概念。这使它们可以与现有的转换组合成一个统一的优化框架。这种统一使我们可以将现有的NAS操作表示为更简单的转换的组合。至关重要的是 它使我们能够生成和探索新的张量卷积。我们在TVM中对组合框架进行了原型设计，并能够在不同的DNN中找到优化方案，从而显着减少了推理时间-在大多数情况下，超过3倍。此外，我们的方案大大减少了NAS搜索时间。可以在〜\ href {https://github.com/jack-willturner/nas-as-program-transformation-exploration} {此https url}处找到代码。