Deep learning (DL) is transforming industry as decision-making processes are being automated by deep neural networks (DNNs) trained on real-world data. Driven partly by rapidly-expanding literature on DNN approximation theory showing they can approximate a rich variety of functions, such tools are increasingly being considered for problems in scientific computing. Yet, unlike traditional algorithms in this field, little is known about DNNs from the principles of numerical analysis, e.g., stability, accuracy, computational efficiency and sample complexity. In this paper we introduce a computational framework for examining DNNs in practice, and use it to study empirical performance with regard to these issues. We study performance of DNNs of different widths & depths on test functions in various dimensions, including smooth and piecewise smooth functions. We also compare DL against best-in-class methods for smooth function approx. based on compressed sensing (CS). Our main conclusion from these experiments is that there is a crucial gap between the approximation theory of DNNs and their practical performance, with trained DNNs performing relatively poorly on functions for which there are strong approximation results (e.g. smooth functions), yet performing well in comparison to best-in-class methods for other functions. To analyze this gap further, we provide some theoretical insights. We establish a practical existence theorem, asserting existence of a DNN architecture and training procedure that offers the same performance as CS. This establishes a key theoretical benchmark, showing the gap can be closed, albeit via a strategy guaranteed to perform as well as, but no better than, current best-in-class schemes. Nevertheless, it demonstrates the promise of practical DNN approx., by highlighting potential for better schemes through careful design of DNN architectures and training strategies.

中文翻译：

---

深度神经网络函数逼近理论与实践的差距

深度学习 (DL) 正在改变行业，因为决策过程由基于真实世界数据训练的深度神经网络 (DNN) 实现自动化。部分由于 DNN 逼近理论的快速扩展文献表明它们可以逼近各种函数，这些工具越来越多地被考虑用于解决科学计算中的问题。然而，与该领域的传统算法不同，从数值分析的原理，例如稳定性、准确性、计算效率和样本复杂性，我们对 DNN 知之甚少。在本文中，我们介绍了一种用于在实践中检查 DNN 的计算框架，并使用它来研究有关这些问题的经验表现。我们研究了不同宽度和深度的 DNN 在不同维度的测试函数上的性能，包括平滑和分段平滑函数。我们还将 DL 与同类最佳方法的平滑函数进行了比较。基于压缩感知（CS）。我们从这些实验中得出的主要结论是，DNN 的逼近理论与其实际性能之间存在重大差距，经过训练的 DNN 在具有强逼近结果的函数（例如平滑函数）上表现相对较差，但相比之下表现良好到其他功能的一流方法。为了进一步分析这一差距，我们提供了一些理论见解。我们建立了一个实用的存在定理，断言 DNN 架构和训练过程的存在提供了与 CS 相同的性能。这建立了一个关键的理论基准，表明差距可以缩小，尽管通过一种策略保证其性能与当前同类最佳方案一样好，但不比当前最佳方案更好。尽管如此，它通过精心设计 DNN 架构和训练策略突出了更好方案的潜力，从而证明了实用 DNN 的前景。