Deep learning has revolutionized the fields of computer vision, natural language understanding, speech recognition, information retrieval, and more. However, with the progressive improvements in deep learning models, their number of parameters, latency, and resources required to train, among others, have all increased significantly. Consequently, it has become important to pay attention to these footprint metrics of a model as well, not just its quality. We present and motivate the problem of efficiency in deep learning, followed by a thorough survey of the five core areas of model efficiency (spanning modeling techniques, infrastructure, and hardware) and the seminal work there. We also present an experiment-based guide along with code for practitioners to optimize their model training and deployment. We believe this is the first comprehensive survey in the efficient deep learning space that covers the landscape of model efficiency from modeling techniques to hardware support. It is our hope that this survey would provide readers with the mental model and the necessary understanding of the field to apply generic efficiency techniques to immediately get significant improvements, and also equip them with ideas for further research and experimentation to achieve additional gains.

深度学习彻底改变了计算机视觉、自然语言理解、语音识别、信息检索等领域。然而，随着深度学习模型的逐步改进，它们的参数数量、延迟和训练所需的资源等都在显着增加。因此，关注模型的这些足迹指标也变得很重要，而不仅仅是它的质量。我们提出并激发了深度学习的效率问题，然后对模型效率的五个核心领域（跨越建模技术、基础设施和硬件）及其开创性工作进行了全面调查。我们还提供了一个基于实验的指南以及代码，供从业者优化他们的模型训练和部署。我们相信这是对高效深度学习领域的首次全面调查，涵盖了从建模技术到硬件支持的模型效率领域。我们希望这项调查能为读者提供心智模型和对该领域的必要理解，以应用通用效率技术立即获得显着改进，并为他们提供进一步研究和实验以取得额外收益的想法。