While deep learning has achieved phenomenal successes in many AI applications, its enormous model size and intensive computation requirements pose a formidable challenge to the deployment in resource-limited nodes. There has recently been an increasing interest in computationally-efficient learning methods, e.g., quantization, pruning and channel gating. However, most existing techniques cannot adapt to different tasks quickly. In this work, we advocate a holistic approach to jointly train the backbone network and the channel gating which enables dynamical selection of a subset of filters for more efficient local computation given the data input. Particularly, we develop a federated meta-learning approach to jointly learn good meta-initializations for both backbone networks and gating modules, by making use of the model similarity across learning tasks on different nodes. In this way, the learnt meta-gating module effectively captures the important filters of a good meta-backbone network, based on which a task-specific conditional channel gated network can be quickly adapted, i.e., through one-step gradient descent, from the meta-initializations in a two-stage procedure using new samples of that task. The convergence of the proposed federated meta-learning algorithm is established under mild conditions. Experimental results corroborate the effectiveness of our method in comparison to related work.

中文翻译：

---

MetaGater：通过联合元学习快速学习条件信道门控网络

尽管深度学习已在许多AI应用程序中取得了惊人的成功，但其巨大的模型规模和密集的计算要求对资源有限的节点中的部署提出了巨大的挑战。最近，人们对计算效率高的学习方法（例如，量化，修剪和通道门控）越来越感兴趣。但是，大多数现有技术无法迅速适应不同的任务。在这项工作中，我们提倡一种整体方法来联合训练骨干网和信道选通，这可以动态选择过滤器的子集，以便在给定数据输入的情况下更有效地进行本地计算。特别是，我们开发了一种联合元学习方法，以便共同为骨干网和门控模块学习良好的元初始化，通过利用跨不同节点上的学习任务的模型相似性。以这种方式，学习的元门控模块有效地捕获了良好的元骨干网的重要过滤器，基于这些过滤器，可以快速地（即，通过一步梯度下降）从网络中适应特定任务的条件信道门控网络。使用该任务的新样本在两阶段过程中进行元初始化。在温和条件下建立了所提出的联合元学习算法的收敛性。与相关工作相比，实验结果证实了我们方法的有效性。通过两步过程中的元初始化，使用该任务的新样本，通过一步式梯度下降。在温和条件下建立了所提出的联合元学习算法的收敛性。与相关工作相比，实验结果证实了我们方法的有效性。通过两步过程中的元初始化，使用该任务的新样本，通过一步式梯度下降。在温和条件下建立了所提出的联合元学习算法的收敛性。与相关工作相比，实验结果证实了我们方法的有效性。