Deep learning recommendation systems at scale have provided remarkable gains through increasing model capacity (i.e. wider and deeper neural networks), but it comes at significant training cost and infrastructure cost. Model pruning is an effective technique to reduce computation overhead for deep neural networks by removing redundant parameters. However, modern recommendation systems are still thirsty for model capacity due to the demand for handling big data. Thus, pruning a recommendation model at scale results in a smaller model capacity and consequently lower accuracy. To reduce computation cost without sacrificing model capacity, we propose a dynamic training scheme, namely alternate model growth and pruning, to alternatively construct and prune weights in the course of training. Our method leverages structured sparsification to reduce computational cost without hurting the model capacity at the end of offline training so that a full-size model is available in the recurring training stage to learn new data in real-time. To the best of our knowledge, this is the first work to provide in-depth experiments and discussion of applying structural dynamics to recommendation systems at scale to reduce training cost. The proposed method is validated with an open-source deep-learning recommendation model (DLRM) and state-of-the-art industrial-scale production models.

中文翻译：

---

替代模型的增长和修剪，以对推荐系统进行有效的培训

大规模的深度学习推荐系统通过增加模型容量（即更广泛和更深入的神经网络）提供了可观的收益，但它付出了可观的培训成本和基础设施成本。模型修剪是一种有效的技术，可通过删除冗余参数来减少深度神经网络的计算开销。但是，由于对处理大数据的需求，现代推荐系统仍然渴望模型容量。因此，按比例修剪推荐模型会导致模型容量变小，因此准确性降低。为了在不牺牲模型容量的情况下降低计算成本，我们提出了一种动态训练方案，即交替模型增长和修剪，以在训练过程中交替构造和修剪权重。我们的方法利用结构化的稀疏化来降低计算成本，而又不会在离线训练结束时损害模型的能力，因此在重复训练阶段就可以使用完整大小的模型来实时学习新数据。据我们所知，这是首次提供深入的实验和讨论，将结构动力学大规模应用于推荐系统以降低培训成本。所提出的方法已通过开源深度学习推荐模型（DLRM）和最新的工业规模生产模型进行了验证。这是提供深入实验和将结构动力学大规模应用于推荐系统以减少培训成本的讨论的第一项工作。所提出的方法已通过开源深度学习推荐模型（DLRM）和最新的工业规模生产模型进行了验证。这是提供深入实验和将结构动力学大规模应用于推荐系统以减少培训成本的讨论的第一项工作。所提出的方法已通过开源深度学习推荐模型（DLRM）和最新的工业规模生产模型进行了验证。