With the proliferation of deep convolutional neural network (CNN) algorithms for mobile processing, limited precision quantization has become an essential tool for CNN efficiency. Consequently, various works have sought to design fixed precision quantization algorithms and quantization-focused optimization techniques that minimize quantization induced performance degradation. However, there is little concrete understanding of how various CNN design decisions/best practices affect quantized inference behaviour. Weight initialization strategies are often associated with solving issues such as vanishing/exploding gradients but an often-overlooked aspect is their impact on the final trained distributions of each layer. We present an in-depth, fine-grained ablation study of the effect of different weights initializations on the final distributions of weights and activations of different CNN architectures. The fine-grained, layerwise analysis enables us to gain deep insights on how initial weights distributions will affect final accuracy and quantized behaviour. To our best knowledge, we are the first to perform such a low-level, in-depth quantitative analysis of weights initialization and its effect on quantized behaviour.

中文翻译：

---

我们应该从哪里开始？权重初始化对深度神经网络量化行为的影响的低级别探索

随着用于移动处理的深度卷积神经网络（CNN）算法的泛滥，有限的精度量化已成为提高CNN效率的重要工具。因此，已经进行了各种工作来设计固定精度量化算法和以量化为中心的优化技术，以最小化量化引起的性能下降。但是，对于各种CNN设计决策/最佳实践如何影响量化推理行为的了解很少。权重初始化策略通常与解决诸如消失/爆炸梯度之类的问题相关，但是经常被忽略的方面是它们对每一层最终训练分布的影响。我们深入介绍 不同权重初始化对权重最终分布和不同CNN架构激活的影响的细粒度消融研究。细粒度的分层分析使我们能够深入了解初始权重分布将如何影响最终准确性和量化行为。据我们所知，我们是第一个对权重初始化及其对量化行为的影响进行低级别，深入定量分析的人。