Weight initialization is critical in being able to successfully train artificial neural networks (ANNs), and even more so for recurrent neural networks (RNNs) which can easily suffer from vanishing and exploding gradients. In neuroevolution, where evolutionary algorithms are applied to neural architecture search, weights typically need to be initialized at three different times: when initial genomes (ANN architectures) are created at the beginning of the search, when offspring genomes are generated by crossover, and when new nodes or edges are created during mutation. This work explores the difference between using Xavier, Kaiming, and uniform random weight initialization methods, as well as novel Lamarckian weight inheritance methods for initializing new weights during crossover and mutation operations. These are examined using the Evolutionary eXploration of Augmenting Memory Models (EXAMM) neuroevolution algorithm, which is capable of evolving RNNs with a variety of modern memory cells (e.g., LSTM, GRU, MGU, UGRNN and Delta-RNN cells) as well recurrent connections with varying time skips through a high performance island based distributed evolutionary algorithm. Results show that with statistical significance, utilizing the Lamarckian strategies outperforms Kaiming, Xavier and uniform random weight initialization, and can speed neuroevolution by requiring less backpropagation epochs to be evaluated for each generated RNN.

中文翻译：

---

权重初始化和权重继承对神经进化影响的实验研究

权重初始化对于成功训练人工神经网络 (ANN) 至关重要，对于容易遭受梯度消失和爆炸的循环神经网络 (RNN) 而言更是如此。在神经进化中，将进化算法应用于神经架构搜索，通常需要在三个不同的时间初始化权重：在搜索开始时创建初始基因组（ANN 架构）时，当通过交叉生成后代基因组时，以及当在变异期间创建新的节点或边。这项工作探讨了使用 Xavier、Kaiming 和均匀随机权重初始化方法之间的区别，以及在交叉和变异操作期间初始化新权重的新型拉马克权重继承方法。这些是使用增强记忆模型的进化探索 (EXAMM) 神经进化算法进行检查的，该算法能够使用各种现代记忆细胞（例​​如 LSTM、GRU、MGU、UGRNN 和 Delta-RNN 细胞）以及循环连接来进化 RNN随着时间的变化，跳过基于高性能孤岛的分布式进化算法。结果表明，具有统计学意义，利用 Lamarckian 策略优于 Kaiming、Xavier 和统一随机权重初始化，并且可以通过为每个生成的 RNN 评估更少的反向传播时期来加速神经进化。UGRNN 和 Delta-RNN 单元）以及具有不同时间的循环连接跳过基于高性能孤岛的分布式进化算法。结果表明，具有统计学意义，利用 Lamarckian 策略优于 Kaiming、Xavier 和统一随机权重初始化，并且可以通过为每个生成的 RNN 评估更少的反向传播时期来加速神经进化。UGRNN 和 Delta-RNN 单元）以及具有不同时间的循环连接跳过基于高性能孤岛的分布式进化算法。结果表明，具有统计学意义，利用 Lamarckian 策略优于 Kaiming、Xavier 和统一随机权重初始化，并且可以通过为每个生成的 RNN 评估更少的反向传播时期来加速神经进化。