The activation functions used in an artificial neural network define how nodes of the network respond to input, directly influence the shape of the error surface and play a role in the difficulty of the neural network training problem. Choice of activation functions is a significant question which must be addressed when applying a neural network to a problem. One issue which must be considered when selecting an activation function is known as activation function saturation. Saturation occurs when a bounded activation function primarily outputs values close to its boundary. Excessive saturation damages the network’s ability to encode information and may prevent successful training. Common functions such as the logistic and hyperbolic tangent functions have been shown to exhibit saturation when the neural network is trained using particle swarm optimization. This study proposes a new measure of activation function saturation, evaluates the saturation behavior of eight common activation functions, and evaluates six measures of controlling activation function saturation in particle swarm optimization based neural network training. Activation functions that result in low levels of saturation are identified. For each activation function recommendations are made regarding which saturation control mechanism is most effective at reducing saturation.

人工神经网络中使用的激活函数定义了网络节点如何响应输入，直接影响错误表面的形状并在神经网络训练问题的难度中起作用。激活函数的选择是一个重要的问题，在将神经网络应用于问题时必须解决。选择激活函数时必须考虑的一个问题称为激活函数饱和。当有界激活函数主要输出接近其边界的值时，就会发生饱和。过度的饱和会破坏网络对信息进行编码的能力，并可能阻止成功的训练。当使用粒子群优化训练神经网络时，诸如逻辑和双曲正切函数之类的常用函数已显示出饱和。这项研究提出了一种新的激活函数饱和度度量，评估了八个常见激活函数的饱和行为，并评估了基于神经网络训练的粒子群优化中六个控制激活函数饱和度的度量。确定导致饱和度低的激活函数。对于每个激活功能，均提出了关于哪种饱和度控制机制最有效地降低饱和度的建议。并评估了基于神经网络训练的粒子群算法中控制激活函数饱和的六种措施。确定导致饱和度低的激活函数。对于每个激活功能，均提出了关于哪种饱和度控制机制最有效地降低饱和度的建议。并评估了基于神经网络训练的粒子群算法中控制激活函数饱和的六种措施。确定导致饱和度低的激活函数。对于每个激活功能，均提出了关于哪种饱和度控制机制最有效地降低饱和度的建议。