当前位置: X-MOL 学术Neural Netw. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Non-differentiable saddle points and sub-optimal local minima exist for deep ReLU networks
Neural Networks ( IF 7.8 ) Pub Date : 2021-08-14 , DOI: 10.1016/j.neunet.2021.08.005
Bo Liu 1 , Zhaoying Liu 1 , Ting Zhang 1 , Tongtong Yuan 1
Affiliation  

Whether sub-optimal local minima and saddle points exist in the highly non-convex loss landscape of deep neural networks has a great impact on the performance of optimization algorithms. Theoretically, we study in this paper the existence of non-differentiable sub-optimal local minima and saddle points for deep ReLU networks with arbitrary depth. We prove that there always exist non-differentiable saddle points in the loss surface of deep ReLU networks with squared loss or cross-entropy loss under reasonable assumptions. We also prove that deep ReLU networks with cross-entropy loss will have non-differentiable sub-optimal local minima if some outermost samples do not belong to a certain class. Experimental results on real and synthetic datasets verify our theoretical findings.



中文翻译:

深度 ReLU 网络存在不可微的鞍点和次优局部最小值

在深度神经网络的高度非凸损失景观中是否存在次优局部最小值和鞍点,对优化算法的性能有很大影响。从理论上讲,我们在本文中研究了具有任意深度的深度 ReLU 网络的不可微次优局部最小值和鞍点的存在。我们证明,在合理假设下,具有平方损失或交叉熵损失的深度 ReLU 网络的损失表面中始终存在不可微的鞍点。我们还证明,如果某些最外层样本不属于某个类别,则具有交叉熵损失的深度 ReLU 网络将具有不可微的次优局部最小值。在真实和合成数据集上的实验结果验证了我们的理论发现。

更新日期:2021-08-26
down
wechat
bug