当前位置: X-MOL 学术Pattern Recogn. Lett. › 论文详情
If dropout limits trainable depth, does critical initialisation still matter? A large-scale statistical analysis on ReLU networks
Pattern Recognition Letters ( IF 3.255 ) Pub Date : 2020-06-28 , DOI: 10.1016/j.patrec.2020.06.025
Arnu Pretorius; Elan van Biljon; Benjamin van Niekerk; Ryan Eloff; Matthew Reynard; Steve James; Benjamin Rosman; Herman Kamper; Steve Kroon

Recent work in signal propagation theory has shown that dropout limits the depth to which information can propagate through a neural network. In this paper, we investigate the effect of initialisation on training speed and generalisation for ReLU networks within this depth limit. We ask the following research question: given that critical initialisation is crucial for training at large depth, if dropout limits the depth at which networks are trainable, does initialising critically still matter? We conduct a large-scale controlled experiment, and perform a statistical analysis of over 12 000 trained networks. We find that (1) trainable networks show no statistically significant difference in performance over a wide range of non-critical initialisations; (2) for initialisations that show a statistically significant difference, the net effect on performance is small; (3) only extreme initialisations (very small or very large) perform worse than criticality. These findings also apply to standard ReLU networks of moderate depth as a special case of zero dropout. Our results therefore suggest that, in the shallow-to-moderate depth setting, critical initialisation provides zero performance gains when compared to off-critical initialisations and that searching for off-critical initialisations that might improve training speed or generalisation, is likely to be a fruitless endeavour.
更新日期:2020-06-28

 

全部期刊列表>>
材料学研究精选
Springer Nature Live 产业与创新线上学术论坛
胸腔和胸部成像专题
自然科研论文编辑服务
ACS ES&T Engineering
ACS ES&T Water
屿渡论文,编辑服务
杨超勇
周一歌
华东师范大学
南京工业大学
清华大学
中科大
唐勇
跟Nature、Science文章学绘图
隐藏1h前已浏览文章
中洪博元
课题组网站
新版X-MOL期刊搜索和高级搜索功能介绍
ACS材料视界
x-mol收录
福州大学
南京大学
王杰
左智伟
湖南大学
清华大学
吴杰
赵延川
中山大学化学工程与技术学院
试剂库存
天合科研
down
wechat
bug