Theory of Gating in Recurrent Neural Networks,Physical Review X

当前位置： X-MOL 学术 › Phys. Rev. X › 论文详情

Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)

Theory of Gating in Recurrent Neural Networks
Physical Review X ( IF 11.6 ) Pub Date : 2022-01-18 , DOI: 10.1103/physrevx.12.011011
Kamesh Krishnamurthy ₁ , Tankut Can ₂ , David J Schwab ₃

Affiliation

Recurrent neural networks (RNNs) are powerful dynamical models, widely used in machine learning (ML) and neuroscience. Prior theoretical work has focused on RNNs with additive interactions. However, gating, i.e., multiplicative, interactions are ubiquitous in real neurons and also the central feature of the best-performing RNNs in ML. Here, we show that gating offers flexible control of two salient features of the collective dynamics: (i) timescales and (ii) dimensionality. The gate controlling timescales leads to a novel, marginally stable state, where the network functions as a flexible integrator. Unlike previous approaches, gating permits this important function without parameter fine-tuning or special symmetries. Gates also provide a flexible, context-dependent mechanism to reset the memory trace, thus complementing the memory function. The gate modulating the dimensionality can induce a novel, discontinuous chaotic transition, where inputs push a stable system to strong chaotic activity, in contrast to the typically stabilizing effect of inputs. At this transition, unlike additive RNNs, the proliferation of critical points (topological complexity) is decoupled from the appearance of chaotic dynamics (dynamical complexity). The rich dynamics are summarized in phase diagrams, thus providing a map for principled parameter initialization choices to ML practitioners.

中文翻译：

循环神经网络中的门控理论

递归神经网络 (RNN) 是强大的动力学模型，广泛用于机器学习 (ML) 和神经科学。先前的理论工作主要集中在具有加性交互的 RNN 上。然而，门控，即乘法相互作用在真实神经元中无处不在，而且ML 中表现最佳的 RNN 的核心特征。在这里，我们表明门控可以灵活控制集体动力学的两个显着特征：(i) 时间尺度和 (ii) 维度。门控制时间尺度导致一种新颖的、边缘稳定的状态，其中网络充当灵活的积分器。与以前的方法不同，门控允许在没有参数微调或特殊对称性的情况下实现这一重要功能。盖茨还提供了一种灵活的、上下文相关的机制来重置记忆轨迹，从而补充记忆功能。调节维度的门可以引发一种新颖的、不连续的混沌转变，在这种转变中，输入将稳定的系统推向强烈的混沌活动，这与输入的典型稳定作用形成鲜明对比。在这种转变中，与加性 RNN 不同，临界点的扩散（拓扑复杂性）与混沌动力学（动态复杂性）的出现是分离的。丰富的动态在相图中进行了总结，从而为 ML 从业者提供了原则性参数初始化选择的映射。

更新日期：2022-01-19

点击分享查看原文

点击收藏

公开下载

阅读更多本刊最新论文本刊介绍/投稿指南11