当前位置: X-MOL 学术IEEE Trans. Pattern Anal. Mach. Intell. › 论文详情
Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)
Optimization Induced Equilibrium Networks: An Explicit Optimization Perspective for Understanding Equilibrium Models
IEEE Transactions on Pattern Analysis and Machine Intelligence ( IF 23.6 ) Pub Date : 2022-06-10 , DOI: 10.1109/tpami.2022.3181425
Xingyu Xie 1 , Qiuhao Wang 1 , Zenan Ling 1 , Xia Li 2 , Guangcan Liu 3 , Zhouchen Lin 1
Affiliation  

To reveal the mystery behind deep neural networks (DNNs), optimization may offer a good perspective. There are already some clues showing the strong connection between DNNs and optimization problems, e.g., under a mild condition, DNN's activation function is indeed a proximal operator. In this paper, we are committed to providing a unified optimization induced interpretability for a special class of networks—equilibrium models, i.e., neural networks defined by fixed point equations, which have become increasingly attractive recently. To this end, we first decompose DNNs into a new class of unit layer that is the proximal operator of an implicit convex function while keeping its output unchanged. Then, the equilibrium model of the unit layer can be derived, we name it Optimization Induced Equilibrium Networks (OptEq). The equilibrium point of OptEq can be theoretically connected to the solution of a convex optimization problem with explicit objectives. Based on this, we can flexibly introduce prior properties to the equilibrium points: 1) modifying the underlying convex problems explicitly so as to change the architectures of OptEq; and 2) merging the information into the fixed point iteration, which guarantees to choose the desired equilibrium point when the fixed point set is non-singleton. We show that OptEq outperforms previous implicit models even with fewer parameters.

中文翻译:

优化诱导平衡网络:理解平衡模型的显式优化视角

为了揭示深度神经网络 (DNN) 背后的奥秘,优化可能提供了一个很好的视角。已经有一些线索表明 DNN 与优化问题之间存在很强的联系,例如,在温和的条件下,DNN 的激活函数确实是一个近端算子。在本文中,我们致力于为一类特殊的网络提供统一的优化诱导可解释性——平衡模型,即由不动点方程定义的神经网络,最近变得越来越有吸引力。为此,我们首先将 DNN 分解为一类新的单元层,它是隐式凸函数的近端算子,同时保持其输出不变。然后,可以导出单元层的均衡模型,我们将其命名为优化诱导均衡网络(OptEq)。OptEq 的平衡点在理论上可以连接到具有显式目标的凸优化问题的解。基于此,我们可以灵活地将先验属性引入平衡点:1)显式修改底层凸问题,从而改变OptEq的体系结构;2) 将信息合并到不动点迭代中,保证在不动点集非单点时选择期望的平衡点。我们证明 OptEq 即使参数更少也优于以前的隐式模型。1) 显式修改底层凸问题,从而改变 OptEq 的架构;2) 将信息合并到不动点迭代中,保证在不动点集非单点时选择期望的平衡点。我们证明 OptEq 即使参数更少也优于以前的隐式模型。1) 显式修改底层凸问题,从而改变 OptEq 的架构;2) 将信息合并到不动点迭代中,保证在不动点集非单点时选择期望的平衡点。我们证明 OptEq 即使参数更少也优于以前的隐式模型。
更新日期:2022-06-10
down
wechat
bug