Neural networks can approximate complex functions, but they struggle to perform exact arithmetic operations over real numbers. The lack of inductive bias for arithmetic operations leaves neural networks without the underlying logic necessary to extrapolate on tasks such as addition, subtraction, and multiplication. We present two new neural network components: the Neural Addition Unit (NAU), which can learn exact addition and subtraction; and the Neural Multiplication Unit (NMU) that can multiply subsets of a vector. The NMU is, to our knowledge, the first arithmetic neural network component that can learn to multiply elements from a vector, when the hidden size is large. The two new components draw inspiration from a theoretical analysis of recently proposed arithmetic components. We find that careful initialization, restricting parameter space, and regularizing for sparsity is important when optimizing the NAU and NMU. Our proposed units NAU and NMU, compared with previous neural units, converge more consistently, have fewer parameters, learn faster, can converge for larger hidden sizes, obtain sparse and meaningful weights, and can extrapolate to negative and small values.

中文翻译：

---

神经算术单元

神经网络可以逼近复杂的函数，但它们很难对实数执行精确的算术运算。算术运算缺乏归纳偏差，使得神经网络没有必要的底层逻辑来推断诸如加法、减法和乘法之类的任务。我们提出了两个新的神经网络组件：神经加法单元 (NAU)，它可以学习精确的加法和减法；以及可以乘以向量子集的神经乘法单元 (NMU)。据我们所知，NMU 是第一个算术神经网络组件，当隐藏大小很大时，它可以学习从向量中乘以元素。这两个新组件的灵感来自对最近提出的算术组件的理论分析。我们发现仔细初始化，限制参数空间，在优化 NAU 和 NMU 时，对稀疏进行正则化很重要。与之前的神经单元相比，我们提出的单元 NAU 和 NMU 收敛更一致，参数更少，学习速度更快，可以收敛更大的隐藏尺寸，获得稀疏和有意义的权重，并且可以外推到负值和小值。