While designing inductive bias in neural architectures has been widely studied, we hypothesize that transformer networks are flexible enough to learn inductive bias from suitable generic tasks. Here, we replace architecture engineering by encoding inductive bias in the form of datasets. Inspired by Peirce's view that deduction, induction, and abduction form an irreducible set of reasoning primitives, we design three synthetic tasks that are intended to require the model to have these three abilities. We specifically design these synthetic tasks in a way that they are devoid of mathematical knowledge to ensure that only the fundamental reasoning biases can be learned from these tasks. This defines a new pre-training methodology called "LIME" (Learning Inductive bias for Mathematical rEasoning). Models trained with LIME significantly outperform vanilla transformers on three very different large mathematical reasoning benchmarks. Unlike dominating the computation cost as traditional pre-training approaches, LIME requires only a small fraction of the computation cost of the typical downstream task.

中文翻译：

---

LIME：学习归纳偏见的数学推理基元

尽管在神经体系结构中设计归纳偏置已得到广泛研究，但我们假设变压器网络足够灵活，可以从合适的通用任务中学习归纳偏置。在这里，我们通过以数据集的形式编码归纳偏差来代替体系结构工程。受皮尔士认为演绎，归纳和绑架构成不可还原的推理原语集合的启发，我们设计了三个综合任务，旨在要求模型具有这三个能力。我们以没有数学知识的方式专门设计了这些合成任务，以确保只能从这些任务中学习基本的推理偏见。这定义了一种新的预训练方法，称为“ LIME”（学习数学推理的归纳偏差）。在三个非常不同的大型数学推理基准上，使用LIME训练的模型明显优于香草变压器。与将计算成本控制为传统的预训练方法不同，LIME仅需要典型下游任务的计算成本的一小部分。