We propose a novel family of connectionist models based on kernel machines and consider the problem of learning layer by layer a compositional hypothesis class (i.e., a feedforward, multilayer architecture) in a supervised setting. In terms of the models, we present a principled method to “kernelize” (partly or completely) any neural network (NN). With this method, we obtain a counterpart of any given NN that is powered by kernel machines instead of neurons. In terms of learning, when learning a feedforward deep architecture in a supervised setting, one needs to train all the components simultaneously using backpropagation (BP) since there are no explicit targets for the hidden layers (Rumelhart, Hinton, & Williams, 1986). We consider without loss of generality the two-layer case and present a general framework that explicitly characterizes a target for the hidden layer that is optimal for minimizing the objective function of the network. This characterization then makes possible a purely greedy training scheme that learns one layer at a time, starting from the input layer. We provide instantiations of the abstract framework under certain architectures and objective functions. Based on these instantiations, we present a layer-wise training algorithm for an l-layer feedforward network for classification, where l≥2 can be arbitrary. This algorithm can be given an intuitive geometric interpretation that makes the learning dynamics transparent. Empirical results are provided to complement our theory. We show that the kernelized networks, trained layer-wise, compare favorably with classical kernel machines as well as other connectionist models trained by BP. We also visualize the inner workings of the greedy kernelized models to validate our claim on the transparency of the layer-wise algorithm.

中文翻译：

---

基于核方法的连接模型和无反向传播的监督深度学习

我们提出了一个基于内核机器的新型连接模型系列，并考虑在监督设置中逐层学习组合假设类（即前馈、多层架构）的问题。在模型方面，我们提出了一种“内核化”（部分或完全）任何神经网络 (NN) 的原则方法。通过这种方法，我们获得了由内核机器而不是神经元驱动的任何给定神经网络的对应物。在学习方面，在监督设置中学习前馈深度架构时，需要使用反向传播 (BP) 同时训练所有组件，因为隐藏层没有明确的目标 (Rumelhart, Hinton, & Williams, 1986)。我们在不失一般性的情况下考虑两层情况，并提出一个通用框架，该框架明确表征隐藏层的目标，该目标对于最小化网络的目标函数是最佳的。然后，这种表征使纯粹的贪婪训练方案成为可能，从输入层开始，一次学习一层。我们在某些架构和目标函数下提供抽象框架的实例。基于这些实例，我们提出了一种用于分类的 l 层前馈网络的分层训练算法，其中 l≥2 可以是任意的。该算法可以给出直观的几何解释，使学习动态透明。提供了实证结果来补充我们的理论。我们展示了逐层训练的内核化网络，与经典的核机器以及由 BP 训练的其他联结模型相媲美。我们还将贪婪内核模型的内部工作可视化，以验证我们对分层算法透明度的主张。