The geometrical features of the (non-convex) loss landscape of neural network models are crucial in ensuring successful optimization and, most importantly, the capability to generalize well. While minimizers' flatness consistently correlates with good generalization, there has been little rigorous work in exploring the condition of existence of such minimizers, even in toy models. Here we consider a simple neural network model, the symmetric perceptron, with binary weights. Phrasing the learning problem as a constraint satisfaction problem, the analogous of a flat minimizer becomes a large and dense cluster of solutions, while the narrowest minimizers are isolated solutions. We perform the first steps toward the rigorous proof of the existence of a dense cluster in certain regimes of the parameters, by computing the first and second moment upper bounds for the existence of pairs of arbitrarily close solutions. Moreover, we present a non rigorous derivation of the same bounds for sets of $y$ solutions at fixed pairwise distances.

中文翻译：

---

对称二元感知器中解的聚类

神经网络模型的（非凸）损失景观的几何特征对于确保成功优化至关重要，最重要的是，它具有良好的泛化能力。虽然最小化器的平坦度始终与良好的泛化相关，但在探索此类最小化器的存在条件方面几乎没有严格的工作，即使在玩具模型中也是如此。在这里，我们考虑一个简单的神经网络模型，对称感知器，具有二进制权重。将学习问题表述为约束满足问题，扁平最小化器的类似物变成了一个大而密集的解决方案集群，而最窄的最小化器是孤立的解决方案。我们迈出了严格证明在某些参数范围内存在密集簇的第一步，通过计算任意接近解对的存在的一阶和二阶矩上限。此外，我们对固定成对距离的 $y$ 解决方案集提出了相同边界的非严格推导。