Effective hyper-parameter tuning is essential to guarantee the performance that neural networks have come to be known for.In this work, a principled approach to choosing the learning rate is proposed for shallow feedforward neural networks. We associate the learning rate with the gradient Lipschitz constant of the objective to be minimized while training. An upper bound on the mentioned constant is derived, and a search algorithm, which always results in non-divergent traces, is proposed to exploit the derived bound. It is shown through simulations that the proposed search method significantly outperforms the existing tuning methods such as Tree Parzen Estimators (TPE). The proposed method is applied to three different existing applications: a) channel estimation in OFDM systems, b) prediction of the exchange currency rates, and c) offset estimation in OFDM receivers, and it is shown to pick better learning rates than the existing methods using the same or lesser compute power.

中文翻译：

---

调整更聪明而不是更难：一种调整浅层网络学习率的原则方法

有效的超参数调整对于保证众所周知的神经网络性能至关重要。在这项工作中，为浅层前馈神经网络提出了一种选择学习率的原则方法。我们将学习率与训练时要最小化的目标的梯度 Lipschitz 常数相关联。导出了上述常数的上限，并提出了一种搜索算法，它总是导致非发散轨迹，以利用导出的界限。通过模拟表明，所提出的搜索方法明显优于现有的调整方法，例如 Tree Parzen Estimators (TPE)。所提出的方法适用于三种不同的现有应用：a) OFDM 系统中的信道估计，b) 汇率预测，