Deep Neural Networks (DNNs) are very popular these days, and are the subject of a very intense investigation. A DNN is made up of layers of internal units (or neurons), each of which computes an affine combination of the output of the units in the previous layer, applies a nonlinear operator, and outputs the corresponding value (also known as activation). A commonly-used nonlinear operator is the so-called rectified linear unit (ReLU), whose output is just the maximum between its input value and zero. In this (and other similar cases like max pooling, where the max operation involves more than one input value), for fixed parameters one can model the DNN as a 0-1 Mixed Integer Linear Program (0-1 MILP) where the continuous variables correspond to the output values of each unit, and a binary variable is associated with each ReLU to model its yes/no nature. In this paper we discuss the peculiarity of this kind of 0-1 MILP models, and describe an effective bound-tightening technique intended to ease its solution. We also present possible applications of the 0-1 MILP model arising in feature visualization and in the construction of adversarial examples. Computational results are reported, aimed at investigating (on small DNNs) the computational performance of a state-of-the-art MILP solver when applied to a known test case, namely, hand-written digit recognition.

中文翻译：

---

深度神经网络和混合整数线性优化

如今，深层神经网络（DNN）十分流行，并且受到了广泛研究。DNN由内部单元（或神经元）的层组成，每个内部单元计算上一层中单元输出的仿射组合，应用非线性算子，并输出相应的值（也称为激活）。常用的非线性算子是所谓的整流线性单元（ReLU），其输出恰好是其输入值和零之间的最大值。在这种情况下（以及其他类似情况，例如最大池，最大操作涉及多个输入值），对于固定参数，可以将DNN建模为0-1混合整数线性程序（0-1 MILP），其中连续变量对应于每个单元的输出值，每个ReLU都关联了一个二进制变量以对它的是/否性质进行建模。在本文中，我们讨论了这种0-1 MILP模型的特殊性，并描述了一种旨在简化其求解的有效的束紧技术。我们还将介绍0-1 MILP模型在特征可视化和对抗性示例的构建中可能的应用。报告了计算结果，旨在研究（在小型DNN上）最新的MILP求解器应用于已知测试用例（即手写数字识别）的计算性能。我们还将介绍0-1 MILP模型在特征可视化和对抗性示例的构建中可能的应用。报告了计算结果，旨在研究（在小型DNN上）最新的MILP求解器应用于已知测试用例（即手写数字识别）的计算性能。我们还将介绍0-1 MILP模型在特征可视化和对抗性示例的构建中可能的应用。报告了计算结果，旨在研究（在小型DNN上）最新的MILP求解器应用于已知测试用例（即手写数字识别）的计算性能。