The quadratic stable set problem (QSSP) is a natural extension of the well-known maximum stable set problem. The QSSP is NP-hard and can be formulated as a binary quadratic program, which makes it an interesting case study to be tackled from different optimization paradigms. In this paper, we propose a novel representation for the QSSP through binary decision diagrams (BDDs) and adapt a hybrid optimization approach which integrates BDDs and mixed-integer programming (MIP) for solving the QSSP. The exact framework highlights the modeling flexibility offered through decision diagrams to handle nonlinear problems. In addition, the hybrid approach leverages two different representations by exploring, in a complementary way, the solution space with BDD and MIP technologies. Machine learning then becomes a valuable component within the method to guide the search mechanisms. In the numerical experiments, the hybrid approach shows to be superior, by at least one order of magnitude, than two leading commercial MIP solvers with quadratic programming capabilities and a semidefinite-based branch-and-bound solver.

二次稳定集问题（QSSP）是众所周知的最大稳定集问题的自然扩展。QSSP是NP难解的，可以公式化为二进制二次程序，这使得它成为一个有趣的案例研究，可以从不同的优化范例中解决。在本文中，我们通过二进制决策图（BDD）提出了一种新颖的QSSP表示形式，并采用了将BDD与混合整数编程（MIP）相集成的混合优化方法来求解QSSP。确切的框架强调了决策图为处理非线性问题提供的建模灵活性。此外，混合方法通过以互补的方式探索BDD和MIP技术的解决方案空间，从而利用了两种不同的表示形式。机器学习于是成为指导搜索机制的方法中的重要组成部分。在数值实验中，混合方法显示出比两个具有二次编程功能和基于半定界的分支定界求解器的领先商业MIP求解器至少好一个数量级。