The delayed weighted gradient method, recently introduced in Oviedo-Leon (Comput Optim Appl 74:729–746, 2019), is a low-cost gradient-type method that exhibits a surprisingly and perhaps unexpected fast convergence behavior that competes favorably with the well-known conjugate gradient method for the minimization of convex quadratic functions. In this work, we establish several orthogonality properties

In this paper we study several classes of stochastic optimization algorithms enriched with heavy ball momentum. Among the methods studied are: stochastic gradient descent, stochastic Newton, stochastic proximal point and stochastic dual subspace ascent. This is the first time momentum variants of several of these methods are studied. We choose to perform our analysis in a setting in which all of the

In the case of singular (and possibly even nonisolated) solutions of nonlinear equations, while superlinear convergence of the Newton method cannot be guaranteed, local linear convergence from large domains of starting points still holds under certain reasonable assumptions. We consider a linesearch globalization of the Newton method, combined with extrapolation and over-relaxation accelerating techniques

In this work, we deal with Truncated Newton methods for solving large scale (possibly nonconvex) unconstrained optimization problems. In particular, we consider the use of a modified Bunch and Kaufman factorization for solving the Newton equation, at each (outer) iteration of the method. The Bunch and Kaufman factorization of a tridiagonal matrix is an effective and stable matrix decomposition, which

In this paper we apply an augmented Lagrange method to a class of semilinear elliptic optimal control problems with pointwise state constraints. We show strong convergence of subsequences of the primal variables to a local solution of the original problem as well as weak convergence of the adjoint states and weak-* convergence of the multipliers associated to the state constraint. Moreover, we show

In this paper, we introduce a Sequential Partial Linearization (SPL) algorithm for finding a solution of the symmetric Eigenvalue Complementarity Problem (EiCP). The algorithm can also be used for the computation of a stationary point of a standard fractional quadratic program. A first version of the SPL algorithm employs a line search technique and possesses global convergence to a solution of the

Many engineering and scientific applications, e.g. resource allocation, control of hybrid systems, scheduling, etc., require the solution of mixed-integer non-linear problems (MINLPs). Problems of such class combine the high computational burden arising from considering discrete variables with the complexity of non-linear functions. As a consequence, the development of algorithms able to efficiently

This paper presents Riemannian conjugate gradient methods and global convergence analyses under the strong Wolfe conditions. The main idea of the proposed methods is to combine the good global convergence properties of the Dai–Yuan method with the efficient numerical performance of the Hestenes–Stiefel method. One of the proposed algorithms is a generalization to Riemannian manifolds of the hybrid

Convergence rates are established for an inexact accelerated alternating direction method of multipliers (I-ADMM) for general separable convex optimization with a linear constraint. Both ergodic and non-ergodic iterates are analyzed. Relative to the iteration number k, the convergence rate is \(\mathcal{{O}}(1/k)\) in a convex setting and \(\mathcal{{O}}(1/k^2)\) in a strongly convex setting. When

In this paper, we first introduce a new class of structured tensors, named strictly positive semidefinite tensors, and show that a strictly positive semidefinite tensor is not an \(R_0\) tensor. We focus on the stochastic tensor complementarity problem (STCP), where the expectation of the involved tensor is a strictly positive semidefinite tensor. We denote such an STCP as a monotone STCP. Based on

We propose a new class of Riemannian conjugate gradient (CG) methods, in which inverse retraction is used instead of vector transport for search direction construction. In existing methods, differentiated retraction is often used for vector transport to move the previous search direction to the current tangent space. However, a different perspective is adopted here, motivated by the fact that inverse

In this work, we introduce self-adaptive methods for solving variational inequalities with Lipschitz continuous and quasimonotone mapping(or Lipschitz continuous mapping without monotonicity) in real Hilbert space. Under suitable assumptions, the convergence of algorithms are established without the knowledge of the Lipschitz constant of the mapping. The results obtained in this paper extend some recent

In this paper, we study the interplay between acceleration and structure identification for the proximal gradient algorithm. While acceleration is generally beneficial in terms of functional decrease, we report and analyze several cases where its interplay with identification has negative effects on the algorithm behavior (iterates oscillation, loss of structure, etc.). Then, we present a generic method

Algorithmic differentiation (AD) is a widely-used approach to compute derivatives of numerical models. Many numerical models include an iterative process to solve non-linear systems of equations. To improve efficiency and numerical stability, AD is typically not applied to the linear solvers. Instead, the differentiated linear solver call is replaced with hand-produced derivative code that exploits

Global optimization problems whose objective function is expensive to evaluate can be solved effectively by recursively fitting a surrogate function to function samples and minimizing an acquisition function to generate new samples. The acquisition step trades off between seeking for a new optimization vector where the surrogate is minimum (exploitation of the surrogate) and looking for regions of

This paper is devoted to a numerical method for the approximation of a class of free boundary problems of Bernoulli’s type, reformulated as optimal shape design problems with appropriate shape functionals. We show the existence of the shape derivative of the cost functional on a class of admissible domains and compute its shape derivative by using the formula proposed in Boulkhemair (SIAM J Control

In this paper, we combine the positive aspects of the gradient sampling (GS) and bundle methods, as the most efficient methods in nonsmooth optimization, to develop a robust method for solving unconstrained nonsmooth convex optimization problems. The main aim of the proposed method is to take advantage of both GS and bundle methods, meanwhile avoiding their drawbacks. At each iteration of this method

In this work we introduce and study novel Quasi Newton minimization methods based on a Hessian approximation Broyden Class-type updating scheme, where a suitable matrix \(\tilde{B}_k\) is updated instead of the current Hessian approximation \(B_k\). We identify conditions which imply the convergence of the algorithm and, if exact line search is chosen, its quadratic termination. By a remarkable connection

The distance between sets is a long-standing computational geometry problem. In robotics, the distance between convex sets with Minkowski sum structure plays a fundamental role in collision detection. However, it is typically nontrivial to be computed, even if the projection onto each component set admits explicit formula. In this paper, we explore the problem of calculating the distance between convex

We consider nested variational inequalities consisting in a (upper-level) variational inequality whose feasible set is given by the solution set of another (lower-level) variational inequality. Purely hierarchical convex bilevel optimization problems and certain multi-follower games are particular instances of nested variational inequalities. We present an explicit and ready-to-implement Tikhonov-type

We develop a unified level-bundle method, called accelerated constrained level-bundle (ACLB) algorithm, for solving constrained convex optimization problems. where the objective and constraint functions can be nonsmooth, weakly smooth, and/or smooth. ACLB employs Nesterov’s accelerated gradient technique, and hence retains the iteration complexity as that of existing bundle-type methods if the objective

Compressed Sensing using \(\ell _1\) regularization is among the most powerful and popular sparsification technique in many applications, but why has it not been used to obtain sparse deep learning model such as convolutional neural network (CNN)? This paper is aimed to provide an answer to this question and to show how to make it work. Following Xiao (J Mach Learn Res 11(Oct):2543–2596, 2010), We

The problem of recovering acoustic sources, more specifically monopoles, from point-wise measurements of the corresponding acoustic pressure at a limited number of frequencies is addressed. To this purpose, a family of sparse optimization problems in measure space in combination with the Helmholtz equation on a bounded domain is considered. A weighted norm with unbounded weight near the observation

The exterior Bernoulli problem is rephrased into a shape optimization problem using a new type of objective function called the Dirichlet-data-gap cost function which measures the \(L^2\)-distance between the Dirichlet data of two state functions. The first-order shape derivative of the cost function is explicitly determined via the chain rule approach. Using the same technique, the second-order shape

The alternating direction method of multipliers (ADMM) is widely used to solve separable convex programming problems. At each iteration, the classical ADMM solves the subproblems exactly. For many problems arising from practical applications, it is usually impossible or too expensive to obtain the exact solution of a subproblem. To overcome this, a special proximal term is added to ease the solvability

In this work we study binary two-stage robust optimization problems with objective uncertainty. We present an algorithm to calculate efficiently lower bounds for the binary two-stage robust problem by solving alternately the underlying deterministic problem and an adversarial problem. For the deterministic problem any oracle can be used which returns an optimal solution for every possible scenario

We consider a misspecified optimization problem, requiring the minimization of a function \(f(\cdot;\theta ^*)\) over a closed and convex set X where \(\theta ^*\) is an unknown vector of parameters that may be learnt by a parallel learning process. Here, we develop coupled schemes that generate iterates \((x_k,\theta _k)\) as \(k \rightarrow \infty\), then \(x_k \rightarrow x^*\), a minimizer of \(f(\cdot;\theta

In this paper, we propose an adaptation of the classical augmented Lagrangian method for dealing with multi-objective optimization problems. Specifically, after a brief review of the literature, we give a suitable definition of Augmented Lagrangian for equality and inequality constrained multi-objective problems. We exploit this object in a general computational scheme that is proved to converge, under

In a series of papers (Solodov and Svaiter in J Convex Anal 6(1):59–70, 1999; Set-Valued Anal 7(4):323–345, 1999; Numer Funct Anal Optim 22(7–8):1013–1035, 2001) Solodov and Svaiter introduced new inexact variants of the proximal point method with relative error tolerances. Point-wise and ergodic iteration-complexity bounds for one of these methods, namely the hybrid proximal extragradient method (1999)

We propose a regularization method for nonlinear least-squares problems with equality constraints. Our approach is modeled after those of Arreckx and Orban (SIAM J Optim 28(2):1613–1639, 2018. https://doi.org/10.1137/16M1088570) and Dehghani et al. (INFOR Inf Syst Oper Res, 2019. https://doi.org/10.1080/03155986.2018.1559428) and applies a selective regularization scheme that may be viewed as a reformulation

Convex and nonconvex finite-sum minimization arises in many scientific computing and machine learning applications. Recently, first-order and second-order methods where objective functions, gradients and Hessians are approximated by randomly sampling components of the sum have received great attention. We propose a new trust-region method which employs suitable approximations of the objective function

This paper presents an approximation scheme for optimal control problems using finite-dimensional linear programs and interval analysis. This is done in two parts. Following Vinter approach (SIAM J Control Optim 31(2):518–538, 1993) and using occupation measures, the optimal control problem is written into a linear programming problem of infinite-dimension (weak formulation). Thanks to Interval arithmetic

In this paper, we describe a new active-set algorithmic framework for minimizing a non-convex function over the unit simplex. At each iteration, the method makes use of a rule for identifying active variables (i.e., variables that are zero at a stationary point) and specific directions (that we name active-set gradient related directions) satisfying a new “nonorthogonality” type of condition. We prove

Quasi-convex optimization acts a pivotal part in many fields including economics and finance; the subgradient method is an effective iterative algorithm for solving large-scale quasi-convex optimization problems. In this paper, we investigate the quantitative convergence theory, including the iteration complexity and convergence rates, of various subgradient methods for solving quasi-convex optimization

This paper proposes and analyzes an inexact variant of the proximal generalized alternating direction method of multipliers (ADMM) for solving separable linearly constrained convex optimization problems. In this variant, the first subproblem is approximately solved using a relative error condition whereas the second one is assumed to be easy to solve. In many ADMM applications, one of the subproblems

This short note considers an efficient variant of the trust-region algorithm with dynamic accuracy proposed by Carter (SIAM J Sci Stat Comput 14(2):368–388, 1993) and by Conn et al. (Trust-region methods. MPS-SIAM series on optimization, SIAM, Philadelphia, 2000) as a tool for very high-performance computing, an area where it is critical to allow multi-precision computations for keeping the energy

In this paper we consider the classical unconstrained nonlinear multiobjective optimization problem. For such a problem, it is particularly interesting to compute as many points as possible in an effort to approximate the so-called Pareto front. Consequently, to solve the problem we define an “a posteriori” algorithm whose generic iterate is represented by a set of points rather than by a single one

The characterization and numerical solution of two non-smooth optimal control problems governed by a Fokker–Planck (FP) equation are investigated in the framework of the Pontryagin maximum principle (PMP). The two FP control problems are related to the problem of determining open- and closed-loop controls for a stochastic process whose probability density function is modelled by the FP equation. In

Euclidean distance matrix optimization with ordinal constraints (EDMOC) has found important applications in sensor network localization and molecular conformation. It can also be viewed as a matrix formulation of multidimensional scaling, which is to embed n points in a r-dimensional space such that the resulting distances follow the ordinal constraints. The ordinal constraints, though proved to be

This paper proposes an efficient adaptive variant of a quadratic penalty accelerated inexact proximal point (QP-AIPP) method proposed earlier by the authors. Both the QP-AIPP method and its variant solve linearly set constrained nonconvex composite optimization problems using a quadratic penalty approach where the generated penalized subproblems are solved by a variant of the underlying AIPP method

We investigate the convergence of a recently popular class of first-order primal–dual algorithms for saddle point problems under the presence of errors in the proximal maps and gradients. We study several types of errors and show that, provided a sufficient decay of these errors, the same convergence rates as for the error-free algorithm can be established. More precisely, we prove the (optimal) \(O\left(

This paper considers a class of two-stage stochastic linear variational inequality problems whose first stage problems are stochastic linear box-constrained variational inequality problems and second stage problems are stochastic linear complementary problems having a unique solution. We first give conditions for the existence of solutions to both the original problem and its perturbed problems. Next

Nonlinear disjunctive convex sets arise naturally in the formulation or solution methods of many discrete–continuous optimization problems. Often, a tight algebraic representation of the disjunctive convex set is sought, with the tightest such representation involving the characterization of the convex hull of the disjunctive convex set. In the most general case, this can be explicitly expressed through

This paper studies an acceleration technique for incremental aggregated gradient (IAG) method through the use of curvature information for solving strongly convex finite sum optimization problems. These optimization problems of interest arise in large-scale learning applications. Our technique utilizes a curvature-aided gradient tracking step to produce accurate gradient estimates incrementally using

We introduce new global and local inexact oracle concepts for a wide class of convex functions in composite convex minimization. Such inexact oracles naturally arise in many situations, including primal–dual frameworks, barrier smoothing, and inexact evaluations of gradients and Hessians. We also provide examples showing that the class of convex functions equipped with the newly inexact oracles is

In this paper, we propose an Augmented Lagrangian algorithm for solving a general class of possible non-convex problems called quasi-equilibrium problems (QEPs). We define an Augmented Lagrangian bifunction associated with QEPs, introduce a secondary QEP as a measure of infeasibility and we discuss several special classes of QEPs within our theoretical framework. For obtaining global convergence under

This paper presents a generalization of the spectral norm and the nuclear norm of a tensor via arbitrary tensor partitions, a much richer concept than block tensors. We show that the spectral p-norm and the nuclear p-norm of a tensor can be lower and upper bounded by manipulating the spectral p-norms and the nuclear p-norms of subtensors in an arbitrary partition of the tensor for \(1\le p\le \infty\)

Our contribution in this paper is twofold. First, the global convergence analysis of the recently proposed pattern search implicit filtering algorithm (PSIFA), aimed at linearly constrained noisy minimization problems, is revisited to address more general locally Lipschitz objective functions corrupted by noise. Second, PSIFA is applied for solving the damped harmonic oscillator parameter identification

This paper is concerned with the computation of perfect stationary point, which is a strict refinement of stationary point. A differentiable homotopy method is developed for finding perfect stationary points of continuous functions on convex polytopes. We constitute an artificial problem by introducing a continuously differentiable function of an extra variable. With the optimality conditions of this

This paper considers the problem of minimizing an expectation function over a closed convex set, coupled with a function or expectation constraint on either decision variables or problem parameters. We first present a new stochastic approximation (SA) type algorithm, namely the cooperative SA (CSA), to handle problems with the constraint on devision variables. We show that this algorithm exhibits the

We consider a class of optimistic bilevel problems. Specifically, we address bilevel problems in which at the lower level the objective function is fully convex and the feasible set does not depend on the upper level variables. We show that this nontrivial class of mathematical programs is sufficiently broad to encompass significant real-world applications and proves to be numerically tractable. From

We study the convergence of a modified proximal point method for DC functions in Hadamard manifolds. We use the iteration computed by the proximal point method for DC function extended to the Riemannian context by Souza and Oliveira (J Glob Optim 63:797–810, 2015) to define a descent direction which improves the convergence of the method. Our method also accelerates the classical proximal point method

The problem of minimizing the sum of a convex quadratic function and the ratio of two quadratic functions can be reformulated as a Celis–Dennis–Tapia (CDT) problem and, thus, according to some recent results, can be polynomially solved. However, the degree of the known polynomial approaches for these problems is fairly large and that justifies the search for efficient local search procedures. In this

We propose a discretization algorithm for solving a class of nonsmooth convex semi-infinite programming problems that is based on a bundle method. Instead of employing the inexact calculation to evaluate the lower level problem, we shall carry out a discretization scheme. The discretization method is used to get a number of discretized problems which are solved by the bundle method. In particular,

In this article a special class of nonlinear optimal control problems involving a bilinear term in the boundary condition is studied. These kind of problems arise for instance in the identification of an unknown space-dependent Robin coefficient from a given measurement of the state, or when the Robin coefficient can be controlled in order to reach a desired state. Necessary and sufficient optimality

This paper considers the efficient minimization of the infinite time average of a stationary ergodic process in the space of a handful of design parameters which affect it. Problems of this class, derived from physical or numerical experiments which are sometimes expensive to perform, are ubiquitous in engineering applications. In such problems, any given function evaluation, determined with finite

Mathematical programs with or-constraints form a new class of disjunctive optimization problems with inherent practical relevance. In this paper, we provide a comparison of three different solution methods for the numerical treatment of this problem class which are inspired by classical approaches from disjunctive programming. First, we study the replacement of the or-constraints as nonlinear inequality

We consider the semi-infinite system of polynomial inequalities of the form$$\begin{aligned} {{\mathbf {K}}}:=\{x\in {{\mathbb {R}}}^m\mid p(x,y)\ge 0,\quad \forall y\in S\subseteq {{\mathbb {R}}}^n\}, \end{aligned}$$where p(x, y) is a real polynomial in the variables x and the parameters y, the index set S is a basic semialgebraic set in \({{\mathbb {R}}}^n\), \(-p(x,y)\) is convex in x for every

Tensor decompositions have become increasingly prevalent in recent years. Traditionally, tensors are represented or decomposed as a sum of rank-one outer products using either the CANDECOMP/PARAFAC, the Tucker model, or some variations thereof. The motivation of these decompositions is to find an approximate representation for a given tensor. The main propose of this paper is to develop two neural

We extend the result on the spectral projected gradient method by Birgin et al. in 2000 to a log-determinant semidefinite problem with linear constraints and propose a spectral projected gradient method for the dual problem. Our method is based on alternate projections on the intersection of two convex sets, which first projects onto the box constraints and then onto a set defined by a linear matrix