Skip to main content
SpringerLink
Log in

Provably optimal sparse solutions to overdetermined linear systems with non-negativity constraints in a least-squares sense by implicit enumeration

  • Research Article
  • Published:
Optimization and Engineering Aims and scope Submit manuscript

Abstract

Computing sparse solutions to overdetermined linear systems is a ubiquitous problem in several fields such as regression analysis, signal and image processing, information theory and machine learning. Additional non-negativity constraints in the solution are useful for interpretability. Most of the previous research efforts aimed at approximating the sparsity constrained linear least squares problem, and/or finding local solutions by means of descent algorithms. The objective of the present paper is to report on an efficient and modular implicit enumeration algorithm to find provably optimal solutions to the NP-hard problem of sparsity-constrained non-negative least squares. We focus on the problem where the system is assumed to be over-determined where the matrix has full column rank. Numerical results with real test data as well as comparisons of competing methods and an application to hyperspectral imaging are reported. Finally, we present a Python library implementation of our algorithm.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18
Fig. 19

Similar content being viewed by others

Notes

  1. As alluded to in Sect. 2, in machine learning and statistics an alternative notation is commonly used:

    $$\begin{aligned} \begin{array}{ll@{}ll} \underset{\beta }{\min }&{}\left\Vert y - X\beta \right\Vert _2^2 &{}\\ \text {s.t.} &{}\beta \ge 0 &{}\\ \end{array} \end{aligned}$$
    (3.2)

    where \(\beta \in {\mathbb {R}}^{k},y \in {\mathbb {R}}^{n}\), \(X \in {\mathbb {R}}^{n\times k}\) and \(\beta\)s are referred to as coefficients. Unless otherwise indicated, we prefer to stick to the usual A, b-notation of linear systems of equations.

References

Download references

Author information

Authors and Affiliations

  1. Department of Industrial Engineering, Bilkent University, Ankara, Turkey

    Fatih S. Aktaş, Ömer Ekmekcioglu & Mustafa Ç. Pinar

Authors
  1. Fatih S. Aktaş
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ömer Ekmekcioglu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mustafa Ç. Pinar
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Mustafa Ç. Pinar.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Submitted to the editors DATE.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 120 KB)

Appendices

Appendices

1.1 Details of parameters and variables

Both the matrix A and the vector b should be registered as numpy arrays and one should specify the data type as float64 even if all the values are integer to avoid potential numerical issues. While registering and working with matrices or vectors using Python, shape definition is very important. Shape of the vector leads to large changes in how linear algebra functions behave. Python has two different definitions of vector. If the user registers a one dimensional vector of ones of length n, Python registers it as a vector that has a shape of (n,). However, if instead numpy.ones([1,10]) is used, then it will mathematically express the same vector but in Python it will have the shape of (1,10). Using matrix operations with these two different vectors will give different results. The vector b should be registered as a column vector of shape (m,1).

NNLSSPAR calls the guppy3 package by Zhu and Nilsson for tracking the memory usage of the algorithm Zhu and Nilsson (2019). As a result, the guppy3 package should be installed to be able to use NNLSSPAR.

A description of the parameters of NNLSSPAR library is given below.

Parameters:

  1. 1.

    A (input) = \(m\, \times \,n\) matrix storing the values of the independent variables where m must be greater than n

  2. 2.

    b (input) = \(m\, \times \, 1\) vector storing the values of the dependent variables

  3. 3.

    s (input) = An integer value indicating sparsity level (maximum number of nonzero elements in the solution)

  4. 4.

    out (input, by default it is 1) = An integer value, that is a parameter controlling the detail level of output:

    1. (a)

      out = 0, Nothing will be printed

    2. (b)

      out = 1, The results and the hardware usage of the algorithm will be printed

    3. (c)

      out = 2, At every iteration, the lower bound of the current node and the number of nodes searched so far will be printed. After the algorithm terminates NNLSSPAR will report the results and the hardware usage. Although it is good to observe the progress of the algorithm, it should be noted here that it slows down the algorithm significantly.

  5. 5.

    C (input, by default it is \(\emptyset\)) = Array of integers, storing the indices of chosen variables

  6. 6.

    many (input, by default it is 4) = An integer specifying number of best subsets to be found

  7. 7.

    residual_squared (output) = a list of length many ( \(s * many\) if all subsets function is used) containing residuals corresponding to subset found for some sparsity level. This only shows the residuals of the system after QR decomposition. True residual squared of the systems are residual_squared + permanent_residual

  8. 8.

    indexes (output) = a list of length many (\(s * many\) if all subsets function is used) containing indices of the independent variables that make the subsets for some sparsity level

  9. 9.

    coefficients (output) = a list of length many (\(s * many\) if all subsets function is used) containing values of the least squares solution of each subset for some sparsity level

  10. 10.

    permanent_residual (output) = a float that records the residual squared of the system after orthogonal transform using QR decomposition of the matrix A

  11. 11.

    cpu (output) = a float showing the CPU time of the algorithm in seconds. CPU time of the algorithm is print if out is not 0

  12. 12.

    memory (output) = a float showing the total memory usage of the algorithm in bytes. Detailed table of memory usage is print if out is not 0

  13. 13.

    real (output) = a float showing the wall time of the algorithm in seconds.

  14. 14.

    nodes (hidden) = a list of integers showing number of visited nodes in the graph. It is equal to the number of branchings done due to best first search if only one solution is found. For each sparsity level it will be reported separately.

  15. 15.

    rem_qsize (hidden) = a list of integers showing the number of unvisited nodes in the graph. This will be equal to nodes if only one solution is found. For each sparsity level it will be reported separately.

1.2 An example

Assume we are trying to solve an instance of a problem where A, b and s are specified for input into NNLSSPAR in Python syntax as follows:

figure b

In the output, the list of values of the variables are in the order given by the indexes. Hence, 6.64783762 is the value of fourth independent variable, 1.62440717 is the value of third independent variable, and so on.

1.3 A special case

If the solution of multiple subsets for each sparsity level \(i = 1,2, \ldots , s-1 ,s\) is desired, then the following function should be called:

figure c

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aktaş, F.S., Ekmekcioglu, Ö. & Pinar, M.Ç. Provably optimal sparse solutions to overdetermined linear systems with non-negativity constraints in a least-squares sense by implicit enumeration. Optim Eng 22, 2505–2535 (2021). https://doi.org/10.1007/s11081-021-09676-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11081-021-09676-2

Keywords

Mathematics Subject Classification

Search

Navigation