Randomized Kaczmarz for tensor linear systems

Abstract

Solving linear systems of equations is a fundamental problem in mathematics. When the linear system is so large that it cannot be loaded into memory at once, iterative methods such as the randomized Kaczmarz method excel. Here, we extend the randomized Kaczmarz method to solve multi-linear (tensor) systems under the tensor–tensor t-product. We present convergence guarantees for tensor randomized Kaczmarz in two ways: using the classical matrix randomized Kaczmarz analysis and taking advantage of the tensor–tensor t-product structure. We demonstrate experimentally that the tensor randomized Kaczmarz method converges faster than traditional randomized Kaczmarz applied to a naively matricized version of the linear system. In addition, we draw connections between the proposed algorithm and a previously known extension of the randomized Kaczmarz algorithm for matrix linear systems.

Appendices

Proof of Fact 1

The following properties of block circulant matrices will be useful in proving Fact 1.

Fact 2

(Lemma 1.iii [19]) For tensors ${{\mathscr {A}}}$ and ${{\mathscr {B}}}$, $\text {bcirc}\left( {{\mathscr {A}}}{{\mathscr {B}}}\right) = \text {bcirc}\left( {{\mathscr {A}}}\right) \text {bcirc}\left( {{\mathscr {B}}}\right)$.

Fact 3

(Theorem 6.ii [19]) The block circulant operator $\text {bcirc}\left( \cdot \right)$ commutes with the conjugate transpose,

$$\begin{aligned} \text {bcirc}\left( {{\mathscr {M}}}^*\right) = \text {bcirc}\left( {{\mathscr {M}}}\right) ^*. \end{aligned}$$

1. Part 1 of Fact 1 states

$$\begin{aligned} \text {bdiag}\left( \widehat{{{\mathscr {A}}}{{\mathscr {B}}}}\right) = \text {bdiag}\left( {\widehat{{{\mathscr {A}}}}}\right) \text {bdiag}\left( {\widehat{{{\mathscr {B}}}}}\right) . \end{aligned}$$

Proof

Let ${{\mathscr {A}}}\in {\mathbb {C}}^{m\times \ell \times n}$ and ${{\mathscr {B}}}\in {\mathbb {C}}^{\ell \times p\times n}$, then

$$\begin{aligned} \text {bdiag}\left( \widehat{{{\mathscr {A}}}{{\mathscr {B}}}}\right) \overset{(4.1)}{=}&\left( \mathbf{F}_n\otimes \mathbf{I}_m\right) \text {bcirc}\left( {{{\mathscr {A}}}}{{\mathscr {B}}}\right) \left( \mathbf{F}_n^*\otimes \mathbf{I}_p\right) \\ \overset{Fact 2}{=}&\left( \mathbf{F}_n\otimes \mathbf{I}_m\right) \text {bcirc}\left( {{{\mathscr {A}}}}\right) \text {bcirc}\left( {{\mathscr {B}}}\right) \left( \mathbf{F}_n^*\otimes \mathbf{I}_p\right) \\ =&\left( \mathbf{F}_n\otimes \mathbf{I}_m\right) \text {bcirc}\left( {{{\mathscr {A}}}}\right) \left( \mathbf{F}_{n}^* \otimes \mathbf{I}_{\ell }\right) \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) \text {bcirc}\left( {{\mathscr {B}}}\right) \left( \mathbf{F}_n^*\otimes \mathbf{I}_p\right) \\ \overset{(4.1)}{=}&\text {bdiag}\left( \widehat{{{\mathscr {A}}}}\right) \text {bdiag}\left( \widehat{{{\mathscr {B}}}}\right) . \end{aligned}$$

$\square$

2. Part 2 of Fact 1 states

$$\begin{aligned} \widehat{{{\mathscr {A}}}+ {{\mathscr {B}}}} = {\widehat{{{\mathscr {A}}}}} + {\widehat{{{\mathscr {B}}}}}. \end{aligned}$$

Proof

Let ${{\mathscr {A}}}\in {\mathbb {C}}^{m\times \ell \times n}$ and ${{\mathscr {B}}}\in {\mathbb {C}}^{m \times \ell \times n}$, then

$$\begin{aligned} \text {bdiag}\left( \widehat{{{\mathscr {A}}}+ {{\mathscr {B}}}}\right) \overset{(4.1)}{=}&\left( \mathbf{F}_n\otimes \mathbf{I}_m\right) \text {bcirc}\left( {{{\mathscr {A}}}} + {{\mathscr {B}}}\right) \left( \mathbf{F}_n^*\otimes \mathbf{I}_{\ell }\right) \\ =&\left( \mathbf{F}_n\otimes \mathbf{I}_m\right) \left( \text {bcirc}\left( {{\mathscr {A}}}\right) + \text {bcirc}\left( {{\mathscr {B}}}\right) \right) \left( \mathbf{F}_n^*\otimes \mathbf{I}_{\ell }\right) \\ =&\left( \mathbf{F}_n\otimes \mathbf{I}_m\right) \text {bcirc}\left( {{{\mathscr {A}}}}\right) \left( \mathbf{F}_{n}^* \otimes \mathbf{I}_{\ell }\right) + \left( \mathbf{F}_{n} \otimes \mathbf{I}_{m}\right) \text {bcirc}\left( {{\mathscr {B}}}\right) \left( \mathbf{F}_n^*\otimes \mathbf{I}_{\ell }\right) \\ \overset{(4.1)}{=}&\text {bdiag}\left( {\widehat{{{\mathscr {A}}}}}\right) + \text {bdiag}\left( {\widehat{{{\mathscr {B}}}}}\right) . \end{aligned}$$

$\square$

3. Part 3 of Fact 1 states that

$$\begin{aligned} \text {bdiag}\left( \widehat{{{\mathscr {M}}}^*}\right) = \text {bdiag}\left( {\widehat{{{\mathscr {M}}}}}\right) ^*. \end{aligned}$$

Additionally, it states that if $\text {bcirc}\left( {{\mathscr {M}}}\right)$ is symmetric, $\text {bdiag}\left( {\widehat{{{\mathscr {M}}}}}\right)$ is also symmetric.

Proof

Let ${{\mathscr {M}}}\in {\mathbb {C}}^{m \times \ell \times n}$. Then

$$\begin{aligned} \text {bdiag}\left( \widehat{{{\mathscr {M}}}^*}\right) \overset{(4.1)}{=}&\left( \mathbf{F}_n \otimes \mathbf{I}_{\ell } \right) \text {bcirc}\left( {{\mathscr {M}}}^*\right) \left( \mathbf{F}_n^* \otimes \mathbf{I}_{m} \right) \\ \overset{Fact 3}{=}&\left( \mathbf{F}_n \otimes \mathbf{I}_{\ell } \right) \text {bcirc}\left( {{\mathscr {M}}}\right) ^* \left( \mathbf{F}_n^* \otimes \mathbf{I}_{m} \right) \\ =&\left[ \left( \mathbf{F}_n \otimes \mathbf{I}_{m} \right) \text {bcirc}\left( {{\mathscr {M}}}\right) \left( \mathbf{F}_n^* \otimes \mathbf{I}_{\ell } \right) \right] ^*\\ \overset{(4.1)}{=}&\text {bdiag}\left( {\widehat{{{\mathscr {M}}}}}\right) ^*. \end{aligned}$$

To see that $\text {bdiag}\left( {\widehat{{{\mathscr {M}}}}}\right)$ is also symmetric when $\text {bcirc}\left( {{\mathscr {M}}}\right)$ is symmetric, note that

$$\begin{aligned} \text {bdiag}\left( {\widehat{{{\mathscr {M}}}}}\right) ^* \overset{(4.1)}{=} \left[ \left( \mathbf{F}_n \otimes \mathbf{I}_m \right) \text {bcirc}\left( {{\mathscr {M}}}\right) \left( \mathbf{F}_n^* \otimes \mathbf{I}_n \right) \right] ^*. \end{aligned}$$

$\square$

4. Finally, part 4 of Fact 1 states

$$\begin{aligned} \text {bdiag}\left( \widehat{{{\mathscr {M}}}^{-1}}\right) =\text {bdiag}\left( {\widehat{{{\mathscr {M}}}}}\right) ^{-1}. \end{aligned}$$

Proof

Let ${{\mathscr {M}}}\in {\mathbb {C}}^{m\times m \times n}$. Note that $\text {bcirc}\left( {{\mathscr {I}}}_m\right) = \mathbf{I}_{mn}$. Using Fact 2 and Eq. (4.1),

$$\begin{aligned} \text {bdiag}\left( \widehat{{{\mathscr {M}}}^{-1}}\right)&\text {bdiag}\left( {\widehat{{{\mathscr {M}}}}}\right) \\ \overset{(4.1)}{=}&\left( \mathbf{F}_n \otimes \mathbf{I}_m \right) \text {bcirc}\left( {{\mathscr {M}}}^{-1}\right) \left( \mathbf{F}_n^* \otimes \mathbf{I}_m \right) \left( \mathbf{F}_n \otimes \mathbf{I}_m \right) \text {bcirc}\left( {{\mathscr {M}}}\right) \left( \mathbf{F}_n^* \otimes \mathbf{I}_m \right) \\ =&\left( \mathbf{F}_n \otimes \mathbf{I}_m \right) \text {bcirc}\left( {{\mathscr {M}}}^{-1}\right) \text {bcirc}\left( {{\mathscr {M}}}\right) \left( \mathbf{F}_n^* \otimes \mathbf{I}_m \right) \\ \overset{Fact 2}{=}&\left( \mathbf{F}_n \otimes \mathbf{I}_m \right) \text {bcirc}\left( {{\mathscr {I}}}_m\right) \left( \mathbf{F}_n^* \otimes \mathbf{I}_m \right) \\ =&\mathbf{I}_{mn}. \end{aligned}$$

Analogously, one can show $\text {bdiag}\left( {\widehat{{{\mathscr {M}}}}}\right) \text {bdiag}\left( \widehat{{{\mathscr {M}}}^{-1}}\right) = \mathbf{I}_{mn}$. $\square$

Proof of Theorem 4.1

We now prove Theorem 4.1.

Proof

Let $\widehat{{{\mathscr {P}}}_i}$ be the tensor formed by applying FFTs to each tube fiber of ${{\mathscr {P}}}_i = {{\mathscr {A}}}_{i::}^* \left( {{\mathscr {A}}}_{i::}{{\mathscr {A}}}_{i::}^*\right) ^{-1}{{\mathscr {A}}}_{i::}$ and ${{\mathscr {E}}}^t = {{\mathscr {X}}}^t - {{\mathscr {X}}}^*$. By Eq. (4.1), we have that

$$\begin{aligned} \text {bdiag}\left( \widehat{{{\mathscr {P}}}_i}\right) = \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) {\text {bcirc}\left( {{\mathscr {P}}}_i\right) }\left( \mathbf{F}_{n}^* \otimes \mathbf{I}_{\ell }\right) , \end{aligned}$$

is a block diagonal matrix with blocks $\left( \widehat{\mathbf{P}_i}\right) _k$, where $\left( \widehat{\mathbf{P}_i}\right) _k$ is the kth frontal slice of the tensor $\widehat{{{\mathscr {P}}}_i}$. We note that the projected error can be rewritten as

$$\begin{aligned}&{\mathbb {E}}\left[ \left\Vert {{\mathscr {P}}}_i{{\mathscr {E}}}^t\right\Vert _F^2\right] = \sum _{j=1}^{p}\langle {\mathbb {E}}\left[ \text {bcirc}\left( {{\mathscr {P}}}_i\right) \right] \text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j},\text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j}\rangle \\&\quad = \sum _{j=1}^{p}{\mathbb {E}}\left[ \langle \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) {\text {bcirc}\left( {{\mathscr {P}}}_i\right) }\left( \mathbf{F}_{n}^* \otimes \mathbf{I}_{\ell }\right) \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) \text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j}, \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) \text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j}\rangle \right] \\&\quad = \sum _{j=1}^{p}{\mathbb {E}}\left[ \langle \text {bdiag}\left( \widehat{{{\mathscr {P}}}_i}\right) \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) \text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j}, \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) \text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j}\rangle \right] . \end{aligned}$$

Note that the rows of $\text {bcirc}\left( {{\mathscr {A}}}_{i::}\right)$ are also rows of $\text {bcirc}\left( {{\mathscr {A}}}\right)$. Thus, ${{\mathscr {X}}}^{t+1} \in \text {rowsp}\left( {{\mathscr {A}}}\right)$. Since ${{\mathscr {X}}}^*$ is the tensor of least Frobenius norm, ${{\mathscr {X}}}^*\in \text {rowsp}\left( {{\mathscr {A}}}\right)$. Therefore ${{\mathscr {E}}}^ = {{\mathscr {X}}}^t - {{\mathscr {X}}}^* \in \text {rowsp}\left( {{\mathscr {A}}}\right)$ as long as ${{\mathscr {X}}}^0\in \text {rowsp}\left( {{\mathscr {A}}}\right)$.

Now, since ${\mathbb {E}}\left[ \text {bdiag}\left( \widehat{{{\mathscr {P}}}_i}\right) \right]$ is symmetric and ${{\mathscr {E}}}^t\in \text {rowsp}\left( {{\mathscr {A}}}\right)$, by Fact 1,

$$\begin{aligned} {\mathbb {E}}\left[ \left\Vert {{\mathscr {P}}}_i{{\mathscr {E}}}^t\right\Vert _F^2\right] \ge \sigma _{\min }^+\left( {\mathbb {E}}\left[ \text {bdiag}\left( \widehat{{{\mathscr {P}}}_i}\right) \right] \right) \left\Vert \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) \text {unfold}\left( {{\mathscr {E}}}^t\right) \right\Vert _F^2. \end{aligned}$$

(B.1)

Note that,

$$\begin{aligned} \left\Vert \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) \text {unfold}\left( {{\mathscr {E}}}^t\right) \right\Vert _F^2&= \sum _{j=1}^{p} \langle \left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) \text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j},\left( \mathbf{F}_{n} \otimes \mathbf{I}_{\ell }\right) \text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j}\rangle \\&= \sum _{j=1}^{p} \langle \text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j},\text {unfold}\left( {{\mathscr {E}}}^t\right) _{:j}\rangle \\&= \left\Vert \text {unfold}\left( {{\mathscr {E}}}^t\right) \right\Vert _F^2\\&= \left\Vert {{\mathscr {E}}}^t\right\Vert _F^2. \end{aligned}$$

Since $\text {bdiag}\left( \widehat{{{\mathscr {P}}}_i}\right)$ is block diagonal,

$$\begin{aligned} \sigma _{\min }^+\left( {\mathbb {E}}\left[ \text {bdiag}\left( \widehat{{{\mathscr {P}}}_i}\right) \right] \right) = \min _{k \in [n-1]} \sigma _{\min }^+\left( {\mathbb {E}}\left[ \left( \widehat{\mathbf{P}_i}\right) _k\right] \right) . \end{aligned}$$

Factoring $\text {bdiag}\left( \widehat{{{\mathscr {P}}}_i}\right)$ and using Fact 1,

Noting that $\text {bdiag}\left( \widehat{{{{\mathscr {A}}}}_{i::}}\right) \text {bdiag}\left( \widehat{{{{\mathscr {A}}}}_{i::}^*}\right)$ is a diagonal matrix, one can see that $\left( \widehat{\mathbf{P}_i}\right) _k$ is the projection onto $\left( \widehat{{{{\mathscr {A}}}}_{i::}}\right) _k$ by rewriting the kth frontal face of $\widehat{{{\mathscr {P}}}_i}$ as

$$\begin{aligned} \left( \widehat{\mathbf{P}_i}\right) _k = \frac{\left( \widehat{{{{\mathscr {A}}}}_{i::}}\right) ^*_k\left( \widehat{ {{{\mathscr {A}}}}_{i::}}\right) _k}{\left( \widehat{{{{\mathscr {A}}}}_{i::}}\widehat{{{{\mathscr {A}}}}_{i::}^*}\right) _k}. \end{aligned}$$

We can thus rewrite Eq. (B.1) as

$$\begin{aligned} {\mathbb {E}}\left[ \left\Vert {{\mathscr {P}}}_i{{\mathscr {E}}}^t\right\Vert _F^2\right] \ge \min _{k\in [n-1]} \sigma _{\min }^+\left( {\mathbb {E}}\left[ \frac{\left( \widehat{{{{\mathscr {A}}}}_{i::}}\right) ^*_k\left( \widehat{ {{{\mathscr {A}}}}_{i::}}\right) _k}{\left( \widehat{{{{\mathscr {A}}}}_{i::}}\widehat{{{{\mathscr {A}}}}_{i::}^*}\right) _k}\right] \right) \left\Vert {{\mathscr {E}}}^t\right\Vert _F^2. \end{aligned}$$

(B.2)

The expectation of Eq. (B.1) can now be calculated explicitly. For simplicity, we assume that the row indices i are sampled uniformly. As in MRK extensions and literature, many other sampling distributions could be used.

To derive a lower bound for the smallest singular value in Eq. (B.2), define

$$\begin{aligned} \left\Vert {\widehat{\mathbf{A}}}_k\right\Vert _{\infty ,2}^2 {:=} \max _{i}\left[ \left( \widehat{{{{\mathscr {A}}}}_{i::}}\widehat{{{{\mathscr {A}}}}_{i::}^*}\right) _k\right] . \end{aligned}$$

(B.3)

The values $\left( \widehat{{{{\mathscr {A}}}}_{i::}}\widehat{{{{\mathscr {A}}}}_{i::}^*}\right) _k$ are necessarily positive for all $k \in [n-1]$ when ${{\mathscr {A}}}_{i::}{{\mathscr {A}}}_{i::}^*$ is invertible for all $i \in [m-1]$. as

$$\begin{aligned} \left( \widehat{{{{\mathscr {A}}}}_{i::}}\widehat{{{{\mathscr {A}}}}_{i::}^*}\right) _k&= \text {bdiag}\left( \widehat{{{\mathscr {A}}}_{i::}}\widehat{{{\mathscr {A}}}_{i::}^*}\right) _{kk} \\&= \text {bdiag}\left( \widehat{{{\mathscr {A}}}_{i::}}\right) _k \text {bdiag}\left( \widehat{{{\mathscr {A}}}_{i::}^*}\right) _k\\&= \left( \mathbf{F}_n\right) _{k:} \text {bcirc}\left( {{\mathscr {A}}}_{i::}\right) \text {bcirc}\left( {{\mathscr {A}}}_{i::}^*\right) \left( \mathbf{F}_n\right) _{k:}^* \\&= \left( \mathbf{F}_n\right) _{k:} \text {bcirc}\left( {{\mathscr {A}}}_{i::}\right) \text {bcirc}\left( {{\mathscr {A}}}_{i::}\right) ^* \left( \mathbf{F}_n\right) _{k:}^*\\&= \left\Vert \text {bcirc}\left( {{\mathscr {A}}}_{i::}\right) ^* \left( \mathbf{F}_n\right) _{k:}^*\right\Vert _2^2. \end{aligned}$$

Now, it can be easily verified that

$$\begin{aligned} \sigma _{\min }^+\left( {\mathbb {E}}\left[ \frac{\left( \widehat{{{{\mathscr {A}}}}_{i::}}\right) _k^*\left( \widehat{ {{{\mathscr {A}}}}_{i::}}\right) _k}{\left( \widehat{{{{\mathscr {A}}}}_{i::}}\widehat{{{{\mathscr {A}}}}_{i::}^*}\right) _k}\right] \right)&\ge \sigma _{\min }^+\left( \frac{1}{m} \sum _{i=0}^{m-1} \frac{\left( \widehat{{{{\mathscr {A}}}}_{i::}}\right) _k^*\left( \widehat{ {{{\mathscr {A}}}}_{i::}}\right) _k}{\left\Vert {\widehat{\mathbf{A}}}_k\right\Vert _{\infty ,2}^2}\right) \\&= \frac{\left[ \sigma ^+_{\min }\left( {\widehat{\mathbf{A}}}_k \right) \right] ^2}{m\left\Vert {\widehat{\mathbf{A}}}_k\right\Vert _{\infty ,2}^2} . \end{aligned}$$

The projected error of Eq. (B.2) then becomes

$$\begin{aligned} {\mathbb {E}}\left[ \left\Vert {{\mathscr {P}}}_i{{\mathscr {E}}}^t\right\Vert _F^2\right] \ge \min _{k\in [n-1]} \frac{\left[ \sigma ^+_{\min }\left( {\widehat{\mathbf{A}}}_k \right) \right] ^2}{m\left\Vert {\widehat{\mathbf{A}}}_k\right\Vert _{\infty ,2}^2} \left\Vert {{\mathscr {E}}}^t\right\Vert _F^2., \end{aligned}$$

We can thus rewrite the guarantee in Theorem 4.1 for uniform random sampling of the row indices i as

$$\begin{aligned} {\mathbb {E}}\left[ \left\Vert {{\mathscr {X}}}^{t+1}-{{\mathscr {X}}}^*\right\Vert _F^2\bigg | {{\mathscr {X}}}^{0}\right] \le \left( 1-\min _{k\in [n-1]} \frac{\left[ \sigma ^+_{\min }\left( {\widehat{\mathbf{A}}}_k \right) \right] ^2}{m\left\Vert {\widehat{\mathbf{A}}}_k\right\Vert _{\infty ,2}^2}\right) ^{t+1} \left\Vert {{\mathscr {X}}}^{0}-{{\mathscr {X}}}^*\right\Vert _F^2. \end{aligned}$$

$\square$