Multilinear algebra for minimum storage regenerating codes: a generalization of the product-matrix construction

Abstract

An $(n, k, d, \alpha )$-MSR (minimum storage regeneration) code is a set of n nodes used to store a file. For a file of total size $k\alpha$, each node stores $\alpha$ symbols, any k nodes determine the file, and any d nodes can repair any other node by each sending out $\alpha /(d-k+1)$ symbols. In this work, we express the product-matrix construction of $\bigl (n, k, 2(k-1), k-1\bigr )$-MSR codes in terms of symmetric algebras. We then generalize the product-matrix construction to $\bigl (n, k, \frac{(k-1)t}{t-1}, \left( {\begin{array}{c}k-1\\ t-1\end{array}}\right) \bigr )$-MSR codes for general $t\geqslant 2$, while the $t=2$ case recovers the product-matrix construction. Our codes’ sub-packetization level—$\alpha$—is small and independent of n. It is less than $L^{2.8(d-k+1)}$, where L is Alrabiah–Guruswami’s lower bound on $\alpha$. Furthermore, it is less than other MSR codes’ $\alpha$ for a set of practical parameters. Finally, we discuss how our code repairs multiple failures at once.

Appendices

Bonus property: repairing two nodes at once

In general, node failures separate in time. But there may be circumstances where multiple nodes fail at once. Definitions 1 and 2 do not intend to optimize over the cases when there are two or more nodes to be repaired. What Definitions 1 and 2 guarantee is that, A, so far as there are k healthy nodes left, the file is safe; and B, if there are d healthy nodes left, one may call the repairing protocol for each and every failing node. This does not capture how efficient the repairing can be done given that help messages to two failing nodes could be correlated. For that, two definitions of simultaneous repairing are made in [4, 33] and are related in [43].

Let there be c failing nodes, and d nodes are to help. The centralized (total) bandwidth $\gamma _{\text {ce}}(c,d)$ is the total number of symbols the d helper nodes send to a central agent, who will repair the failing nodes after gathering all help messages. Note that the bandwidth from the agent to a failing node is presumably $\alpha$, so this fixed cost does not count toward $\gamma$. The cooperative (total) bandwidth $\gamma _{\text {co}}(c,d)$ is how many symbols are sent over the network, from a helper node or a failing one to a failing one, that contribute to repairing. See Fig. 2 for illustrations of the two models.

Fig. 2

To the left: centralized model. Healthy nodes send help messages to an agent. The total bandwidth $\gamma _{\text {ce}}(2,6)$ is the number of symbols passing solid lines. To the right: cooperative model. Healthy nodes send help messages directly to the failing nodes, and the latter can help each other. The total bandwidth $\gamma _{\text {co}}(2,6)$ is the number of symbols passing solid lines

Note that we do not normalize the total bandwidth by the number of helping, or failing, nodes. But note that when there is one failing node, $\gamma _{\text {ce}}(1,d)=\gamma _{\text {co}}(1,d)=d\beta$. We now demonstrate how a $t=3$ Atrahasis code can repair two failing nodes. Recall the parameters $(d,\alpha ,\beta ,M)=\bigl (\frac{3(k-1)}{2},\big (\begin{array}{c}{k-1}\\ {2}\end{array}\big ),k-2,k\alpha \bigr )$ and definitions $X{:}{=}{\mathbb {F}}^3$ and $Y{:}{=}{\mathbb {F}}^{k-2}$.

1.1 The $t=3$ Atrahasis code under centralized model

Say the fth and gth nodes fail and the first d nodes will help through an agent (that is, we assume the centralized model). Recall that, when the fth node is the only node that fails, it is supposed to learn $\phi \mathbin \upharpoonright X\otimes Y\odot Y\odot y_f$ from the help messages. For two failing nodes, there is a natural generalization: the agent needs to learn $\phi \mathbin \upharpoonright X\otimes Y\odot Y\odot \langle y_f,y_g\rangle$. This subspace has dimension $3(k-2)^2$. Here, we further assume that healthy nodes will communicate ahead and decide who should send which symbols, such that exactly $3(k-2)^2$ symbols will be sent in total. Regardless of which node should send what, we claim the bandwidth

$$\begin{aligned} \gamma _{\text {ce}}(2,d)=3(k-2)^2=3k^2-12k+12. \end{aligned}$$

According to [43], the least possible bandwidth is

$$\begin{aligned} \frac{2d}{d+2-k}\cdot \alpha =\frac{3(k-1)}{3(k-1)/2+2-k}\cdot \frac{(k-1)(k-2)}{2}=3k^2-15k+O(1). \end{aligned}$$

This indicates that Atrahasis is 3k away from optimality.

1.2 More failures and higher t under centralized model

When there are more nodes failing, the preceding argument generalizes to that the agent needs to learn $\phi \mathbin \upharpoonright X\otimes Y\odot Y\odot \langle y_{f_1},y_{f_2},\dotsc ,y_{f_c}\rangle$, where $f_1,f_2,\dotsc ,f_c$ are the indices of the failing nodes. The total bandwidth $\gamma _{\text {ce}}(c,d)$ is then the dimension of the subspace $X\otimes Y\odot Y\odot \langle y_{f_1},y_{f_2},\dotsc ,y_{f_c}\rangle$.

When $t>3$, the agent needs to learn $\phi \mathbin \upharpoonright X\otimes S^{t-1}Y\odot \langle y_{f_1},y_{f_2},\dotsc ,y_{f_c}\rangle$ and the total bandwidth $\gamma _{\text {ce}}(c,d)$ is the dimension of the subspace $\phi$ is restricted to.

Benchmarks

In Figs. 3, 4, 5, 6, 7, 8 and 9, we list some $\alpha$’s from this and various other works for some small parameters k and d. Compare them to the lower bounds in Fig. 3. See also Tables 1 and 2. Note that Atrahasis’s $\alpha$ does not depend on n while other works either stick to $n=d+1$ or have $\alpha \rightarrow \infty$ as $n\rightarrow \infty$. Finally, in the following manner our sub-packetization level is polynomial in Alrabiah–Guruswami’s lower bound (i.e., Theorem 3).

Theorem 26

Let $t=d/(d-k+1)$. Let $\alpha =\big (\begin{array}{c}{k-1}\\ {t-1}\end{array}\big )$. Then

$$\begin{aligned} \alpha \leqslant \min \bigl (2^{k-1},(k-1)^{t-1}\bigr ). \end{aligned}$$

Moreover, if $d,k,t,\alpha$ go to infinity with $d-k+1<C$ bounded, then

$$\begin{aligned} \alpha <\exp \Bigl (\frac{k-1}{4(d-k+1)}\Bigr )^{4C\log 2}. \end{aligned}$$

Fig. 3

Lower bounds of the sub-packetization level $\alpha$ from [2, Theorem 1]. The horizontal axis is k. The vertical axis is $d-k+1$ as the cases $d<k$ are meaningless and $d-(k-1)=d/t=(k-1)/(t-1)=\alpha /\beta$ bears more semantics than $d-k$ does. The plot omits $k=1$ for triviality. It is later discovered that their proof (and the bound) is not valid when $k=d$, the trivial case where $\alpha =1$ is easily achievable

Fig. 4

Upper bounds on $\alpha$ by Atrahasis code, our proposal of MSR codes with arbitrary (n, k, d). A box encloses the sub-packetization level $\alpha$ of the primitive construction at that point, which does not depend on n. Boxes on the same strip share the same t. From left to right $t=2,3,4,5,6$; the rest are omitted. One arrow means shortening once

Fig. 5

[29] gave MSR codes with parameter quadruple $(n,k,d,\alpha )=(sq,n-q,n-1,q^s)$ where q is a prime power. The $\alpha$’s of their primitive constructions are put in boxes as shown. One arrow means shortening once. The shaded region is where their $\alpha$ falls below ours

Fig. 6

[27] extended [29]’s result via providing MSR codes with parameters $(n,k,d,\alpha )=(sq,n-q-m,n-1-m,q^s)$. In other words, the former now allows $n>d+1$. We plot the $\alpha$’s when $m=1$ (i.e., when $n=d+2$) and shade the area when their $\alpha$ falls below ours. Note that for any fixed k, d such that $k<d$, their $\alpha$ keeps increasing as $m=n-1-d\rightarrow \infty$, and eventually exceeds our $\alpha$. (Graphically speaking, the boxes keep moving toward left and disappear as they hit $k=1$, while the shaded region “retreat” toward right to infinity)

Fig. 7

[40] contributed MSR codes with parameters $(n,k,d,\alpha )=(k+r,(r+1)m,n-1,r^m)$. They only enforce the optimal repair bandwidth for systematic nodes. This results in the least possible $\alpha$ among all MSR-related codes we have seen

Fig. 8

[17] gave codes with parameters $(n,k,d,\alpha )=\biggl (k+r,k,k+\rho +1,\rho ^{k\big (\begin{array}{c}{r}\\ {\rho }\end{array}\big )}\biggr )$. They only enforce the optimal repair bandwidth for systematic nodes. Note that their $\alpha$’s depend on n beyond k, d. When $n=d+1$, it coincides with Fig. 5. We display the $n=d+2$ case here. It was remarked that their $\alpha$’s could be optimized further but we decided to print the very $\alpha$ given therein. Note that [17] is chronologically before [27]

Fig. 9

[15] provided regenerating codes with arbitrary parameters (n, k, d). These include MSR codes, whose $\alpha$ are shown above, with $(n,k,d,\alpha )=(n,k,d,(d-k+1)^k)$. Their $\alpha$’s are higher than ours in general. The same parameters can be achieved using other constructions; see [10, 13]