1 Introduction

In combinatorial topology, shelling of a simplicial complex is a very useful and important notion that has been well studied.

Definition 1

An (abstract) simplicial complex \(\Delta \) is called pure if all of its maximal simplicies have the same dimension. Given a finite (or countably infinite) simplicial complex \(\Delta \) that is pure of dimension d, a shelling is a total ordering of its maximal simplicies \(C_1,C_2,\ldots \) such that for every \(k>1\), \(C_k\cap \left( \bigcup _{i=1}^{k-1}C_i\right) \) is pure of dimension \(d-1\). A simplicial complex that admits a shelling is called shellable.

Shellable complexes enjoy many strong algebraic and topological properties. For example, a shellable complex is homotopy equivalent to a wedge sum of spheres; thus, its homology groups (over \(\mathbb {Z}\)) are torsion free. The study of shellability in its combinatorial aspects has turned out to be very fruitful as well. The arguably earliest notable result that polytopes are shellable is due to Brugesser and Mani (Sect. 8 of [12]). Later on, Björner and Wachs developed theories on lexicographic shellability (Sect. 12 of [8]). In particular, they introduced powerful notions of EL- and CL-shellability to study graded posets whose order complexes are shellable. In a recent work, Goaoc et al. [5] proved that testing shellability is NP-complete.

A great deal of work has been done in understanding properties of shellable complexes and identifying classes of shellable complexes; however, little work has been done in counting the number of shellings of a fixed simplicial complex. One result on the number of shellings of the complete graphs is given by Stanley [9]. It is generally believed that if a simplicial complex is shellable, then it usually admits a lot of shellings, but no precise arguments have been given.

In this paper, we investigate the problem of counting shellings, aiming to start a new line of research. We restrict our attention to finite simplicial complexes that are pure of dimension 1, namely, undirected graphs, where interesting combinatorial arguments are already taking place. Let’s first reformulate Definition 1 in the language of graph theory.

Definition 2

(Graph Shelling) Given an undirected graph \(G=(V,E)\), where V is the vertex set of G and E is the edge set of G, a shelling of G is a total ordering of the edge set \(\sigma \in \mathfrak {S}_E\), where \(\mathfrak {S}\) stands for symmetric group, such that \(\sigma (1),\ldots ,\sigma (k)\) form a connected subgraph of G for all \(k=1,\ldots ,|E|\).

Throughout this paper, we will use F(G) to denote the number of shellings of a graph G.

Clearly, a graph admits a shelling if and only if it is connected, which is equivalent to \(F(G)>0\). A few results are already known.

Theorem 1

([9]) Let \(K_n\) be the complete graph on n vertices. Then,

$$\begin{aligned} F(K_n)=\frac{2^{n-2}}{C_{n-1}}\left( {\begin{array}{c}n\\ 2\end{array}}\right) ! \end{aligned}$$

where \(C_{n-1}=\left( {\begin{array}{c}2n-2\\ n-1\end{array}}\right) /n\) is the \((n-1)^{th}\) Catalan number.

As an overview for the paper, in Sect. 2, we will give an explicit formula for the number of shellings of complete bipartite graphs, resolving a MathOverflow question [10] posed by Sebastien Palcoux; in Sect. 3, we will provide exact formula for the number of shellings of trees, which is closely related to the number of linear extensions of posets, and utilize the formula to obtain some upper and lower bounds for them.

2 Complete bipartite graphs

Denote \(K_{m,n}\) as the complete bipartite graph with part sizes m and n. The following is our main theorem.

Theorem 2

For all positive integers m and n, the number of shellings of \(K_{m,n}\) is given by

$$\begin{aligned} F(K_{m,n})=\frac{m!n!(mn)!}{(m+n-1)!}. \end{aligned}$$

Remark 1

Theorem 2 can also be rephrased as \(F(K_{m,n})=(mn)!\cdot (m+n)/{m+n\atopwithdelims ()n}\). In other words, there is a \((m+n)/{m+n\atopwithdelims ()n}\) probability that a uniformly random edge ordering is a shelling order.

The formula in Theorem 2 is conjectured in the MathOverflow post [10]. Partial progress has been made: Lemma 1, given by Richard Stanley [10], serves as an important tool for our computation.

Lemma 1

\(F(K_{m,n})\) is equal to the following expression:

$$\begin{aligned} m!n!(mn-1)!\sum _{\alpha }\frac{b_{1}b_{2}\cdots b_{m+n-2}}{b_{m+n-2}(b_{m+n-2}+b_{m+n-3})\cdots (b_{m+n-2}+b_{m+n-3}+\ldots +b_1)}, \end{aligned}$$

where the sum is over all sequences \(\alpha = (a_1,a_2,\ldots ,a_{m+n-2})\) of \((m-1)\) 0’s and \((n-1)\) 1’s, and

$$\begin{aligned} b_{i} = 1 + |\{1\le j\le i: a_j \ne a_i\}|. \end{aligned}$$

Proof

Let \(\sigma \) be a shelling of \(K_{m,n}\). In each part of \(K_{m,n}\), consider the order of the appearance of the vertices. Here, we say that vertex v appears in \(\sigma \) at time t if t is the first index such that \(v\in \sigma (t)\). There are m! ways to choose such order in the part of size m and n! ways in the part of size n. Fix the order of vertex appearance in each part to be \((u_0,u_1,\ldots ,u_{m-1})\) and \((v_0,v_1,\ldots ,v_{n-1})\), respectively.

Consider a fixed order of appearance of all \((m+n)\) vertices \(w = w_{-1}w_0\ldots w_{m+n-2}.\) Note that \(\sigma (1)\) must be the edge \(e_0=(u_0,v_0)\), so \(\{w_{-1},w_0\} = \{u_0,v_0\}.\) For \(1\le i\le m+n-2\), define

$$\begin{aligned} a_{i} = {\left\{ \begin{array}{ll} 0, &{} \text { if } w_{i} = u_j \text { for some }j, \\ 1, &{} \text { if } w_{i} = v_k \text { for some }k, \end{array}\right. } \end{aligned}$$

and

$$\begin{aligned} b_i = 1 + |\{1\le j\le i: a_j \ne a_i\}|. \end{aligned}$$

Now, for each \(w_i\) (\(i\ge 1\)), consider the first edge \(e_i\) incident to \(w_i\) in \(\sigma \). This edge must be of the form \((w_i,w_j)\) where \(j<i\) and \(w_i, w_j\) are in different parts of \(K_{m,n}\). There are \(b_{i}\) choices for this edge, where the 1 in the definition of \(b_i\) refers to the edge connecting to either \(u_0\) or \(v_0\). Thus, there are \(b_1b_2\cdots b_{m+n-2}\) ways to choose \(e_1,e_2,\ldots ,e_{m+n-2}.\)

Fix these edges \(e_0,e_1,\ldots ,e_{m+n-2}\). Note that the rest of the \(b_{m+n-2}\) edges incident to \(w_{m+n-2}\) must appear after \(e_{m+n-2}\) in \(\sigma \), so there are \((b_{m+n-2}-1)!\) ways to arrange these edges. After making this arrangement, the edges which are incident to \(w_{m+n-3}\) and not yet arranged must appear after \(e_{m+n-3}\), so there are

$$\begin{aligned} (b_{m+n-2}+1)(b_{m+n-2}+2)\cdots (b_{m+n-2}+b_{m+n-3}+1) = \frac{(b_{m+n-2}+b_{m+n-3}+1)!}{b_{m+n-2}!} \end{aligned}$$

ways to arrange them (since there are already \(b_{m+n-2}\) edges arranged after \(e_{m+n-3}\)). Similarly, for each i, after making the arrangement of all edges incident to vertices appearing after \(w_i\), there are

$$\begin{aligned} \frac{(b_{m+n-2}+b_{m+n-3}+\ldots + b_i + 1)!}{(b_{m+n-2}+b_{m+n-3}+\ldots + b_{i+1})!} \end{aligned}$$

ways to arrange all the edges which are incident to \(w_i\) and not yet arranged. Therefore, after fixing \(e_0,e_1,\ldots ,e_{m+n-2}\), the number of shellings is

$$\begin{aligned}&\prod _{i=1}^{m+n-2}\frac{(b_{m+n-2}+\ldots + b_i + 1)!}{(b_{m+n-2}+\ldots + b_{i+1})!} \\&\quad = \frac{(mn-1)!}{b_{m+n-2}(b_{m+n-2}+b_{m+n-3})\cdots (b_{m+n-2}+\ldots +b_1)}. \end{aligned}$$

Combining all discussions above, we obtain Lemma 1. \(\square \)

We will now compute the expression in Lemma 1 explicitly to finish the proof of Theorem 2.

Proof of Theorem 2

For \(m,n\in \mathbb {Z}_{\ge 0}\), let’s define the following sum

$$\begin{aligned} S(p,q,m,n):=\sum _{\alpha }\frac{b_1b_2\cdots b_{m+n}}{b_{m+n}(b_{m+n}+b_{m+n-1})\cdots (b_{m+n}+\cdots +b_1)}, \end{aligned}$$

over all sequences \(\alpha = (a_1,a_2,\ldots ,a_{m+n})\) of m 0’s and n 1’s, and

$$\begin{aligned} b_{i} = {\left\{ \begin{array}{ll} q + |\{1\le j\le i: a_j=1\}|,&{}\quad a_i=0\\ p + |\{1\le j\le i: a_j=0\}|,&{}\quad a_i=1 \end{array}\right. }. \end{aligned}$$

Visually (see Fig. 1), the sum in S(p, q, m, n) goes through all lattice paths from (0, 0) to (m, n), and \(b_i\) is the length of the (horizontal or vertical) strip formed by the \(i^{th}\) step of the path with the boundary \(x=-q\) and \(y=-p\).

Fig. 1
figure 1

A visualization for the sum S(p, q, m, n)

Full size image

By Lemma 1, it suffices to compute \(S(1,1,m-1,n-1)\). As a piece of notation, for \(n\in \mathbb {Z}_{\ge 0}\), write \(t^{(n)}\) to denote \(t\cdot (t+1)\cdots (t+n-1)\) where \(t^{(0)}=1\). We are now going to use induction on \(m+n\) to show the following claim:

$$\begin{aligned} S(p,q,m,n)=\frac{1}{m!n!}\frac{(p+q)^{(m)}(p+q)^{(n)}}{(p+q)^{(m+n)}}. \end{aligned}$$

The base case is \(m=0\) or \(n=0\). If \(m=0\), then the sum in S(p, q, m, n) has only one term with \(b_1=b_2=\cdots =b_n=q\). We then obtain \(S(p,q,0,n)=1/n!\) as desired. The case \(n=0\) is similar. Now fix \(m,n>0\) and assume that the claim is true for \(S(p,q,m',n')\), where \(m'+n'<m+n\) and p, q are arbitrary.

Consider S(p, q, m, n). For a sequence \(\alpha =(\alpha _1,\ldots ,\alpha _{m+n})\) in the summand, either \(\alpha _1=0\) or \(\alpha _1=1\). Notice that \(b_{m+n}+\cdots +b_1=(m+p)(n+q)-pq=mn+pn+qm\). If \(\alpha _1=0\), then \(b_1=q\) and

$$\begin{aligned}\frac{b_2b_3\cdots b_{m+n}}{b_{m+n}(b_{m+n}+b_{m+n-1})\cdots (b_{m+n}+\cdots +b_2)} \end{aligned}$$

is a summand in \(S(p+1,q,m-1,n)\). We can similarly analyze the case \(\alpha _1=1\) and obtain the following recursive formula

$$\begin{aligned} S(p,q,m,n)=&\frac{q}{mn+pn+qm}S(p+1,q,m-1,n)\\&+\frac{p}{mn+pn+qm}S(p,q+1,m,n-1). \end{aligned}$$

By the induction hypothesis, the above calculation goes on as follows

$$\begin{aligned} =&\frac{qS(p+1,q,m-1,n)+pS(p,q+1,m,n-1)}{mn+pn+qm}\\ =&\frac{qm(p+q+1)^{(m-1)}(p+q+1)^{(n)}+pn(p+q+1)^{(m)}(p+q+1)^{(n-1)}}{(mn+pn+qm)m!n!(p+q+1)^{(m+n-1)}}\\ =&\frac{qm(p+q)^{(m)}(p+q+1)^{(n)}+pn(p+q+1)^{(m)}(p+q)^{(n)}}{(mn+pn+qm)m!n!(p+q)^{(m+n)}}\\ =&\frac{(p+q)^{(m)}(p+q)^{(n)}}{m!n!(p+q)^{(m+n)}}\frac{1}{mn+pn+qm}\left( \frac{(qm)(p+q+n)}{p+q}+\frac{(pn)(p+q+m)}{p+q}\right) \\ =&\frac{(p+q)^{(m)}(p+q)^{(n)}}{m!n!(p+q)^{(m+n)}}. \end{aligned}$$

Therefore, the induction step goes through and the claim is established.

In particular, by Lemma 1,

$$\begin{aligned} F(K_{m,n})=&m!n!(mn-1)!S(1,1,m-1,n-1)\\ =&m!n!(mn-1)!\frac{m!n!}{(m-1)!(n-1)!(m+n-1)!}\\ =&\frac{m!n!(mn)!}{(m+n-1)!}. \end{aligned}$$

\(\square \)

Remark 2

An earlier version of this paper goes from Lemma 1 to Theorem 2 via heavy calculation of algebraic expressions. The current proof presented here is much shorter but a bijective proof would be very desirable. A similar proof was provided in the MathOverflow post [10] by Fedor Petrov after this paper was submitted.

3 Trees

Trees are one of the most fundamental types of graphs. However, unlike the complete bipartite graph case, there is no simple formula for tree shelling numbers. The goal of this section is to give a relatively easy method to compute the number of shellings of a tree and to provide upper and lower bounds of this number.

In the process, we will see intimate connections between the number of shellings of a tree and the number of linear extensions of tree posets, which is a much better explored subject in the literature.

Throughout this section, let T be a tree with n vertices and \(n-1\) edges.

3.1 Tree shelling number computation

We first focus on computing the number of shellings of rooted trees, whose definition is given below.

Definition 3

Let v be a vertex of T. The rooted tree induced by T and rooted at v is denoted as \(T_v\). A shelling of a rooted tree \(T_v\) is a shelling \(\sigma \) of T such that \(\sigma (1)\) is an edge incident to v.

The following definitions are used to efficiently describe structures in a (rooted) tree.

Definition 4

Let \(T_v\) be a tree rooted at vertex v. We say a vertex u is a parent of vertex w (and w is a child of u) if (w, u) is an edge and u lies closer to the root than w. A descending path from u to w in the rooted tree \(T_v\) is a structure

$$\begin{aligned} u-v_1-v_2-\cdots -v_r-w \end{aligned}$$

where each vertex is a parent of the subsequent vertex. We say u is an ancestor of w (and w is a descendant of u) if there exists a descending path from u to w.

Definition 5

Let \(u,v\in T.\) The (rooted) subtree of \(T_v\) rooted at u, denoted as \(T_v(u)\), is a subgraph of T rooted at u and induced by the set of vertices

$$\begin{aligned} \{w\in T: w \text { is a descendant of } u \text { in }T_v\}. \end{aligned}$$

See Fig. 2 for an example.

Fig. 2
figure 2

Definition of \(T_v(u)\)

Full size image

For a tree T, the edge set of T is denoted as E(T). The vertex set of T is denoted as V(T), or T for simplicity. Accordingly, |T| is the number of vertices in T. The same notations are used for rooted trees.

The following proposition provides a way to calculate the number of shellings of a rooted tree \(T_v\) based on the size of its rooted subtrees.

Proposition 1

Let T be a tree on n vertices and \(v\in V(T)\). Then,

$$\begin{aligned} F(T_v) = \frac{n!}{\prod _{u\in T} |T_v(u)|}. \end{aligned}$$

Proposition 1 is, in fact, Knuth’s hook length formula [7] for the number of linear extensions of a partially ordered set whose Hasse diagram is the rooted tree \(T_v\). Recall that a linear extension of a poset P on n elements is a bijection \(\beta :P\rightarrow \{1,2,\ldots ,n\}\) such that \(\beta (x)<\beta (y)\) if \(x<y\) in P. Specifically, shellings of \(T_v\) are in simple bijection with linear extensions of the poset \(P_v\) where \(u<w\) if u is a descendent of w in \(T_v\): given a shelling \(\alpha \) of \(T_v\), we can construct a linear extension \(\beta :P\rightarrow \{1,\ldots ,n\}\) such that \(\beta (v)=n\) and \(\beta ^{-1}(n-1),\ldots ,\beta ^{-1}(1)\) is the order of appearances of vertices \(P_v\setminus \{v\}\) provided by \(\alpha \).

Because of this correspondence, we omit the proof of Proposition 1, which is in the literature and is straightforward via induction. Knuth’s hook length formula for trees has multiple q-analogues given by Björner and Wachs [2] and a further multivariate generalization by Hivert and Reiner [6]. Viewing such linear extensions of tree posets as intervals in the weak Bruhat orders, we see that those intervals are called “forest quotients in the symmetric group” in the language of generalized quotients developed by Björner and Wachs [1] and that they are nice intervals to provide splittings of the Coxeter groups [3, 4].

For our purposes to compute F(T), however, the root of the tree is constantly moving. Thus, we need new tools for our estimation.

Corollary 1

Suppose that (u, v) is an edge of T, then

$$\begin{aligned} \frac{F(T_v)}{F(T_u)} = \frac{|T_u(v)|}{|T_v(u)|} = \frac{|T_u(v)|}{n-|T_u(v)|}. \end{aligned}$$

Proof

For any vertex \(w\ne u,v\), \(T_u(w)\) and \(T_v(w)\) are the same subtree of \(T_v\). Therefore, by Proposition 1,

$$\begin{aligned} \frac{F(T_v)}{F(T_u)}&= \frac{\prod _{w\in T} |T_u(w)|}{\prod _{w\in T} |T_v(w)|} = \frac{|T_u(v)|\cdot |T_u(u)|}{|T_v(u)|\cdot |T_v(v)|} \\&= \frac{|T_u(v)|}{|T_v(u)|} = \frac{|T_u(v)|}{n-|T_u(v)|}. \end{aligned}$$

\(\square \)

Corollary  1 establishes a simple relationship between the number of shellings of T rooted at adjacent vertices. In this way, by only calculating \(F(T_v)\) for a single vertex v, one can quickly derive \(F(T_u)\) for all \(u\in T.\) For example, suppose T is a path of length \(n-1\), as shown in Fig. 3. Then, \(F(T_{v_1}) = 1\), and

$$\begin{aligned}F(T_{v_{i+1}}) = \frac{|T_{v_i}(v_{i+1})|}{n-|T_{v_i}(v_{i+1})|}F(T_{v_i}) = \frac{n-i}{i}F(T_{v_i}) \end{aligned}$$

by Corollary 1. This gives \(F(T_{v_i}) = \left( {\begin{array}{c}n-1\\ i-1\end{array}}\right) \) for all \(i=1,2,\ldots ,n.\)

Fig. 3
figure 3

A path of length \(n-1\). The shelling number is \(2^{n-2}\)

Full size image

Finally, the following proposition relates the number of shellings of T with that of its rooted trees.

Proposition 2

$$\begin{aligned} F(T) = \frac{1}{2}\sum _{v\in T} F(T_v). \end{aligned}$$

Proof

Note that any shelling of T beginning with edge (u, v) is counted as a shelling of both \(T_u\) and \(T_v\). Thus, Proposition 2 follows. \(\square \)

Recall that \(F(T_v)=e(P_v)\) where \(P_v\) is the poset where \(u<w\) if u is a descendent of w and e denotes the number of linear extensions of a poset. In other words, Proposition 2 is saying that \(F(T)=\frac{n}{2}\mathbb {E}[e(P_v)]\) where the root is chosen uniformly at random. Equivalently, in the rest of the paper, we will be providing bounds for the expectation \(\mathbb {E}[e(P_v)]\), which is a novel statistic on trees.

Example 1

By Proposition 2 and the discussion under Corollary 1, the number of shellings of a path of length \(n-1\) is

$$\begin{aligned} \frac{1}{2}\sum _{i=1}^n \left( {\begin{array}{c}n-1\\ i-1\end{array}}\right) = 2^{n-2}. \end{aligned}$$

3.2 Bounds on tree shelling number

The goal of this section is to give several bounds of tree shelling numbers based on various parameters of a graph, such as vertex degree and diameter (the diameter of a connected graph is the greatest distance between any pairs of vertices). A trivial upper bound is \((n-1)!\), since every shelling is also a permutation of edges. The upper bound is achieved when T is a star, in which every two edges are adjacent to each other.

Here are the main theorems of the section.

Theorem 3

Let T be a tree. For each \(v\in V(T)\), let d(v) denote its degree. Then,

$$\begin{aligned} F(T) \ge \prod _{v\in T} d(v)!. \end{aligned}$$

The equality holds if and only if T is a path of length \(n-1\) or a star.

Remark 3

A weaker lower bound \(F(T)\ge \prod _{v\in T}\big (d(v)-1\big )!\) is easy to see. However, an extra factor of \(\prod _{v\in T}d(v)\) in Theorem 3 requires much more effort.

Theorem 4

Suppose the diameter of T is \(\ell \). When \(\ell \) is even,

$$\begin{aligned} F(T)\le \frac{2(n-1-\frac{\ell }{2})!}{(\frac{\ell }{2})!}\bigg [\left( {\begin{array}{c}n-2\\ \frac{\ell }{2}\end{array}}\right) +\sum _{i=0}^{\frac{\ell }{2}-1}\left( {\begin{array}{c}n-1\\ i\end{array}}\right) \bigg ]. \end{aligned}$$

When \(\ell \) is odd,

$$\begin{aligned} F(T) \le \frac{(n-\frac{\ell +3}{2})!}{(\frac{\ell +1}{2})!}\bigg [(n-1-\ell )\left( {\begin{array}{c}n-2\\ \frac{\ell -1}{2}\end{array}}\right) +n\sum _{i=0}^{\frac{\ell -1}{2}}\left( {\begin{array}{c}n-1\\ i\end{array}}\right) \bigg ]. \end{aligned}$$

The equality holds if and only if T has the following form: there exists a path

$$\begin{aligned}v_0 - v_1 - \cdots -v_\ell \end{aligned}$$

such that every edge not in this path is adjacent to \(v_{\lfloor \frac{\ell }{2}\rfloor }.\)

Before proving Theorem 3, it is worth noticing the following inequality, which relates the number of shellings of T and \(T_v\).

Lemma 2

Let v be a vertex in T and \(\ell \) be the length of the longest descending path in \(T_v\). Then,

$$\begin{aligned} F(T)\le \bigg [\sum _{k=0}^{\ell -1}\left( {\begin{array}{c}n-2\\ k\end{array}}\right) \bigg ] F(T_v). \end{aligned}$$

In particular, \(F(T)\le 2^{n-2}F(T_v).\)

Proof

Let \(L = v - v_1 - v_2 -\cdots - v_\ell \) be the longest descending path in \(T_v\). Perform the following operation on T until it can be performed no longer:

  • Suppose \(i\le \ell -2\) is the first index such that \(v_i\) has a child \(v'\ne v_{i+1}\) in \(T_v\). Remove \(T_v(v')\) and attach it on \(v_{i+1}\) (i.e., children of \(v'\) become children of \(v_{i+1}\)). Furthermore, remove edge \((v',v_i)\) and add a new edge \((v',v_{i+1}).\) This operation is illustrated in Fig. 4.

Fig. 4
figure 4

Operation on T: moving edges away from the root

Full size image

Such operations preserve the length of the longest descending path in \(T_v\) and would eventually stop within a finite number of steps: the sum of distances from all vertices to \(v_{\ell }\) strictly decreases after each step. Let \(T^{(k)}\) be the tree after kth operation. For each \(u\in T\), define the weight of u in \(T^{(k)}\)

$$\begin{aligned} W_k(u) = \frac{F(T^{(k)}_u)}{F(T^{(k)}_v)}. \end{aligned}$$

We claim that the sum of weights of all vertices is non-decreasing after each operation, i.e.,

$$\begin{aligned} \sum _{u\in T}W_k(u) \le \sum _{u\in T}W_{k+1}(u). \end{aligned}$$
(1)

It suffices to prove the claim for \(k=0.\) By Corollary 1, suppose (u, w) is an edge in \(T^{(0)}\), then

$$\begin{aligned} \frac{W_0(u)}{W_0(w)} = \frac{|T^{(0)}_w(u)|}{|T^{(0)}_u(w)|} = \frac{|T^{(0)}_w(u)|}{n-|T^{(0)}_w(u)|}. \end{aligned}$$
(2)

Therefore, suppose \(v-u_1-u_2-\cdots - u_r = u\) is a path in \(T^{(0)}_v\), then

$$\begin{aligned} W_0(u) = \prod _{j=1}^r \frac{|T^{(0)}_v(u_j)|}{n-|T^{(0)}_v(u_j)|}. \end{aligned}$$

Note that for all \(u\not \in \{v', v_{i+1}\}\), \(|T^{(1)}_v(u)| = |T_v(u)|\). For each \(w\not \in T_v(v')\cup T_v(v_{i+1})\), \(v',v_{i+1}\) are not on the path from v to w, so

$$\begin{aligned} W_0(w) = W_1(w). \end{aligned}$$

Write \(|T_v(v'))|=a, |T_v(v_{i+1})| = b\), then \(|T^{(1)}_v(v')| = 1\), \(|T^{(1)}_v(v_{i+1})| = |T_v(v')|+|T_v(v_{i+1})| = a+b.\) By (2),

$$\begin{aligned}&W_0(v') = \frac{a}{n-a}W_0(v_i).\\&W_0(v_{i+1}) = \frac{b}{n-b}W_0(v_i).\\&W_1(v_{i+1}) = \frac{a+b}{n-a-b}W_1(v_i).\\ \end{aligned}$$

Since \(W_0(v_i) = W_1(v_i)\),

$$\begin{aligned} W_0(v')+W_0(v_{i+1})\le W_1(v_{i+1}). \end{aligned}$$

For each \(w\in T_v(v')\setminus \{v'\}\), by (2),

$$\begin{aligned} \frac{W_1(w)}{W_1(v_{i+1})} = \frac{W_0(w)}{W_0(v')} \implies W_0(w)\le W_1(w). \end{aligned}$$

Similarly, for each \(w\in T_v(v_{i+1})\setminus \{v_{i+1}\}\),

$$\begin{aligned} \frac{W_1(w)}{W_1(v_{i+1})} = \frac{W_0(w)}{W_0(v_{i+1})}\implies W_0(w)\le W_1(w). \end{aligned}$$

Therefore, we conclude that

$$\begin{aligned} \sum _{w\in T} W_0 (w)\le \sum _{w\in T} W_1(w), \end{aligned}$$

and (1) is proved.

Finally, suppose the operation stops after step M, then \(T^{(M)}\) is the tree where all vertices not in L are incident to \(v_{\ell -1}\). Thus, by (2),

$$\begin{aligned} \sum _{u\in T} W_{M}(u)&= W_M(v)+W_M(v_1)+\cdots +W_M(v_{\ell -1}) + (n-\ell )W_M(v_\ell ) \\&=\sum _{i=0}^{\ell -1}\left( {\begin{array}{c}n-1\\ i\end{array}}\right) + (n-\ell )\frac{\left( {\begin{array}{c}n-1\\ \ell -1\end{array}}\right) }{n-1}\\&=2\sum _{i=0}^{\ell -1}\left( {\begin{array}{c}n-2\\ i\end{array}}\right) , \end{aligned}$$

where the last equality uses the identity

$$\begin{aligned} \left( {\begin{array}{c}n-1\\ i\end{array}}\right) = \left( {\begin{array}{c}n-2\\ i\end{array}}\right) + \left( {\begin{array}{c}n-2\\ i-1\end{array}}\right) \end{aligned}$$

for all \(i\ge 1\). According to equation (1), Proposition 2,

$$\begin{aligned} \frac{F(T)}{F(T_v)} = \frac{1}{2}\sum _{u\in T}W_0(u) \le \frac{1}{2}\sum _{u\in T}W_M(u) = \sum _{i=0}^{\ell -1}\left( {\begin{array}{c}n-2\\ i\end{array}}\right) , \end{aligned}$$

so the proof is complete. \(\square \)

Now we are ready to prove Theorem 3 and 4.

Proof of Theorem 3

Induct on |V(T)|. When \(|V(T)| = 2\), \(F(T) = 2 = \prod _{v\in T} d(v)!\). In this case, T is both a path and a star, which justifies the case of equality.

Let \(n>2\). Assume the claim holds for all trees with fewer than n vertices and let T be a tree with n vertices. If T is a path of length \(n-1\), then by Example 1,

$$\begin{aligned} F(T) = 2^{n-2} = \prod _{v\in T} d(v)!, \end{aligned}$$

as desired.

Suppose T is not a path. Then, there exists a vertex v of degree \(d\ge 3.\) Let \(u_1,u_2,\ldots ,u_d\) be the vertices adjacent to v and write \(|T_v(u_i)| = s_i\) for \(i=1,2,\ldots ,d\). Assume \(s_1\le s_2\le \ldots \le s_d.\) Let \(T'\) be the subtree of T obtained by removing all vertices in \(T_v(u_1)\) and all edges incident to those vertices. Let \(T''\) be the subtree of T induced by edges in \(E(T)\setminus E(T').\) See Fig. 5 for illustration.

Fig. 5
figure 5

Merging a shelling of \(T'\) and \(T''_v\) to a shelling of T

Full size image

Suppose \(\sigma '\) is a shelling of \(T'\) and \(\sigma ''\) a shelling of \(T''_v\). Merge \(\sigma '\) and \(\sigma ''\) into a new permutation \(\sigma \) of E(T) such that (i) the order of edges in \(\sigma '\) and in \(\sigma ''\) are preserved; (ii) \(\sigma ''(1) = (v,u_1)\) is not one of the first \(s_d\) edges after merge. Note that \(\sigma \) must be a shelling of T, since at least one of \(\{\sigma '(k): 1\le k\le s_d\}\) is incident to v and \((v,u_1)\) is adjacent to some previous edges in \(\sigma \).

For each fixed \(\sigma '\) and \(\sigma ''\), the number of \(\sigma \) constructed by the above merging method is

$$\begin{aligned} \left( {\begin{array}{c}n-1-s_d\\ |E(T'')|\end{array}}\right) = \left( {\begin{array}{c}s_1+s_2+\cdots +s_{d-1}\\ s_1\end{array}}\right) . \end{aligned}$$

Therefore,

$$\begin{aligned} F(T) \ge F(T')F(T''_v)\left( {\begin{array}{c}s_1+s_2+\cdots + s_{d-1}\\ s_1\end{array}}\right) . \end{aligned}$$
(3)

Note that the shellings of T constructed above do not include those whose first edge is \((v,u_1)\), so we can replace “\(\ge \)” with “>” in (3). Furthermore, by Lemma 2,

$$\begin{aligned} F(T''_v) \ge \frac{F(T'')}{2^{s_1-1}}. \end{aligned}$$

Note that v has degree \(d-1\) in \(T'\) and degree 1 in \(T''\), by the induction hypothesis,

$$\begin{aligned} F(T')F(T'') \ge \frac{1}{d}\prod _{u\in T} d(u)!. \end{aligned}$$

Thus, (3) implies

$$\begin{aligned} F(T) > \frac{1}{2^{s_1-1}d}\left( {\begin{array}{c}s_1+s_2+\cdots +s_{d-1}\\ s_1\end{array}}\right) \prod _{u\in T}d(u)!. \end{aligned}$$

If for some choices of v with degree \(d\ge 3\), \(\left( {\begin{array}{c}s_1+s_2+\cdots +s_{d-1}\\ s_1\end{array}}\right) \ge 2^{s_1-1}d\), then \(F(T) > \prod _{u\in T} d(u)!\) and equality never holds.

If not, for all choices of v, \(\left( {\begin{array}{c}s_1+s_2+\cdots +s_{d-1}\\ s_1\end{array}}\right) < 2^{s_1-1}d.\) By Lemma 3 in appendix, \(s_1 = s_2 = \cdots = s_{d-1} = 1.\) Therefore, T must be the following type of trees: for every vertex v of degree \(d(v) \ge 3\), it connects at least \(d(v) - 1\) leaves. If T is a star, then \(F(T) = (n-1)!\) is an equality case. If not, T has the form shown in Fig. 6, where \(v_0\) and \(v_m\) are the only two possible vertices with degree at least 3.

Fig. 6
figure 6

The only type of trees that satisfy case 2 condition

Full size image

Suppose \(d(v_0) = d_1\), \(d(v_m) = d_2\) with \(2\le d_1\le d_2\). If \(m=1\), by Proposition 1,

$$\begin{aligned} F(T_v) = {\left\{ \begin{array}{ll} \frac{(d_1+d_2-1)!}{d_2}, &{}\text { if }v = v_0, \\ \frac{(d_1+d_2-1)!}{d_1}, &{}\text { if }v = v_1, \\ \frac{(d_1+d_2-2)!}{d_2}, &{}\text { if }v \ne v_1, \text { and }(v,v_0)\in E(T), \\ \frac{(d_1+d_2-2)!}{d_1}, &{}\text { if }v \ne v_0, \text { and }(v,v_1)\in E(T).\\ \end{array}\right. } \end{aligned}$$

Therefore, by Proposition 1,

$$\begin{aligned} F(T)&= \frac{1}{2}\sum _{v\in T} F(T_v) \\&= \frac{(d_1+d_2-2)!}{2}\left( \frac{d_1+d_2-1}{d_2}+\frac{d_1+d_2-1}{d_1} + \frac{d_1-1}{d_2}+\frac{d_2-1}{d_1}\right) \\&=\frac{d_1^2+d_2^2+d_1d_2-d_1-d_2}{d_1d_2}(d_1+d_2-2)! \\&\ge 2\cdot (d_1+d_2-2)!\\&\ge d_1!d_2! = \prod _{u\in T} d(u)!, \end{aligned}$$

where the last two lines are due to Lemma 4 in appendix. The equality holds only if \(d_1 = d_2 = 2\) and T is a single path.

Now suppose \(m\ge 2.\) Consider the following type of shelling of T: The first \(m-1\) edges of \(\sigma \) consist of \(\{(v_i,v_{i+1}): 0\le i\le m-2\}\). The number of shellings of such type is

$$\begin{aligned} 2^{m-2}(d_1-1)!(d_2-1)!\left( {\begin{array}{c}d_1+d_2-1\\ d_2\end{array}}\right) . \end{aligned}$$

Similarly, the number of shellings whose first \(m-1\) edges consist of \(\{(v_i,v_{i+1}): 1\le i\le m-1\}\) is

$$\begin{aligned} 2^{m-2}(d_1-1)!(d_2-1)!\left( {\begin{array}{c}d_1+d_2-1\\ d_1\end{array}}\right) . \end{aligned}$$

Thus, we have

$$\begin{aligned} F(T)&\ge 2^{m-2}(d_1-1)!(d_2-1)!\bigg [\left( {\begin{array}{c}d_1+d_2-1\\ d_2\end{array}}\right) +\left( {\begin{array}{c}d_1+d_2-1\\ d_1\end{array}}\right) \bigg ] \\&= 2^{m-2}(d_1-1)!(d_2-1)!\left( {\begin{array}{c}d_1+d_2\\ d_1\end{array}}\right) . \end{aligned}$$

By Lemma 5 in appendix,

$$\begin{aligned} 2^{m-2}(d_1-1)!(d_2-1)!\left( {\begin{array}{c}d_1+d_2\\ d_1\end{array}}\right) > 2^{m-1}d_1!d_2! = \prod _{u\in T}d(u)! \end{aligned}$$

unless \(d_1=2, d_2\le 4\), in which cases we have:

  • \((d_1,d_2) = (2,2).\) \(F(T) = 2^{n-2} = \prod _{u\in T}d(u)!.\) In this equality case, T is a single path.

  • \((d_1,d_2) = (2,3).\) \(F(T) = 2^{n-1}-2 > 3!\cdot 2^{n-4} = \prod _{u\in T}d(u)!.\)

  • \((d_1,d_2) = (2,4).\) \(F(T) = 6(2^{n-2}-n+1) > 4!\cdot 2^{n-5} = \prod _{u\in T}d(u)!.\)

By induction, the proof of Theorem 3 is complete. \(\square \)

Proof of Theorem 4

Let \(v_0 - v_1 -\cdots - v_\ell \) be a longest path in T. Firstly, we reduce the problem to the case where all edges in T are incident to \(\{v_1,v_2,\ldots ,v_{\ell -1}\}\). If not, construct a new tree \(T'\) by removing every edge e not incident to \(\{v_i:1\le i\le \ell -1\}\) and adding a corresponding edge incident to \(v_j\), where \(v_j\) is the closest vertex from e among L. Every shelling of T is still a shelling of \(T'\) by considering the corresponding edges. Thus, \(F(T)\le F(T')\) while the longest path remains the same.

Under this assumption, denote \(V' = T\setminus \{v_0,v_1,\ldots ,v_\ell \}.\) Consider the following operations:

  1. 1.

    Let i be the smallest index such that \(v_i\) has degree \(\ge 3.\) If \(i<\frac{\ell }{2}\), we remove all edges of the form \((v_i,u)\) for \(u\in V'\) and add edges \((u,v_{i+1}).\)

  2. 2.

    Repeat step 1 until no further operations can be performed.

  3. 3.

    Let j be the largest index such that \(v_j\) has degree \(\ge 3.\) If \(j>\frac{\ell }{2}\), we remove all edges of the form \((v_j,u)\) for \(u\in V'\) and add edges \((u,v_{j-1}).\)

  4. 4.

    Repeat step 3 until no further operations can be performed.

Suppose the above operations end in step M. Let \(T^{(t)}\) be the tree after \(t^{\text {th}}\) operation. We claim that for all \(t<M\),

$$\begin{aligned} F(T^{(t+1)}) \ge F(T^{(t)}). \end{aligned}$$

It suffices to prove the case when \(t = 0.\) By symmetry, we can assume \(i<\frac{\ell }{2}\). Let \(V_i\) be the set of vertices adjacent to \(v_i\) in T except \(v_{i-1}, v_{i+1}\). Define

$$\begin{aligned} S_{T\cap T^{(1)}}&:= \{\sigma \text { is a shelling of }T: \forall u\in V_i, (v_i,u) \text { appears after }(v_i,v_{i+1}) \text { in } \sigma \}, \\ S_{T\cap T^{(1)}}^{(1)}&:= \{\tau \text { is a shelling of }T^{(1)}: \forall u\in V_i, (v_{i+1},u) \text { appears after }(v_i,v_{i+1}) \text { in } \tau \}, \\ S_{T\setminus T^{(1)}}&:= \{\sigma \text { is a shelling of }T: \exists u\in V_i, (v_i,u) \text { appears before }(v_i,v_{i+1}) \text { in } \sigma \},\\ S_{T^{(1)}\setminus T}^{(1)}&:= \{\tau \text { is a shelling of }T^{(1)}: \exists u\in V_i, (v_{i+1},u) \\&\qquad \text { appears before }(v_i,v_{i+1}) \text { in } \tau \}. \end{aligned}$$

Note that there is a bijection between \(S_{T\cap T^{(1)}}\) and \(S_{T\cap T^{(1)}}^{(1)}\) by replacing edges of the form \((v_i,u)\) in every \(\sigma \in S_{T\cap T^{(1)}}\) with \((v_{i+1},u)\), for all \(u\in V_i\). Thus, \(|S_{T\cap T^{(1)}}|=|S_{T\cap T^{(1)}}^{(1)}|\) and

$$\begin{aligned} F(T^{(1)}) - F(T) = |S_{T^{(1)}\setminus T}^{(1)}| - |S_{T\setminus T^{(1)}}|. \end{aligned}$$

Define a function \(g: S_{T\setminus T^{(1)}} \rightarrow S_{T^{(1)}\setminus T}^{(1)}\) as follows. Suppose \(\sigma \in S_{T\setminus T^{(1)}}\) and denote \(\tau = g(\sigma )\). If \(\sigma (k) = (v_i, u)\) for some \(u\in V_i\), \(\tau (k) = (v_{i+1}, u)\); if \(\sigma (k) = (v_j,u)\) for some \(j\ne i\) and \(u\in V'\), set \(\tau (k) = (v_j,u)\). It remains to define \(\tau (k)\)’s where \(\sigma (k)\) is an edge of L.

Write \(e(j) = (v_{j},v_{j+1})\) for \(j=0,1,\ldots ,\ell -1.\) For each r with \(1\le r\le \ell \), suppose \(\sigma (k_r) = e(j_r)\) where \(k_1<k_2<\cdots < k_\ell .\) Define \(\tau (k_r)\) inductively: when \(r=1\), \(\tau (k_1) = e(2i-j_1).\) When \(r\ge 2\),

$$\begin{aligned} \tau (k_{r}) = {\left\{ \begin{array}{ll} e(j_{r}), &{} \text { if } \{\tau (k_1), \tau (k_2),\ldots , \tau (k_{r-1})\} =\{e(j_1), e(j_2),\ldots , e(j_{r-1})\} \\ e(2i-j_{r}), &{} \text { otherwise.} \end{array}\right. } \end{aligned}$$

The idea is that g maps an edge not in L to itself or to the corresponding edge in \(T^{(1)}\setminus T\) (in the case that this edge is incident to \(v_i\) in T). For edges in L, g acts as a reflection with respect to e(i), until the reflection image matches with the preimage. An example of g is in Fig. 7.

Fig. 7
figure 7

An example of operation on T: moving edges toward middle. The shellings are indicated by the number on the edges. g maps a shelling of the first tree to a shelling of the second tree

Full size image

We check the following properties of g:

  • g is well defined.

    We first note that for any \(r\le \ell \), both \(\{\sigma (k_1),\sigma (k_2),\ldots ,\sigma (k_r)\}\) and \(\{\tau (k_1),\tau (k_2),\ldots ,\tau (k_r)\}\) form a path \(P_r\) and \(P^{(1)}_r\) in L, respectively. Since \(j_1 \le i\), the right endpoint of \(P^{(1)}_r\) is never on the left side of the right endpoint of \(P_r\) (assuming that L is a horizontal path with left endpoint \(v_0\) and right endpoint \(v_\ell \), as illustrated in Fig. 7). Furthermore, since the “branching edges” of T (edges in \(E(T)\setminus E(L)\)) are not on the left side of \(v_i\), every branching edge adjacent to \(P_r\) must be adjacent to \(P^{(1)}_r\). Thus, \(\tau \) is a shelling of \(T^{(1)}\). Moreover, \(\tau \in S_{T^{(1)}\setminus T}^{(1)}\) by the correspondence between \((v_i,u)\in \sigma \) and \((v_{i+1},u)\in \tau \) for all \(u\in V_i\). Therefore, g is well defined.

  • g is injective.

    Suppose \(g(\sigma ) = \tau .\) By the definition of g, \(\sigma (k)\) is uniquely determined whenever \(\tau (k) \not \in L.\) Suppose \(\tau (k_r) = e(i_r)\) for \(1\le r\le \ell \). we can recover \(\sigma (k_r)\) from \(\tau \): \(\sigma (k_1) = e(2i-i_1)\). When \(r\ge 2\),

    $$\begin{aligned}&\sigma (k_{r}) \\&\quad = {\left\{ \begin{array}{ll} e(i_{r}), &{} \text { if } \{\sigma (k_1), \sigma (k_2),\ldots , \sigma (k_{r-1})\} =\{e(i_1), e(i_2),\ldots , e(i_{r-1})\} \\ e(2i-i_{r}), &{} \text { otherwise.} \end{array}\right. } \end{aligned}$$

    Therefore, \(\sigma \) is uniquely determined by \(\tau \) and g is injective.

Since g is injective, \(|S_{T^{(1)}\setminus T}^{(1)}| \ge |S_{T\setminus T^{(1)}}|\) and thus \(F(T^{(1)}) \ge F(T).\)

Finally, note that \(T^{(M)}\) is the tree where all edges not in L are incident to \(v_{\lfloor \frac{\ell }{2}\rfloor }\). By Propositions 1 and  2,

$$\begin{aligned} F(T^{(M)})= {\left\{ \begin{array}{ll} \frac{2(n-1-\frac{\ell }{2})!}{(\frac{\ell }{2})!}\bigg [\left( {\begin{array}{c}n-2\\ \frac{\ell }{2}\end{array}}\right) +\sum _{i=0}^{\frac{\ell }{2}-1}\left( {\begin{array}{c}n-1\\ i\end{array}}\right) \bigg ], &{}\text { if }\ell \text { is even,} \\ \frac{(n-\frac{\ell +3}{2})!}{(\frac{\ell +1}{2})!}\bigg [(n-1-\ell )\left( {\begin{array}{c}n-2\\ \frac{\ell -1}{2}\end{array}}\right) +n\sum _{i=0}^{\frac{\ell -1}{2}}\left( {\begin{array}{c}n-1\\ i\end{array}}\right) \bigg ],&\text { if } \ell \text { is odd.} \end{array}\right. } \end{aligned}$$

Thus, the proof of inequality is complete.

Furthermore, we shall prove that g is surjective only if T is isomorphic to \(T^{(M)}.\) If not, then there are two cases:

Case 1. \(i< \frac{\ell -1}{2}\).

In this case, \(2i<\ell -1\). Thus, for every \(\sigma \in S_{T\setminus T^{(1)}}\), \(g(\sigma )(1)\ne e(\ell -1) = (v_{\ell -1},v_\ell )\). However, there exists \(\tau \in S_{T^{(1)}\setminus T}^{(1)}\) whose first edge is \((v_{\ell -1},v_{\ell })\), contradiction!

Case 2. \(i= \frac{\ell -1}{2}\) and there exists another vertex \(v_j\) of degree at least 3.

Suppose \((v_j, u)\) is an edge not in L, then for every \(\sigma \in S_{T\setminus T^{(1)}}\), \(g(\sigma )(1)\) cannot be this edge. However, there exists \(\tau \in S_{T^{(1)}\setminus T}^{(1)}\) whose first edge is \((v_j,u)\), contradiction!

Therefore, g is surjective only if T is isomorphic to \(T^{(M)}\), so

$$\begin{aligned}F(T) = F(T^{(M)}) \end{aligned}$$

if and only if T is isomorphic to \(T^{(M)}\). This completes the proof of Theorem 4. \(\square \)