1 Introduction

Let H be a finite group, and let R, L and S be three subsets of H such that \(R^{-1}=R\), \(L^{-1}=L\), and \(R\cup L\) does not contain the identity element 1 of H. The Cayley graph of H relative to the subset R,  denoted by \(\mathrm{Cay}(H,R),\) is the graph having vertex set H,  and edge set \(\{ \{h,xh\} : x \in R, h \in H \}\), and the bi-Cayley graph of H relative to the triple (RLS), denoted by \(\mathrm{BiCay}(H, R, L, S)\), is the graph having vertex set the union of the right part \(H_0=\{h_0 : h\in H\}\) and the left part \(H_1=\{h_1 : h\in H\}\), and edge set being the union of the following three sets

  • \(\big \{ \{h_0,(xh)_0\} : x \in R, \, h \in H \big \}\) (right edges),

  • \(\big \{ \{h_1,(xh)_1\}: x \in L, \, h \in H \big \}\) (left edges),

  • \(\big \{ \{h_0,(xh)_1\} : x \in S, \, h \in H\big \}\) (spokes).

In the particular case when \(R=L=\emptyset ,\) the bi-Cayley graph \(\mathrm{BiCay}(H, \emptyset , \emptyset , S)\) is also known as a Haar graph, denoted by \(\mathrm{H}(H,S)\). This name is due to Hladnik et al. [12], who studied Haar graphs of cyclic groups.

Symmetries of Cayley graphs have always been an active topic among algebraic combinatorists, and lately, the symmetries of bi-Cayley graphs have also received considerable attention. For various results and constructions in connection with bi-Cayley graphs and their automorphisms, we refer the reader to [1, 2, 5, 8, 17, 19, 30, 31] and all the references therein. In particular, Estélyi and Pisanski [8] initiated the investigation the relationship between Cayley graphs and Haar graphs. A Cayley graph is a Haar graph exactly when it is bipartite, but no simple condition is known for a Haar graph to be a Cayley graph. An elementary argument shows that every Haar graph of every abelian group is also a Cayley graph. (This also follows from Proposition 2.1.) On the other hand, Lu et al. [18] have constructed cubic semisymmetric graphs, that is, edge-transitive but not vertex-transitive graphs, as Haar graphs of alternating groups. Clearly, as these graphs are not vertex-transitive, they are examples of Haar graphs which are not Cayley graphs. Motivated by these observations, the following problem was posed.

Problem 1.1

([8, Problem 1]) Determine the finite non-abelian groups H for which all Haar graphs \(\mathrm{H}(H,S)\) are Cayley graphs.

Estélyi and Pisanski [8] also solved Problem 1.1 for dihedral groups, and they showed that all Haar graphs of the dihedral group \(D_{2n}\) are Cayley graphs if and only if \(n=2,3,4,5\) (see [8, Theorem 8]). We denote by \(D_{2n}\) the dihedral group of order 2n,  and by \(Q_8\) the quaternion group. It is worth mentioning that all non-Cayley Haar graphs given in [8] were not vertex-transitive, and the first examples of vertex-transitive non-Cayley Haar graphs were constructed later by Conder et al. in [5].

Our first goal in this paper is to solve Problem 1.1 for the class of inner abelian groups. Recall that a group H is called inner abelian if H is non-abelian, and all proper subgroups of H are abelian. We prove the following theorem.

Theorem 1.2

Let H be a finite inner abelian group for which all Haar graphs \(\mathrm{H}(H,S)\) are Cayley graphs. Then H is isomorphic to \(D_6, \, D_8, \, D_{10}\) or \(Q_8\).

We would like to note that Problem 1.1 is closely related to graphical regular representations of finite groups. By a graphical regular representation (GRR for short) for a group H we mean a Cayley graph \(\Gamma \) of H such that \(H \cong \mathrm{Aut}(\Gamma )\). Classifying finite groups admitting a GRR was an active research topic, and the solution was derived in several papers (see, for instance, [3, 10, 11, 14,15,16, 22, 23, 27, 28]). Motivated by a GRR, a Haar graph \(\Gamma \) of the group H such that \(H \cong \mathrm{Aut}(\Gamma )\) can be regarded as a graphical biregular representation for H. Clearly, if H admits such a representation, then it is not a solution to Problem 1.1. As the last step in determining the class of finite groups admitting a GRR, Godsil [9] showed that every non-solvable group belongs to this class. As an application of Theorem 1.2, we prove here the following similar result.

Theorem 1.3

Every finite non-solvable group admits a non-Cayley Haar graph.

The rest of the paper is organized as follows: In the next section we collect all concepts and results that will be used later. In Sect. 3, we give some preparatory lemmas and then prove Theorems 1.2 and 1.3 in Sect. 4.

2 Preliminaries

All groups in this paper are finite, and all graphs are finite and undirected. For a graph \(\Gamma \) we denote by \(V(\Gamma ),\)\(E(\Gamma )\) and \(\mathrm{Aut}(\Gamma )\) the vertex set, the edge set and the group of all automorphisms of \(\Gamma \). Given a vertex \(v \in V(\Gamma ),\) we denote by \(\Gamma (v)\) the set of vertices adjacent to v,  and if \(G \le \mathrm{Aut}(\Gamma ),\) then by \(G_v\) the stabilizer of v in G.

Let \(\Gamma =\mathrm{H}(H,S)\) be a Haar graph of a group H with identity element 1. By [30, Lemma 3.1(2)], up to graph isomorphism, we may always assume that \(1\in S\). The graph \(\Gamma \) is then connected exactly when \(H=\langle S \rangle \). For \(g \in H\), the right translationR(g) is the permutation of H defined by \(R(g) : h \mapsto hg\) for \(h \in H,\) and the left translationL(g) is the permutation of H defined by \(L(g) : h \mapsto g^{-1}h\) for \(h \in H\). Set \(R(H):=\{ R(h) \, : \, h \in H\}\). It is easy to see that R(H) can also be regarded as a group of automorphisms of \(\mathrm{H}(H,S)\) acting on its vertices by the rule

$$\begin{aligned} \forall i\in \{0,1\},\quad \forall h,g\in H:~ h_i^{R(g)}=(hg)_i. \end{aligned}$$

For an automorphism \(\alpha \in \mathrm{Aut}(H)\) and \(x, y, g \in H\), define two permutations on \(V(\Gamma )=H_0 \cup H_1\) as follows

$$\begin{aligned}&\forall h\in H : h_0^{\delta _{\alpha ,x,y}}=(xh^\alpha )_1,\quad h_1^{\delta _{\alpha ,x,y}}=(yh^\alpha )_0, \end{aligned}$$
(1)
$$\begin{aligned}&\forall h\in H : h_0^{\sigma _{\alpha ,g}}=(h^\alpha )_0,\quad h_1^{\sigma _{\alpha ,g}}=(gh^\alpha )_1. \end{aligned}$$
(2)

Set

$$\begin{aligned} \mathrm{I}= & {} \{\delta _{\alpha ,x,y} : \alpha \in \mathrm{Aut}(H),~S^\alpha =y^{-1}S^{-1}x\},\\ \mathrm{F}= & {} \{\sigma _{\alpha ,g} : \alpha \in \mathrm{Aut}(H),~S^\alpha =g^{-1}S \}. \end{aligned}$$

By [30, Lemma 3.3], \(\mathrm{F}\le \mathrm{Aut}(\Gamma )_{1_0}\). If \(\Gamma \) is connected, then \(\mathrm{F}\) acts on the set \(\Gamma (1_0)\) consisting of all neighbors of \(1_0\) faithfully. By [30, Theorem 1.1 and Lemma 3.2], we have the following proposition.

Proposition 2.1

([30]) Let \(\Gamma =\mathrm{H}(H,S)\) be a connected Haar graph, and let \(A=\mathrm{Aut}(\Gamma )\).

  1. (i)

    If \(\mathrm{I}=\emptyset ,\) then the normalizer \(N_{A}(R(H))=R(H)\rtimes \mathrm{F}\).

  2. (ii)

    If \(\mathrm{I}\ne \emptyset \), then \(N_{A}(R(H))=R(H)\langle \mathrm{F},\delta _{\alpha ,x,y} \rangle \) for \(\delta _{\alpha ,x,y}\in \mathrm{I}\).

Moreover, \(\langle R(H),\delta _{\alpha ,x,y} \rangle \) acts transitively on \(V(\Gamma )\) for any \(\delta _{\alpha ,x,y}\in \mathrm{I}\).

Let H be a permutation group on a finite set \(\Omega \). For convenience, let \(\Omega =\{1,\ldots ,n\}\). Let G be a permutation group on a finite set \(\Delta ,\) and let \(N=G \times \cdots \times G\) with n factors. We define the action of H on N by letting

$$\begin{aligned} \forall g_i \in G,\quad \forall h\in H:~(g_1,\ldots ,g_n)^h=(g_{1^{h^{-1}}},\ldots ,g_{n^{h^{-1}}}). \end{aligned}$$

The semidirect product of N by the group H with respect to the above action is called the wreath product of G and H,  denoted by \(G \wr H\). The group \(G \wr H\) can be viewed as a permutation group of the set \(\Omega \times \Delta ,\) by letting the element \((g_1,\ldots ,g_n;h) \in G \wr H\) act as

$$\begin{aligned} \forall ~(i,\delta ) \in \Omega \times \Delta :~(i,\delta )^{(g_1,\ldots ,g_n;h)}=(i^h,\delta ^{g_i}). \end{aligned}$$

Notice that, if H is intransitive on \(\Omega ,\) then the group \(G \wr H\) is obviously intransitive on \(\Omega \times \Delta \). This observation will be used in the next section.

Given two graphs \(\Gamma _1\) and \(\Gamma _2,\) the lexicographical product\(\Gamma _1[\Gamma _2]\) is defined to be the graph with vertex set \(V(\Gamma _1) \times V(\Gamma _2),\) and two vertices \((u_1,u_2)\) and \((v_1,v_2)\) are adjacent in \(\Gamma _1[\Gamma _2]\) if and only if \(u_1=v_1\) and \(u_2\) is adjacent to \(v_2\) in \(\Gamma _2,\) or \(u_1\) is adjacent to \(v_1\) in \(\Gamma _1\). In view of [25, Theorem], we have the following proposition.

Proposition 2.2

([25]) Let \(\Gamma _1\) and \(\Gamma _2\) be two graphs. Then \(\mathrm{Aut}(\Gamma _1[\Gamma _2])=\mathrm{Aut}(\Gamma _2) \wr \mathrm{Aut}(\Gamma _1)\) if and only if the following conditions hold:

  1. (i)

    If there exist two distinct vertices \(u,v \in V(\Gamma _1)\) such that \(\Gamma _1(u)=\Gamma _1(v),\) then \(\Gamma _2\) is connected.

  2. (ii)

    If there exist two distinct vertices \(u,v \in V(\Gamma _1)\) such that \(\Gamma _1(u) \cup \{u\}=\Gamma _1(v) \cup \{v\},\) then the complement \((\Gamma _2)^c\) of the graph \(\Gamma _2\) is connected.

Let \(\mathbb {Z}H\) denote the group ring of H over the ring of integers. We denote by \(\cdot \) the usual multiplication and by \(\circ \) the Schur–Hadamard multiplication of \(\mathbb {Z}H,\) that is,

$$\begin{aligned} \sum _{h\in H}c_h h \cdot \sum _{h\in H}d_h h= & {} \sum _{h\in H}\left( \sum _{g \in H}c_gd_{g^{-1}h}\right) h, \\ \sum _{h\in H}c_h h \circ \sum _{h\in H}d_h h= & {} \sum _{h\in H}(c_hd_h) h. \end{aligned}$$

Given a subset \(S \subseteq H,\) let \(\underline{S}\) denote the \(\mathbb {Z}H\)-element \(\sum _{h\in S}h\). After Wielandt [29], we call such elements simple quantities (see [29, p. 54]).

Let G be a permutation group of H such that \(R(H) \le G,\), and let \(G_1\) denote the stabilizer of the identity element 1 in G. Schur [26] proved that the \(\mathbb {Z}\)-module spanned by all simple quantities \(\underline{X}\) where X runs over the set of all \(G_1\)-orbits is a subring of \(\mathbb {Z}H\) (also see [29, Theorem 24.1]). This \(\mathbb {Z}\)-module is called the transitivity module of \(G_1,\) denoted by \(\mathfrak {R}(H,G_1)\). The simple quantity \(\underline{X}\) for a \(G_1\)-orbit X will also be called a basic quantity of \(\mathfrak {R}(H,G_1)\). A couple of properties of transitivity modules are listed in the proposition below. In fact, the statements in (i)–(iii) are from [29, Propositions 22.1, 22.4 and 23.6].

Proposition 2.3

([29]) Let \(\mathfrak {S}=\mathfrak {R}(H,G_1)\) be a transitivity module of a group \(G_1\). The following properties hold:

  1. (i)

    If \(\sum _{h \in H}c_h h \in \mathfrak {S}\) and \(c \in \mathbb {Z},\) then the simple quantity \(\underline{\{h \in H : c_h=c \}} \in \mathfrak {S}\).

  2. (ii)

    \(\mathfrak {S}\) is closed under the Schur–Hadamard product of \(\mathbb {Z}H\).

  3. (iii)

    If \(\underline{S} \in \mathfrak {S}\) for a subset \(S \subseteq H,\) then \(\underline{\langle S \rangle } \in \mathfrak {S}\).

The next properties are widely used and also easy to show; nevertheless, for easier reading we present a proof.

Proposition 2.4

Let \(\mathfrak {S}=\mathfrak {R}(H,G_1)\) be a transitivity module, and \(S \subseteq H\) be a subset such that \(\underline{S} \in \mathfrak {S}\). The following properties hold:

  1. (i)

    \(\underline{S^{-1}} \in \mathfrak {S}\).

  2. (ii)

    If \(\underline{\{h\}} \in \mathfrak {S}\) for some \(h\in H\) and \(\underline{S}\) is a basic quantity of \(\mathfrak {S},\) then \(\underline{h S}\) and \(\underline{S h}\) are also basic quantities of \(\mathfrak {S}\).

  3. (iii)

    \(\langle S \rangle \) is block of imprimitivity for G.

  4. (iv)

    If \(\underline{\{h\}} \in \mathfrak {S},\) then \(L(h) \in C_{S_H}(G)\).

Proof

(i): It is well known that there is a one-to-one correspondence between the \(G_1\)-orbits on H and the G-orbits on \(H \times H\) given as follows: If \(\Delta \) is the G-orbit of \((h,k) \in H \times H,\) then the corresponding \(G_1\)-orbit is equal to the set \(\{x \in H : (1,x) \in \Delta \}\). Denote the latter set by T,  so in other words, \(\underline{T}\) is a basic quantity of \(\mathfrak {S}\). Let \(T'\) be the \(G_1\)-orbit corresponding to the G-orbit of (kh) on \(H \times H\). Then \(T'\) can be expressed as \(T'=\{x \in H : (x,1) \in \Delta \}\). On the other hand, using that \(R(H) \le G,\) we can write

$$\begin{aligned} (x,1) \in \Delta \iff (1,x^{-1})=(x,1)^{R(x^{-1})} \in \Delta \iff x \in T^{-1}. \end{aligned}$$

This shows that \(T'=T^{-1}\). Since \(\underline{S}=\sum \underline{T_i}\) for some basic quantities \(\underline{T_i},\) by the previous observation we get \(\underline{S^{-1}}=\sum \underline{T_i^{-1}} \in \mathfrak {S},\) and (i) follows.

(ii): In the group ring \(\mathbb {Z}H\) it holds \(\underline{h S}=\underline{\{h\}} \cdot \underline{S},\) and as \(\mathfrak {S}\) is a subring of \(\mathbb {Z}H,\)\(\underline{h S} \in \mathfrak {S}\). Choose a basic quantity \(\underline{T}\) of \(\mathfrak {S}\) with \(T\subseteq hS\). By (i), \(\underline{h^{-1}} \in \mathfrak {S}\). If \(T\ne h S,\) then \(h^{-1}T \subsetneq S,\) contradicting that \(\underline{S}\) was chosen to be a basic quantity, and (ii) follows for hS. The proof of the other statement with Sh goes in the same way.

(iii): Let \(K=\langle S \rangle \). By Proposition 2.3 (iii), \(\underline{K}\in \mathfrak {S},\) or in other words, \(G_1\) fixes K setwise. Let \(G_{\{K\}}\) denote the setwise stabilizer of K in G. Then \(G_{\{K\}}=G_1R(K)\). In particular, \(G_1 \le G_{\{K\}},\) and the \(G_{\{K\}}\)-orbit of 1 is a block of imprimitivity for G (see [7, Theorem 1.5A]). As the latter orbit is K,  (iii) follows.

(iv): Let \(x \in H\) and \(g \in G\). Observe that we can choose elements \(y \in H\) and \(g_1\in G_1\) such that \(R(x) g=g_1 R(y)\). By (i), \(\underline{\{h^{-1}\}} \in \mathfrak {S},\) and hence \((h^{-1})^{g_1}=h^{-1}\). Thus, \((h^{-1}x)^g=(h^{-1})^{g_1 R(y)}=h^{-1}y\) and \(x^g=1^{g_1 R(y)}=y\). These give \((h^{-1}x)^g=h^{-1}x^g,\) that is, L(h) commutes with g. \(\square \)

We would like to remark that the transitivity module \(\mathfrak {R}(H,G_1)\) is an example of the so-called S-rings over the group H. For more information on these rings, we refer the reader to [29, Chapter IV], and for a survey on various applications of S-rings in algebraic graph theory, we refer to [20].

3 Properties of Cayley Haar graphs

In this section, we give two lemmas about groups all of whose Haar graphs are Cayley graphs. We introduce the following notation which will be used throughout the paper.

$$\begin{aligned} \mathcal{BC}=\big \{ H \; \text {is a finite group} : \forall S \subseteq H,~~\mathrm{H}(H,S) \; \text {is a Cayley graph} \big \}. \end{aligned}$$

Lemma 3.1

The class \(\mathcal{BC}\) is closed under taking subgroups.

Proof

Let H be a group in the class \({\mathcal {BC}}\), and let \(K\le H\). Let \(\mathrm{H}(K,S)\) be a Haar graph of K for some subset \(1\in S\subseteq K\). It is sufficient to prove that \(\mathrm{H}(K,S)\) is a Cayley graph. Note that \(\mathrm{H}(K,S)\) is a union of some components of \(\mathrm{H}(H,S)\) each being isomorphic to \(\mathrm{H}(\langle S \rangle ,S)\). Denote by \(\Gamma \) a component of \(\mathrm{H}(K,S)\). Then \(\Gamma \cong \mathrm{H}(\langle S \rangle ,S)\) and \(V(\Gamma )\) is a block of imprimitivity for \(\mathrm{Aut}(\mathrm{H}(H,S))\).

First, we prove that \(\Gamma \) is a Cayley graph. Since \(H\in \mathcal {BC}\), the Haar graph \(\mathrm{H}(H,S)\) is a Cayley graph, and thus \(\mathrm{Aut}(\mathrm{H}(H,S))\) has a subgroup R acting regularly on \(V(\mathrm{H}(H,S))\). Since \(V(\Gamma )\) is a block of imprimitivity, the setwise stabilizer \(R_{\{V(\Gamma )\}}\) is transitive on \(V(\Gamma )\). This implies that \(R_{\{V(\Gamma )\}}\) is regular on \(V(\Gamma ),\) and thus \(\Gamma \) is a Cayley graph.

Now, we prove that \(\mathrm{H}(K,S)\) is a Cayley graph. Noting that each component of \(\mathrm{H}(K,S)\) is isomorphic to \(\Gamma \), assume that \(\mathrm{H}(K,S)\) has m components, and one may identify \(\{(i,u)~|~1\le i\le m, u\in V(\Gamma )\}\) and \(\{\{(i,u),(i,v)\}~|~1\le i\le m, \{u,v\}\in E(\Gamma )\}\) with the vertex set and the edge set of \(\mathrm{H}(K,S)\), respectively. Since \(\Gamma \) is a Cayley graph, \(\mathrm{Aut}(\Gamma )\) has a regular subgroup on \(V(\Gamma )\), say G. Write \(\sigma =(1\ 2\ \ldots \ m)\), the cyclic permutation on \(\{1,2,\ldots ,m\}\). Then it is easy to see that the group \(\langle \sigma \rangle \times G\cong \mathbb {Z}_m\times G\) acts regularly on \(V(\mathrm{H}(K,S))\) with the action given by

$$\begin{aligned} (i,u)^{(\alpha ,\beta )}\mapsto (i^{\alpha },u^{\beta }),\quad \forall (i,u)\in V(\mathrm{H}(K,S)), \end{aligned}$$

where \((\alpha ,\beta )\in \langle \sigma \rangle \times G\). Hence, \(\mathrm{H}(K,S)\) is a Cayley graph, as required. \(\square \)

Lemma 3.2

Let H be a group, \(N \trianglelefteq H\) be a normal subgroup, and \(\bar{S} \subseteq H/N\) be a subset such that

  1. (i)

    \(\mathrm{H}(H/N,\bar{S})\) is not vertex-transitive, and

  2. (ii)

    \(\bar{S} \ne \bar{S} x\) and \(\bar{S} \ne x \bar{S}\) for all non-identity elements \(x \in H/N\).

Then H does not belong to \(\mathcal{BC}\).

Proof

Let \(S=\bigcup _{aN\in \bar{S}} aN\), a subset of H. Let \(\Gamma =\mathrm{H}(H,S)\) and \(\bar{\Gamma }=\mathrm{H}(H/N,\bar{S})\). It is straightforward to show that \(\Gamma \cong \bar{\Gamma }[nK_1]\) where \(n=|N|\). Since the complement \((nK_1)^c\) of \(nK_1\) is \(K_n\) that is connected, by Proposition 2.2, the equality \(\mathrm{Aut}(\Gamma ) = \mathrm{Aut}(nK_1) \wr \mathrm{Aut}(\bar{\Gamma })\) holds if

$$\begin{aligned} \forall u,v \in V(\bar{\Gamma }):~ \bar{\Gamma }(u)=\bar{\Gamma }(v) \Rightarrow u = v. \end{aligned}$$
(3)

Suppose that \(\bar{\Gamma }(u)=\bar{\Gamma }(v)\) for the vertices \(u=x_i\) and \(v=y_j,\) where \(x, y \in H/N\) and \(i,j \in \{0,1\}\). It is easy to see that this implies that \(i=j=0\) and \(\bar{S} x=\bar{S} y,\) or \(i=j=1\) and \(\bar{S}^{-1} x = \bar{S}^{-1} y\). Thus, \(\bar{S}=\bar{S} x y^{-1}\) or \(\bar{S}=x y^{-1} \bar{S}\). By (ii), \(x=y\) and so \(u=v\), which means that Eq. (3) holds.

By Proposition 2.2, \(\mathrm{Aut}(\Gamma ) = \mathrm{Aut}(nK_1) \wr \mathrm{Aut}(\bar{\Gamma })\). Since \(\mathrm{Aut}(\bar{\Gamma })\) is not transitive on \(V(\bar{\Gamma })\), we have that \(\mathrm{Aut}(\Gamma )\) is not transitive on \(V(\Gamma )\). In particular, \(\Gamma \) is not a Cayley graph, and hence, H is not in \(\mathcal{BC}\). \(\square \)

We finish the section with a corollary of Lemmas 3.1 and 3.2 which will be useful when we deal with Haar graphs of non-solvable groups.

Corollary 3.3

Let H be a group with normal series

$$\begin{aligned} 1=H_0 \unlhd H_1 \unlhd \cdots \unlhd H_{n-1} \unlhd H_n=H. \end{aligned}$$

If for some \(i \in \{0,\ldots ,n-1\},\) there exists a subset \(R \subset H_{i+1}/H_i\) such that

  1. (i)

    \(\mathrm{H}(H_{i+1}/H_i,R)\) is not vertex-transitive, and

  2. (ii)

    \(R \ne R x\) and \(R \ne x R\) for all non-identity elements \(x \in H_{i+1}/H_i\),

then H does not belong to \(\mathcal{BC}\).

4 Proof of Theorems 1.2 and 1.3

Throughout this section H denotes an inner abelian group. In our first lemma we consider the case when H is a p group. Rédei [24] proved that H is then isomorphic to one of the following groups:

  • The quaternion group \(Q_8\),

  • \(M_p(m,n)=\langle a,b,c \, \mid \, a^{p^m}=b^{p^n}=c^p=1,[a,b]=c=a^{p^{m-1}} \rangle \) with \(m \ge 2\) and \(n \ge 1\),

  • \(M_p(m,n,1)=\langle a,b,c \, \mid \, a^{p^m}=b^{p^n}=c^p=1, [a,b]=c, [c,a]=[c,b]=1 \rangle \) with \(m \ge n\) and if \(p=2\) then \(m+n \ge 3\).

For the groups \(H=M_p(m,n)\) or \(M_p(m,n,1),\) it is easy to find that \([a,b]=c \in Z(H),\) the center of H. Hence, \([a^i,b^j]=[a,b]^{ij}=c^{ij}\) (see [10, Lemma 2.2.2]), and \(a^ib^j=b^ja^ic^{ij}\), where \(i \in \mathbb {Z}_{p^m}\) and \(j \in \mathbb {Z}_{p^n}\).

Lemma 4.1

Suppose that \(H=M_p(m,n)\) or \(M_p(m,n,1)\). Let \(p\ge 3\) or \(p=2\) with \(m \ge 3\) and \(n=1\). Then, the Haar graph \(\Gamma =\mathrm{H}(H,S)\) is not vertex-transitive for \(S=\{1,a,a^{-1},b,ab\}\).

Proof

If \((p,m)=(3,1)\), then \(H\cong M_3(1,1,1)\), and we check by Magma [4] that the Haar graph \(\mathrm{H}(H,S)\) is not vertex-transitive. Hence, we assume that \((p,m)\ne (3,1)\).

Let \(A=\mathrm{Aut}(\Gamma )\), and let \(\bar{A}\) be the maximal subgroup of A that fixes the two bipartite sets of \(\Gamma \) setwise. We settle the lemma in two steps.

Claim 1The normalizer\(N_A(R(H)) \le \bar{A}\).

Suppose to the contrary that \(N_A(R(H)) \nleq \bar{A}\). By Proposition 2.1, there exists a permutation \(\delta _{\alpha ,x,y} \in N_A(R(H))\) for some \(\alpha \in \mathrm{Aut}(H)\) and \(x,y \in H\) such that \(S^{\alpha }=y^{-1}S^{-1}x\). Since R(H) acts transitively on \(H_1,\) we may further assume that \(1_0^{\delta _{\alpha ,x,y}}=1_1\). By Eq. (1), \(1_0^{\delta _{\alpha ,x,y}}=(x1^{\alpha })_1=1_1,\) it follows that \(x=1,\) and thus \(S^{\alpha }=y^{-1}S^{-1}\). We can write

$$\begin{aligned} \{1^{\alpha },a^{\alpha },(a^{-1})^{\alpha },b^{\alpha },(ab)^{\alpha }\}= y^{-1}\{1,a^{-1},a,b^{-1}, b^{-1}a^{-1}\}, \end{aligned}$$
(4)

implying that \(y=1\), \(a^{-1}\), a, \(b^{-1}\) or \(b^{-1}a^{-1}\). Recall that \(a^ib^j=b^ja^ic^{ij}\) for \(i \in \mathbb {Z}_{p^m}\) and \(j \in \mathbb {Z}_{p}\).

Let \(y=1\). Then \(\{a^{\alpha },(a^{-1})^{\alpha },b^{\alpha },(ab)^{\alpha }\}= \{a^{-1},a,b^{-1}, b^{-1}a^{-1}\}\) by Eq. (4). If \(a^{\alpha }=b^{-1}\), then \((a^{-1})^{\alpha }=b\in \{a^{-1},a, b^{-1}a^{-1}\}\), which is impossible. Similarly, \(a^{\alpha }\ne b^{-1}a^{-1}\). Hence, \(\{a^{\alpha },(a^{-1})^{\alpha }\}= \{a^{-1},a\}\) and \(\{b^{\alpha },(ab)^{\alpha }\}= \{b^{-1},b^{-1}a^{-1}\}\). It follows that \(a^{\alpha }=(ab)^{\alpha }\cdot (b^{-1})^{\alpha }=b^{-1}a^{-1}\cdot b=a^{-1}c^{-1}\) or \(b^{-1}\cdot ab=ac\), and since \(a^{\alpha }\in \{a^{-1},a\}\) and the order of c in H is p, we have \(c=a^{-2}\), forcing that \((p,m)=(2,2)\), contrary to the assumption that \(m \ge 3\) if \(p=2\).

Let \(y=a^{-1}\) or a. By Eq. (4), we have \(\{a^{\alpha },(a^{-1})^{\alpha },b^{\alpha },(ab)^{\alpha }\}= \{a,a^2,ab^{-1}, b^{-1}c^{-1}\}\) or \(\{a^{-1},a^{-2},a^{-1}b^{-1}, b^{-1}a^{-2}c\}\), respectively. By a similar argument as above, we have \(\{a^{\alpha },(a^{-1})^{\alpha }\}=\{a,a^2\}\) or \(\{a^{-1},a^{-2}\}\), forcing that \(p=3\) and \(m=1\), contrary to the assumption that \((p,m)\ne (3,1)\). Let \(y=b^{-1}\). Then \(\{a^{\alpha },(a^{-1})^{\alpha },b^{\alpha },(ab)^{\alpha }\}= \{b,ba^{-1},ba,a^{-1}\}\), while the latter set cannot contain both \(a^{\alpha }\) and \((a^{-1})^{\alpha }\), a contradiction. Let \(y=b^{-1}a^{-1}\). Then \(\{a^{\alpha },(a^{-1})^{\alpha },b^{\alpha },(ab)^{\alpha }\}= \{ab,bc,ba^2c,a\}\), and \(\{a^{\alpha },(a^{-1})^{\alpha }\}=\{bc,ba^2c\}\). Hence, \(bc\cdot ba^2c=b^2a^2c^2=1\) and \((p,m,n)=(2,1,1)\), a contradiction.

To show that \(\Gamma =\mathrm{H}(H,S)\) is not vertex-transitive it is sufficient to prove the following statement.

Claim 2\(A=\bar{A}\).

Note that R(H) is a p-subgroup of \(\bar{A}\). Let P be a Sylow p-subgroup of \(\bar{A}\) containing R(H). Assume for the moment that \(P \ne R(H)\). Then \(R(H) < N_P(R(H))\) (see [10, Theorem 1.2.11(ii)]). By Claim 1 and Proposition 2.1, we have \(N_P(R(H)) \le R(H)\rtimes \mathrm{F}\), where \(\mathrm{F}=N_A(R(H))_{1_0}\). Since \(\langle S \rangle =H\), the Haar graph \(\Gamma \) is connected, and thus \(\mathrm{F}\) acts faithfully on the neighborhood \(\Gamma (1_0)=\{1_1,a_1,(a^{-1})_1,b_1,(ab)_1\}\). Since \(\Gamma \) has valence 5,  \(F \le S_5,\) and we have \(p=2, \, 3\) or 5. Since \(|\mathrm{F}|\ne 1,\) by Eq. (2), there exists a \(\sigma _{\alpha ,g}\in \mathrm{F}\) of order p for some \(\alpha \in \mathrm{Aut}(H)\) and \(g \in H\) such that \(S^{\alpha }=g^{-1}S\), that is,

$$\begin{aligned} \{1^{\alpha },a^{\alpha },(a^{-1})^{\alpha },b^{\alpha },(ab)^{\alpha }\}= g^{-1}\{1,a,a^{-1},b, ab\}, \end{aligned}$$
(5)

implying that \(g=1\), a, \(a^{-1}\), b or ab.

Let \(g=1\). By Eq. (5), we have \(\{a^{\alpha },(a^{-1})^{\alpha },b^{\alpha },(ab)^{\alpha }\}=\{a,a^{-1},b, ab\}\), and hence \(\{a^{\alpha },(a^{-1})^{\alpha }\}=\{a,a^{-1}\}\) and \(\{b^{\alpha },(ab)^{\alpha }\}=\{b, ab\}\). If \(b^{\alpha }=b\) and \((ab)^{\alpha }=ab\), then \(a^{\alpha }=(ab)^{\alpha }\cdot (b^{-1})^{\alpha }=ab\cdot b^{-1}=a\). It is easy to check that \(\sigma _{\alpha ,1}\) fixes each neighbor of \(1_0\), and since \(\langle \sigma _{\alpha ,1} \rangle \le \mathrm{F}\) acts on the neighbors of \(1_0\) faithfully, \(\sigma _{\alpha ,1}=1\), a contradiction. If \(b^{\alpha }=ab\) and \((ab)^{\alpha }=b\), then \(a^{\alpha }=a^{-1}\), and \(\sigma _{\alpha ,1}\) interchanges \(a_1\) and \((a^{-1})_1\), forcing that \(\sigma _{\alpha ,1}\) has order 2 and \(p=2\). Since \(b^{\alpha }=ab\) and \(b^{2^n}=b^2=1\), we have \((ab)^2=b^2a^2c^3=1\), which is impossible.

Let \(g=a\) or \(a^{-1}\). Then \(\{a^{\alpha },(a^{-1})^{\alpha },b^{\alpha },(ab)^{\alpha }\}=\{a^{-1},a^{-2},a^{-1}b, b\}\) or \(\{a,a^{2},ab, a^2b\}\), respectively. It follows that \(\{a^{\alpha },(a^{-1})^{\alpha }\}=\{a^{-1},a^{-2}\}\) or \(\{a,a^2\}\), and hence \(a^3=1\) and \((p,m)=(3,1)\), a contradiction. Similarly, if \(g=b\) or ab, then \(\{a^{\alpha },(a^{-1})^{\alpha },b^{\alpha },(ab)^{\alpha }\}=\{b^{-1},b^{-1}a,b^{-1}a^{-1}, ac\}\) or \(\{b^{-1}a^{-1},b^{-1},b^{-1}a^{-2}, a^{-1}c^{-1}\}\), and none of them can contain both \(a^{\alpha }\) and \((a^{-1})^{\alpha }\) in the same time, a contradiction. We conclude that \(R(H)=P\).

Since \(\langle S\rangle =H\), \(\Gamma \) is connected and so \(|A:\bar{A}|=2\), implying that \(\bar{A}\unlhd A\). The Frattini argument (see [10, Theorem 1.3.7]) together with Claim 1 yields \(A=\bar{A}N_A(R(H))=\bar{A},\) which completes the proof of the lemma. \(\square \)

Now, we turn to the case when H is not a p-group. Miller and Moreno [21] proved that

$$\begin{aligned} H \cong \mathbb {Z}_p^n \rtimes \mathbb {Z}_{q^m} \end{aligned}$$

for distinct primes p and q. For the next four lemmas we set P and Q to be a Sylow p- and q-subgroup of H,  respectively, and b for a generator of Q. Furthermore, let \(S_1 \subseteq P\) be a subset such that \(1 \in S_1\). By [10, Theorem 5.2.3], \(P=C_P(Q) \times [P,Q]\). Since H is inner abelian, it follows that \(H=[P,Q]Q,\) implying that \(C_P(Q)=1\). Also, as \(P\langle b^q \rangle < H,\) the group \(P\langle b^q \rangle \) is abelian. We conclude that b acts on P as a fixed point free automorphism of order q. Therefore, for any non-identity element \(a \in P,\) the Q-orbit of a can be written as \(a^Q=\{a,a^b,\ldots ,a^{b^{q-1}}\}\). Since \(C_P(Q)=1,\)\(\langle a^Q,Q \rangle \) is non-abelian, and thus \(\langle a^Q \rangle =P\). Observe that, since the product \(aa^b \cdots a^{b^{q-1}}\) is fixed by b,  it is equal to the identity 1; hence, the rank of P is at most \(q-1\). All these yield the following conditions:

$$\begin{aligned} H=\langle a,b \rangle ,\quad n < q \text { and } q \mid (p^n-1). \end{aligned}$$
(6)

Finally, we set \(\Gamma =\mathrm{H}(H,S_1 \cup \{b\})\) with \(1\in S_1\subseteq P,\)\(A=\mathrm{Aut}(\Gamma )\) and \(\bar{A} \le A\) for the maximal subgroup that fixes the two bipartite sets of \(\Gamma \) setwise.

Lemma 4.2

With the above notation, suppose that \(q > 2\). If either \(S_1=S_1^{-1}\), or \(|S_1|=4,\)\(|S_1 \cap S_1^{-1}|=3\) and \(\langle S_1 \rangle =p^2,\) then \(\bar{A}\) acts faithfully on \(H_0\).

Proof

Let \(S=S_1\cup \{b\}\). We consider the 4-cycles of \(\Gamma \) going through the vertex \(1_0\) and denote by \(\mathcal {C}\) the set of all such 4-cycles. We claim that no 4-cycle in \(\mathcal {C}\) goes through the vertex \(b_1\). Indeed, if such a 4-cycle existed, then it would be in the form

$$\begin{aligned} \big (1_0, b_1, (x^{-1}b)_0=(z^{-1}y)_0, y_1, 1_0\big ) \end{aligned}$$

for suitable \(x, y \in S_1\) and \(z \in S\). Thus, \(x^{-1}b=z^{-1}y,\) and since \(x,y \in S_1 \subseteq P,\) it follows that \(z \notin P,\) hence \(z=b\). This, however, implies \(x^{-1}b^2=y^{b} \in P,\) and so \(b^2=1,\) which contradicts that b has order \(q^m > 2\).

Suppose that \(S_1=S_1^{-1}\). If \(|S_1|=1\), then \(S_1=\{1\}\) and the above paragraph implies that \(\Gamma \) is a union of cycles of length at least 5, which implies that \(\bar{A}\) is faithful on \(H_0\). Now assume that \(|S_1|\ge 2\). For any \(a \in S_1\) with \(a \not =1\), we find that \((1_0,a_1,a_0,1_1) \in \mathcal {C}\). We conclude that \(b_1\) is the only neighbor of \(1_0\) which is not contained in a 4-cycle from \(\mathcal {C}\). Therefore, \(A_{1_0} \le A_{b_1},\) and hence, \(A_{h_0}=R(h)^{-1}A_{1_0}R(h) \le R(h)^{-1}A_{b_1}R(h)= A_{(bh)_1}\) for all \(h \in H\). This implies that \(\bar{A}\) acts faithfully on \(H_0\).

Now, suppose that \(|S_1|=4,\)\(|S_1\cap S_1^{-1}|=3\) and \(|\langle S_1 \rangle |=p^2\). Thus, \(p>2,\) and \(S_1=\{1,a,a^{-1},c\}\) for some \(a,c \in H\) such that \(|\langle a,c \rangle |=p^2\). It follows that \(\mathcal {C}\) contains exactly two 4-cycles \((1_0,a_1,a_0,1_1)\) and \((1_0,a^{-1}_1,a^{-1}_0,1_1)\) for \(p>3\) and exactly four 4-cycles \((1_0,a_1,a_0,1_1)\), \((1_0,a^{-1}_1,a^{-1}_0,1_1)\), \((1_0,a_1,a^{-1}_0,1_1)\) and \((1_0,a^{-1}_1,a_0,1_1)\) for \(p=3\). This implies in turn that \(A_{1_0} \le A_{1_1},\)\(A_{h_0} \le A_{h_1}\) for all \(h \in H,\) and \(\bar{A}\) acts faithfully on \(H_0\). \(\square \)

Lemma 4.3

With the above notation, let \(n=1, \, q > 2\) and \(S_1=\{1,a,a^{-1}\}\) for a non-identity element \(a \in P\). Then \(A=\bar{A}\).

Proof

Let \(S=S_1 \cup \{b\},\) and hence, \(\Gamma =\mathrm{H}(H,S)\). Notice that, since \(q>2\), it follows that \(p>5\). In this case, \(a^b=a^r\) for some integer r coprime to p such that r has multiplicative order q modulo p.

We are going to show below that \(R(P) \unlhd A\). It is sufficient to show that \(R(P) \; {\mathrm {char}} \; \bar{A}\). Let \(L = f^{-1} \bar{A}^{H_0} f,\) where \(\bar{A}^{H_0}\) denotes the permutation group of \(H_0\) induced by \(\bar{A}\) acting on \(H_0,\) and \(f : H_0 \mapsto H\) is the bijective mapping \(f : h_0 \mapsto h, \, h \in H\). By Lemma 4.2, \(L \cong \bar{A}\). We warn the reader that, in what follows, R(P) will also denote the permutation group of H consisting of the right translations \(x \mapsto xh, \, x \in H, \ h \in P\). It is sufficient to show that \(R(P) \; {\mathrm {char}} \; L\).

Let us consider the transitivity module \(\mathfrak {S}=\mathfrak {R}(H,L_1)\). It is not hard to show, considering the 4-cycles of \(\Gamma \) through \(1_0,\) that \(\bar{A}_{1_0} \le \bar{A}_{1_1}\). Since the neighbors of \(1_1\) are \(1_0, \, a_0,\, a^{-1}_0\) and \(b^{-1}_0,\) we obtain that \(\underline{\{a,a^{-1},b^{-1}\}} \in \mathfrak {S}\). By Propositions 2.3 (ii) and 2.4 (i), \(\underline{\{a,a^{-1}\}} = \underline{\{a,a^{-1},b^{-1}\}} \circ \underline{\{a,a^{-1},b\}} \in \mathfrak {S},\) and thus also \(\underline{\{b\}} \in \mathfrak {S}\). By Eq. (6), \(H=\langle a,b \rangle \). Thus, if \(\underline{\{a\}} \in \mathfrak {S},\) then it follows by this and Proposition 2.4 (ii) that \(\mathfrak {S}=\mathbb {Z}H\). Hence, \(L=R(H),\) and \(R(P) \; {\mathrm {char}} \; L,\) as required. Now, suppose that \(\underline{\{a,a^{-1}\}}\) is a basic quantity of \(\mathfrak {S}\). By Proposition 2.4 (iii), \(P=\langle a,a^{-1} \rangle \) is a block of imprimitivity for L. Let \(L_{\{P\}}\) denote the setwise stabilizer of the block P in L. Since \(L_1\) fixes P setwise, it follows that \(L_{\{P\}}=L_1R(P)\). Let \(\rho \in L_{\{P\}}\) such that \(x^\rho =x\) for all \(x \in P\). Since, \(\underline{\{b^i\}} \in \mathfrak {S},\) for every \(b^i \in Q,\) by Proposition 2.4 (iv), \(\rho L(b^i) = L(b^i) \rho ,\) and we find \((b^{-i}x)^\rho =x^{L(b^i) \rho }=x^{\rho L(b^i)}=b^{-i}x\) for any \(x\in P\). Thus, \(\rho \) is the identity mapping, so \(L_{\{P\}}\) acts faithfully on P,  and \(L_{\{P\}}\) can be regarded as a permutation group of P. By Burnside’s theorem on transitive permutation groups of degree p (see [7, Theorem 3.5B]), \(L_{\{P\}}\) is doubly transitive on P or it is solvable. On the other hand, \(\{a,a^{-1}\}\) is an orbit under \(L_1=(L_{\{P\}})_1\) where \(|P|=p > 5\). All these show that \(L_{\{P\}}\) is a solvable group. This implies that \(|(L_{\{P\}})_1|=|L_1|=2,\) and we obtain that R(H) is normal in L. Since R(P) is characteristic in R(H), it is a normal Sylow p-subgroup of L; in particular, it is characteristic in L. We have shown that \(R(P) \unlhd A\).

Suppose to the contrary that \(A \ne \bar{A}\). Since \(|\bar{A}_1|=|L_1| \le 2,\) it follows that R(Q) is a Sylow q-subgroup of \(\bar{A}\). By the Frattini argument, \(N_A(R(Q)) \bar{A}=A,\) and therefore, \(N_A(R(Q)) \setminus \bar{A} \ne \emptyset \). Choose \(\rho \in N_A(R(Q)) \setminus \bar{A}\). Since \(R(P) \unlhd A,\)\(\rho \) normalizes R(P) as well, hence also \(R(P)R(Q)=R(H)\). By Eq. (1), \(\rho =\delta _{\alpha ,x,y}\) for some \(\alpha \in \mathrm{Aut}(H)\) and \(x,y \in H\) that satisfy \(yS^\alpha x^{-1}=S^{-1}\). Let \(\iota _{x^{-1}}\) denote the inner automorphism of H induced by \(x^{-1}\). Then \(y S^\alpha x^{-1}=yx^{-1} S^{\alpha \iota _{x^{-1}}}\). Replacing \(yx^{-1}\) with y and \(\alpha \iota _{x^{-1}}\) with \(\alpha ,\) we obtain that \(y S^\alpha = S^{-1},\) that is,

$$\begin{aligned} \{1^\alpha ,a^\alpha ,(a^{-1})^\alpha ,b^\alpha \}= y^{-1}\{1,a,a^{-1},b^{-1}\}, \end{aligned}$$

implying that \(y=1, a, a^{-1}\) or \(b^{-1}\). Using this condition, that \(a^\alpha \) and \((a^{-1})^\alpha \) are in \(y^{-1}\{1,a,a^{-1},b^{-1}\},\) and that \(p>3,\) we obtain that \(y=1,\) and \(\alpha \) satisfies \(a^\alpha =a^{\pm 1}\) and \(b^\alpha =b^{-1}\). Recall that \(a^b=a^r\) for some integer r coprime to p such that r has multiplicative order q modulo p. We can write \(a^{\pm r}=(a^r)^\alpha =(b^{-1}ab)^\alpha =b a^{\pm 1} b^{-1}\). Then, \(b^{-1} a^{\pm r} b=a^{\pm 1},\) and it follows that \(r^2 \equiv 1 \pmod p\). This contradicts that r has multiplicative order \(q > 2\) modulo p. This completes the proof of the lemma. \(\square \)

Lemma 4.4

With the above notation, let \(n>1, \, p > 2\) and \(S_1=\{1,a,a^{-1},a^b\}\) for a non-identity element \(a \in P\). Then \(A=R(H)\).

Proof

Let \(S=S_1 \cup \{b\},\) and hence, \(\Gamma =\mathrm{H}(H,S)\). We set \(c=a^b\). Since \(P=\langle a^Q \rangle \) and \(n>1,\) it follows that \(c \notin \langle a \rangle \).

Let \(L = f^{-1} \bar{A}^{H_0} f,\) where \(\bar{A}^{H_0}\) denotes the permutation group of \(H_0\) induced by \(\bar{A}\) acting on \(H_0,\) and \(f : H_0 \mapsto H\) is the bijective mapping \(f : h_0 \mapsto h, \, h \in H\). By Lemma 4.2, \(L \cong \bar{A}\).

We prove below that \(L=R(H)\). Here again, R(H) denotes the permutation group of H consisting of the right translations \(x \mapsto xh, \, x,h \in H\). This is equivalent to showing that the transitivity module \(\mathfrak {R}(H,L_1)=\mathbb {Z}H\). For the sake of simplicity we set \(\mathfrak {S}=\mathfrak {R}(H,L_1)\). It has been shown in the proof of Lemma 4.2 that \(\bar{A}_{1_0} \le \bar{A}_{1_1}\). Since the neighbors of \(1_1\) are \(1_0, \, a_0,\, a^{-1}_0, \, c^{-1}_0\) and \(b^{-1}_0,\) we obtain that \(\underline{\{a,a^{-1},c^{-1},b^{-1}\}} \in \mathfrak {S}\).

By Propositions 2.3 (ii), \(\underline{\{a,a^{-1}\}}=\underline{\{a,a^{-1},c^{-1},b^{-1}\}} \circ \underline{\{a,a^{-1},c,b\}} \in \mathfrak {S}\), and hence \(\underline{\{b^{-1},c^{-1}\}}\in \mathfrak {S}\) and \(\underline{\{b,c\}}\in \mathfrak {S}\) by Proposition 2.4 (i). Since \(c \notin \langle a \rangle \), both b and c do not belong to \(b^{-1}\{a,a^{-1}\}c\) and \(c^{-1}\{a,a^{-1}\}b\), and since \(b^{-1}\{a,a^{-1}\}b=\{c,c^{-1}\}\) and \(c^{-1}\{a,a^{-1}\}c=\{a,a^{-1}\}\), we find \( (\underline{\{b^{-1},c^{-1}\}} \cdot \underline{\{a,a^{-1}\}} \cdot \underline{\{b,c\}}) \circ \underline{\{b,c\}} = \underline{\{c\}}\). Thus, \(\underline{\{c\}}, \underline{\{b\}} \in \mathfrak {S},\) and by Proposition 2.4 (ii), \(\underline{\{c^{b^i}\}} \in \mathfrak {S}\) for all \(i \in \{0,1,\ldots ,q-1\}\). As the latter elements generate H,  we conclude that \(\mathfrak {S}=\mathbb {Z}H,\) and so \(L=R(H),\) as required.

Now, since \(\bar{A} \cong L,\) we obtain that \(\bar{A}=R(H),\) the permutation group of \(V(\Gamma )\) induced by the right translation. It follows that either \(A=R(H),\) or \(|A : R(H)|=2\) and A acts regularly on \(V(\Gamma )\). We finish the proof by showing that the latter possibility leads to a contradiction.

Assume that \(|A : R(H)|=2\). Then, one can choose \(\rho \in A\) such that \(1_0^\rho =1_1\). Since \(\rho \) normalizes R(H),  we can write \(\rho =\delta _{\alpha ,x,y}\) for some \(x,y \in H\) and \(\alpha \in \mathrm{Aut}(H)\) that satisfy \(yS^\alpha x^{-1}=S^{-1},\) see Proposition 2.1. Since \(1_0^\rho =1_1,\) it follows that \(x=1\). By Eq. (1), \((1_0)^{\rho ^2}=y_0\) and \((1_1)^{\rho ^2}=(y^\alpha )_1\). Using these and that \(\rho ^2 \in R(H),\) we conclude that \(\rho ^2=R(y)=R(y^\alpha )\) and so \(y^\alpha =y\). Also, \((hy)_0=(h_0)^{\rho ^2}=(y h^{\alpha ^2})_0\). It follows that \(\alpha ^2=\iota _y,\) the inner automorphism of H induced by y.

Since \(S^\alpha =y^{-1}S^{-1},\) we can write

$$\begin{aligned} \{1^\alpha ,a^\alpha ,(a^{-1})^\alpha ,c^\alpha ,b^\alpha \}= y^{-1}\{1,a,a^{-1},c^{-1},b^{-1}\}, \end{aligned}$$

implying that \(y=1, a, a^{-1}, c^{-1}\) or \(b^{-1}\).

Let \(y=1\). Then, \(a^\alpha =a^{\pm 1}, \, c^\alpha =c^{-1}\) and \(b^\alpha =b^{-1}\). We can write \(c^{-1}=c^\alpha =(b^{-1}ab)^\alpha =b a^{\pm 1} b^{-1},\) from which \(a^{b^2}=b^{-1} c b=a^{\pm 1}\). This implies that \(b^4\) fixes a,  hence \(q=2\). Then, by Eq. (6), \(n < q=2,\) a contradiction.

Let \(y=a^{\pm 1}\). Since \(\alpha \) fixes y,  it follows that \(a^\alpha =a\). Thus, \(\{a,a^{-1}\} \subset \{a^{-1},a^{-2},\)\(a^{-1}c^{-1},a^{-1}b^{-1}\}\) or \(\{a,a^2,ac^{-1},ab^{-1}\}\). Since \(c \notin \langle a \rangle ,\) we find \(p=3,\)\(c^\alpha =a^{\pm 1}c^{-1}\) and \(b^\alpha =a^{\pm 1}b^{-1}\). We can write \(a^{\pm 1}c^{-1}=c^\alpha =(b^{-1}ab)^\alpha =b a b^{-1},\) from which \(c^b=a^{-1}c^{\pm 1}\). This gives that \(H=\langle a,a^b,\ldots ,a^{b^{q-1}} \rangle = \langle a,c \rangle \). Thus, \(n=2,\)\(p^n-1=8,\) and as \(q \mid (p^n-1),\) see Eq. (6), it follows that \(q=2\) and \(n=1,\) a contradiction.

If \(y=c^{-1},\) then it follows that \(\{a^\alpha ,(a^{-1})^\alpha \} \not \subset c S^{-1}\). Finally, let \(y=b^{-1}\). Then \(b S^{-1} \cap P=\{1\}\). On the other hand, \(a^\alpha \in P^\alpha =P,\) and so \(b S^{-1} \cap P=S^\alpha \cap P \ne \{1\},\) a contradiction. This completes the proof of the lemma. \(\square \)

Lemma 4.5

With the above notation, let \(n>4, \, p=2\) and

$$\begin{aligned} S_1={\left\{ \begin{array}{ll} \{1,a,a^b,a^{b^2}\} &{}\quad \text {if}\;\; aa^{b^2} \in a^Q \\ \{1,a,a^b,a^{b^2},a^ba^{b^3}\} &{}\quad \text {if}\;\; a a^{b^2} \notin a^Q \end{array}\right. } \end{aligned}$$

for a non-identity element \(a \in P\). Then \(A=R(H)\).

Proof

Let \(S=S_1 \cup \{b\},\) and hence, \(\Gamma =\mathrm{H}(H,S)\). We set \(c=a^b, \, d=a^{b^2}, \, e=a^b a^{b^3}\) and \(E=\langle a,c,d \rangle \). Since \(n > 2,\)\(E \cong \mathbb {Z}_2^3\). Note that, since \(\langle a^Q \rangle =P\) and \(|P| > 16,\) the following conditions hold:

$$\begin{aligned} \{a^{b^{-1}},d^b,d^{b^2}\} \cap E = \emptyset . \end{aligned}$$
(7)

Let \(L = f^{-1} \bar{A}^{H_0} f,\) where \(\bar{A}^{H_0}\) denotes the permutation group of \(H_0\) induced by \(\bar{A}\) acting on \(H_0,\) and \(f : H_0 \mapsto H\) is the bijective mapping \(f : h_0 \mapsto h, \, h \in H\). By Lemma 4.2, \(L \cong \bar{A}\).

We prove below that \(L=R(H)\). This is equivalent to showing that the transitivity module \(\mathfrak {R}(H,L_1)=\mathbb {Z}H\). For the sake of simplicity we set \(\mathfrak {S}=\mathfrak {R}(H,L_1)\). As before, let \(\mathcal {C}\) denote the set of all 4-cycles of \(\Gamma \) through \(1_0\). Define the subsets of H as

$$\begin{aligned} X=\{h \in H : h_0 \in V(C) \text { for some } C \in \mathcal {C}, h\ne 1 \}, \; \text {and} \; Y=\{h \in H : b_1 \sim h_0 \text { in } \Gamma , h\ne 1 \}. \end{aligned}$$

It is clear that \(\underline{X} \in \mathfrak {S}\). Also, as \(b_1\) is the only neighbor of \(1_0\) which is not contained in some \(C \in \mathcal {C},\) see the proof of Lemma 4.2, it follows that \(\underline{Y} \in \mathfrak {S}\) also holds.

Let \(aa^{b^2}\in a^Q\). Then \(S=\{1,a,b,c,d\}\), \(X=\{a,c,d,ac,ad,cd\}\) and so \(\underline{\{acd\}}=\underline{\langle X \rangle }-\underline{X}-\underline{\{1\}} \in \mathfrak {S}\). Moreover, \(Y=\{ab,cb,db,b\}\). Write \((\underline{Y})^2=\sum _{h\in H}c_h h\). Using that \(d^b \notin E,\) see Eq. (7), we find that \(c_{b^2}=3\) and \(c_h < 3\) for all \(h \in H \setminus \{b^2\}\). By Proposition 2.3 (i), \(\underline{\{b^2\}} \in \mathfrak {S}\). Since \(H=\langle acd,b^2 \rangle \), we have \(\mathfrak {S}=\mathbb {Z}H,\) as required. (Note that \(q > 2\) because of Eq. (6) and the assumption \(n > 4\).)

Let \(aa^{b^2}\notin a^Q\). Then \(S=\{1,a,b,c,d,e\}\), \(X=\{a,c,d,e,ac,ad,ae,cd,ce,de\}\) and \(Y=\{ab,cb,db,eb,b\}\). Note that \(\langle X \rangle =\langle a,c,d,e \rangle =\langle a,a^b,a^{b^2},a^{b^3} \rangle \). Since \(\langle a^Q \rangle =P\) and \(|P|>16\), we have \(a^{b^4}\notin \langle X \rangle \), and thus \(e^b=da^{b^4}\notin \langle X \rangle \). Using this we compute

$$\begin{aligned} (\underline{Y^{-1}} \cdot \underline{Y}) \circ \underline{\langle X \rangle }= 5 \; \underline{\{1\}} + 2 \; \underline{\{c,d,e,cd,ce,cde\}}. \end{aligned}$$
(8)

This together with Proposition 2.3 (i)–(ii) yields that \(\underline{\{c,d,e,cd,ce,cde\}}\in \mathfrak {S}\). It follows that \(\underline{\{c,d,e,cd,ce,cde\}}\circ \underline{X}=\underline{\{c,d,e,cd,ce\}}\in \mathfrak {S}\) and \(\underline{\{cde\}}= \underline{\{c,d,e,cd,ce,cde\}}-\underline{\{c,d,e,cd,ce\}}\in \mathfrak {S}\). Since \(e=(ad)^b=cd^b\) and \(d^b \notin E,\) it follows that \(e \notin E\). Using this and that \(Y=\{ab,cb,db,eb,b\},\) we compute

$$\begin{aligned} (\underline{Y^{-1}} \cdot \underline{\{c,d,e,cd,ce\}}) \circ \underline{Y^{-1}}= 3 \; \underline{\{b^{-1},b^{-1}c\}} + 2 \; \underline{\{b^{-1}d,b^{-1}e\}}. \end{aligned}$$

This shows that \(\underline{\{b^{-1},b^{-1}c\}}, \underline{\{b^{-1}d,b^{-1}e\}} \in \mathfrak {S}\). Since \((\underline{\{b^{-1},b^{-1}c\}})^2 \circ (\underline{\{b^{-1}d,b^{-1}e\}})^2=\{b^{-2}a\},\) we have \(\underline{\{b^{-2}a\}} \in \mathfrak {S}\). Since \(e\notin E\), we have \(cde\notin E\), and since \((cde)^{b^{-2}a}=ac\in E\), we have \((cde)^{b^{-2}a}\ne cde\) and \(\langle cde,b^{-2}a \rangle \) is non-abelian. Hence, \(\langle cde,b^{-2}a \rangle =H\), and we have \(\mathfrak {S}=\mathbb {Z}H\), as required.

Now, since \(\bar{A} \cong L,\) we obtain that \(\bar{A}=R(H),\) the permutation group of \(V(\Gamma )\) induced by the right translation. It follows that either \(A=R(H),\) or \(|A : R(H)|=2\) and A acts regularly on \(V(\Gamma )\). As in the previous proof, we finish the proof by showing that the latter possibility leads to a contradiction.

Assume that \(|A : R(H)|=2\). It follows in the same way as in the proof of the previous lemma that \(y S^\alpha =S^{-1}\) for some \(\alpha \in \mathrm{Aut}(H)\) and \(y \in H\) such that \(y^\alpha =y\) and \(\alpha ^2=\iota _y,\) the inner automorphism of H induced by y. We can write

$$\begin{aligned} \{1^\alpha ,a^\alpha , c^\alpha , d^\alpha ,b^\alpha \}= & {} y^{-1}\{1, a, c, d, b^{-1}\}, \ \text {or} \\ \{1^\alpha ,a^\alpha , c^\alpha , d^\alpha , e^\alpha , b^\alpha \}= & {} y^{-1}\{1, a, c, d, e, b^{-1}\}, \end{aligned}$$

depending on whether \(S=\{1,a,c,d,b\}\) or \(\{1,a,c,d,e,b\}\). These imply that \(y=1, a, c, d, e\) or \(b^{-1}\).

Since \(P^\alpha =P,\) it follows that \(|y^{-1} S^{-1} \cap P|=|S^\alpha \cap P| =|S \cap P| \ge 4\). This shows that \(y \ne b^{-1}\) and \(b^\alpha =y^{-1}b^{-1}=y b^{-1}\) for some \(y \in S_1\). Let \(\beta \) denote the automorphism of P induced by the action of b on P. Then, for any \(x \in P,\)\(x^{\alpha ^{-1} \beta \alpha }=(b^{-1} x^{\alpha ^{-1}} b)^\alpha = byxyb^{-1}=x^{\beta ^{-1}}\). We obtain that \(\beta ^\alpha =\beta ^{-1},\) and therefore, \(\alpha \) maps any Q-orbit on P to a Q-orbit.

Case 1\(S=\{1,a,b,c,d\}\).

In this case \(ad \in a^{Q}\) and \(\beta ,\) viewed as a permutation of P,  is written in the form

$$\begin{aligned} \beta =(a\; c \; d\; d^b \; \ldots \; ad \; \ldots ) \; \cdots . \end{aligned}$$

We claim that \(\alpha \) fixes some element in \(\{a,c,d\}\). This element is y if \(y \ne 1\) since \(y \in \{a,c,d\}\) and \(y^\alpha =y\). Suppose that \(y=1\). Then \(\alpha \) is an involution and \(\{a,c,d\}^\alpha =\{a,c,d\}\). This implies that one of ac and d is fixed by \(\alpha ,\) and the claim follows. In particular, we have \((a^Q)^\alpha =a^Q\).

Let \(c^\alpha =c\). Then \(\beta ^{-1}=\beta ^\alpha =(a^\alpha \; c\; d^\alpha \; \ldots ),\) implying that \(a^\alpha =d\) and \(d^\alpha =a\). Thus, \((ad)^\alpha =ad,\) and we obtain that \(\alpha \) fixes two points of the cycle \((a\; c \; d\; d^b \; \ldots \; ad \; \ldots )\). This contradicts that \(\beta ^\alpha =\beta ^{-1},\) and the latter cycle has length \(q >2\).

Let \(a^\alpha =a\). Observe that \(c^\alpha \in S_1 S_1 \subset E\). On the other hand, \(\beta ^{-1}=\beta ^\alpha =(a\; c^\alpha \; d^\alpha \; \ldots ),\) implying that \(c^\alpha =a^{b^{-1}}\). This together with the previous observation yields \(a^{b^{-1}} \in E,\) a contradiction to Eq. (7).

Finally, if \(d^\alpha =d,\) then a similar argument yields \(d^b=d^{\beta } =d^{(\beta ^{\alpha })^{-1}}=c^\alpha \in E\), which contradicts again Eq. (7).

Case 2\(S=\{1,a,b,c,d,e\}\).

In this case \(ad\notin a^{Q}\) and \(\beta \) is written in the form

$$\begin{aligned} \beta =(a\; c \; d\; d^b \; \ldots )(ad \; e \; e^b \; \ldots ) \; \cdots . \end{aligned}$$

It can be shown, using the same argument as in Case 1 and that \(\alpha \) permutes the Q-orbits, that \(\alpha \) fixes some element in the set \(\{a,c,d,e\}\). If this element is a or d,  then we can copy the argument used above in Case 1.

Let \(c^\alpha =c\). This implies as above that \(a^\alpha =d\) and \(d^\alpha =a\). It follows in turn that \((ad)^\alpha =ad,\)\((ad)^Q\) is mapped by \(\alpha \) to itself, and \(e^\alpha \ne e\) because \(\alpha \) cannot fix two points of the cycle \((ad\; e \; e^b \; \ldots )\). It also follows that \(y \ne e,\) and thus \(y=1\) or c. If \(y=1,\) then \(\{a,c,d\}^\alpha =\{a,c,d\},\) and so \(e^\alpha =e,\) a contradiction. It follows that \(y=c\). Then, \(\{a,c,d\}=\{a,c,d\}^\alpha \subset \{c,ca,cd,ce\}\). Since \(e \notin E,\)\(\{a,d\}=\{ca,cd\},\) which contradicts that \(E=\langle a,c,d \rangle \cong \mathbb {Z}_2^3\).

Finally, let \(e^\alpha =e\) and assume that none of ac and d is fixed by \(\alpha \). It follows that \(y=1\) or e,  and thus \((ad)^\alpha \in E\). On the other hand, since \(\beta ^\alpha = \beta ^{-1}\) and \(e^\alpha =e,\) it follows that \((ad)^\alpha =e^{(\beta ^{\alpha })^{-1}}=e^{\beta }=e^b=(ad)^{b^2}=d d^{b^2},\) and hence \(d^{b^2} \in E,\) a contradiction to Eq. (7). This completes the proof of the lemma. \(\square \)

Everything is prepared to prove Theorem 1.2.

Proof of Theorem 1.2

We show first that each of \(D_6, D_8, D_{10}\) and \(Q_8\) belongs to \(\mathcal{BC}\). For the first three groups this follows from [8, Theorem 8]. Let \(\Gamma =\mathrm{H}(Q_8,S)\) be a Haar graph with \(1 \in S\). If \(\Gamma \) is disconnected, then \(\langle S \rangle < Q_8\) (see [8, Lemma 1(i)]), and thus \(\langle S \rangle \) is abelian. This implies that \(\mathrm{H}(\langle S \rangle ,S)\) is a Cayley graph. Since \(\Gamma \) is a union of some components each isomorphic to \(\mathrm{H}(\langle S \rangle ,S)\), the Haar graph \(\Gamma \) is also Cayley. Hence, we assume that \(\Gamma \) is connected. By Magma [4], all connected Haar graphs of \(Q_8\) are Cayley graphs (note that the Haar graphs of \(Q_8\) of valency 7 or 8 are isomorphic to \(K_{8,8}-8K_2\) or \(K_{8,8}\), respectively), and hence \(Q_8\) is also in the class \(\mathcal{BC}\).

Let H be an inner abelian group such that \(H \not \cong D_6, D_8, D_{10}\) and \(Q_8\). We finish the proof by showing that H does not belong to \(\mathcal{BC}\).

Case 1H is a p-group.

Then \(H=M_p(m,n)\) or \(M_p(m,n,1)\). Now, if \(p \ge 3\), then \(H \notin \mathcal{BC}\) by Lemma 4.1. Assume that \(p=2\). Then \(m \ge 2\).

Let \(m \ge 3\). We consider the subgroup \(N=\langle b^2 \rangle \). Then \(N \unlhd H\) and \(H/N=\langle aN,bN \rangle \cong M_2(m,1)\) or \(M_2(m,1,1)\). Let \(\bar{\Gamma }=\mathrm{H}(H/N,\bar{S})\) with \(\bar{S}=\{N,aN,a^{-1}N,bN,abN\})\). By Lemma 4.1, \(\bar{\Gamma }\) is not vertex-transitive. If \(\bar{S} \bar{x}=\bar{S}\) for some \(\bar{x} \in H/N\), then \(\bar{S}\) is a union of some left cosets of \(\langle \bar{x} \rangle \) in H / N, and since \(|\bar{S}|=5\), we have \(\bar{x}=N\) (the identity of H / N) or \(\bar{S}= \langle \bar{x} \rangle \) is a subgroup of H / N. Clearly, \(\bar{S}=\{N,aN,a^{-1}N,bN,abN\})\) is not a subgroup, and we get \(\bar{x}=N\). By a similar argument we can obtain that \(\bar{x}=N\) if \(\bar{x}\bar{S}=\bar{S}\), and then Lemma 3.2 implies that \(H \notin \mathcal{BC}\).

Let \(m=2\). Recall that \(H \ne M_2(2,1)\) as \(M_2(2,1) \cong D_8\). Thus, if \(n=1\), then \(H=M_2(2,1,1)\). In this case we obtain by the help of Magma [4] that the Haar graph \(\mathrm{H}(H,\{1,a,a^{-1},b,ab\})\) is not vertex-transitive. Hence, \(H \notin \mathcal{BC}\). Let \(n \ge 2\). Then \(H=M_2(2,n)\) or \(M_2(2,2,1)\). We consider the subgroup \(N=\langle b^{2^2} \rangle \). Then \(N \unlhd H\) and \(H/N=\langle aN, bN \rangle \cong M_2(2,2)\) or \(M_2(2,2,1)\). Let \(\bar{\Gamma }=\mathrm{H}(H/N,\bar{S})\) with

$$\begin{aligned} \bar{S}=\{N,aN,bN,abN,ab^2N,ab^3N\}). \end{aligned}$$

By Magma [4], \(\bar{\Gamma }\) is not vertex-transitive. If \(\bar{S}\bar{x}=\bar{S}\) for some \(\bar{x} \in H/N\), then \(\bar{S}\) is a union of some left cosets of \(\langle \bar{x} \rangle \) in H / N. In particular, \(\bar{x} \in \bar{S}\). Since \(|\bar{S}|=6\), the order of \(\bar{x}\) in H / N is 1, 2, 3 or 6. On the other hand, since each element in \(\bar{S} \setminus \{N\}\) has order 4,  we have \(\bar{x}=N\). Similarly, we have that \(\bar{x}\bar{S} \ne \bar{S}\) for all non-identity elements \(\bar{x} \in H/N,\) and thus \(H \notin \mathcal{BC}\) by Lemma 3.2. This completes the proof for p-groups.

Case 2H is not a p-group.

Then \(H=\mathbb {Z}_p^n \rtimes \mathbb {Z}_{q^m}\) for distinct primes p and q. In view of Lemmas 4.34.5, we may assume that \(n=1\) and \(q=2,\) or \(2 \le n \le 4\) and \(p=2\).

Let \(n=1\) and \(q=2\). Let \(N=\langle b^2 \rangle \). Then, \(N \unlhd H\) and \(H/N=\langle aN, bN \rangle \cong D_{2p}\). Consider the Haar graph \(\bar{\Gamma }=\mathrm{H}(H/N,\bar{S})\) with

$$\begin{aligned} \bar{S}=\{N, aN, a^3N, bN, abN, a^2b N, a^4b N\}. \end{aligned}$$

The graph \(\bar{\Gamma }\) is not vertex-transitive for \(p>5\). This can be verified by the help of Magma [4] for \(p=7,\) and it was proved in [8, Proposition 7] if \(p> 7\). If \(\bar{S}=\bar{S} \bar{x}\) for some \(\bar{x} \in N/H,\) then \(\bar{S}\) is a union of left cosets of \(\langle \bar{x} \rangle \). Since \(N \in \bar{S}\) and \(|\bar{S}|=7,\) it follows that \(\bar{x}=N\) or S is a subgroup of N / H of order 7. Clearly, the latter option is impossible, and we get \(\bar{x}=N\). It can be shown in the same way that \(\bar{S}=\bar{x} \bar{S}\) forces that \(\bar{x}=N,\) and thus \(H \notin \mathcal{BC}\) by Lemma 3.2 if \(p > 5\).

Now, suppose that \(p=3\) or 5. Since \(H \ncong D_6, D_{10},\) we obtain that \(m \ge 2\). Let \(N=\langle b^4 \rangle \). Then, \(N \unlhd H\) and \(H/N=\langle aN, bN \rangle \cong \mathbb {Z}_p \rtimes \mathbb {Z}_4\). Consider the Haar graph \(\bar{\Gamma }=\mathrm{H}(H/N,\bar{S})\) with

$$\begin{aligned} \bar{S}=\{N, aN, bN, abN, ab^2N, ab^3 N\}. \end{aligned}$$

We find by Magma [4] that \(\Gamma \) is not vertex-transitive. If \(\bar{S}=\bar{S} \bar{x}\) for some \(\bar{x} \in N/H,\) then \(\bar{S}\) is a union of some left cosets of \(\langle \bar{x} \rangle \); in particular, \(\bar{x}\) has order 1, 2, 3 or 6. Since each element in \(\bar{S} \setminus \{N,aN,ab^2N\}\) has order 4,  it follows that \(\bar{x}=N\). It can be shown in the same way that \(\bar{S}=\bar{x} \bar{S}\) forces that \(\bar{x}=N,\) and thus \(H \notin \mathcal{BC}\) by Lemma 3.2.

Finally, let \(2 \le n \le 4\) and \(p=2\). By Eq. (6), \(n < q\) and \(q \mid (2^n-1)\). It follows that \(q=3\) if \(n=2,\)\(q=7\) if \(n=3,\) and \(q=5\) if \(n=4\). The group \(\mathrm{GL}(n,2)\) has Sylow q-subgroup of order q. Let \(N=\langle b^q \rangle \). Then \(N \unlhd H,\) and for \(n=2, 3, 4,\) the quotient group H / N is isomorphic to \(\mathbb {Z}_2^2 \rtimes \mathbb {Z}_3 \cong A_4, \mathbb {Z}_2^3 \rtimes \mathbb {Z}_7\) and \(\mathbb {Z}_2^4 \rtimes \mathbb {Z}_5,\) respectively, given by the presentations

  • \(\mathbb {Z}_2^2 \rtimes \mathbb {Z}_3= \big \langle x,y,z \mid x^2=y^2=z^3=1, [x,y]=1, x^z=y, y^z=xy \big \rangle \),

  • \(\mathbb {Z}_2^3 \rtimes \mathbb {Z}_7= \big \langle x,y,z,v \mid x^2=y^2=z^2=u^7=1, [x,y]=[x,z]=[y,z]=1, x^u=y, y^u=z, z^u=xy \big \rangle ,\)

  • \(\mathbb {Z}_2^4 \rtimes \mathbb {Z}_5= \big \langle x,y,z,v,w \mid x^2=y^2=z^2=v^2=w^5=1, [x,y]=[x,z]=[x,v]=[y,z]=[y,v]=[z,v]=1, x^w=v, y^w=xy, z^w=yz, v^w=zv \big \rangle \).

Then, we find by the help of Magma [4] that the Haar graph \(\mathrm{H}(H/N,S)\) is not vertex-transitive where \(S=\{1,x,z,xyz\}\) for \(H/N=\mathbb {Z}_2^2 \rtimes \mathbb {Z}_3,\)\(S=\{1,x,u,xyu,xz u\}\) for \(H/N=\mathbb {Z}_2^3 \rtimes \mathbb {Z}_7,\) and \(S=\{1,x,w,xyw,xzw\}\) for \(H/N=\mathbb {Z}_2^4 \rtimes \mathbb {Z}_5\). Furthermore, a direct computation yields \(S \ne Sx\) and \(S \ne xS\) for any non-identity element \(x \in H/N\). Thus, by Lemma 3.2, \(H \notin \mathcal{BC}\). This completes the proof of the theorem. \(\square \)

Since each non-abelian group has an inner abelian subgroup, we have the following corollary by Theorem 1.2 and Lemma 3.1.

Corollary 4.6

Let G be a group in the class \(\mathcal{BC}\). Then the following hold:

  1. (i)

    Each Sylow p-subgroup of G with a prime \(p\ge 3\) is abelian.

  2. (ii)

    If G is non-abelian, then G has a subgroup isomorphic to \(D_6\), \(D_8\), \(D_{10}\) or \(Q_8\).

Now we are ready to prove Theorem 1.3, that is, every non-solvable group has a non-Cayley Haar graph, which is equivalent to saying that each group in the class \(\mathcal{BC}\) must be solvable.

Proof of Theorem 1.3

Suppose to the contrary that G is a non-solvable group in the class \(\mathcal{BC}\). We claim that G contains a non-solvable \(\{2,3,5\}\)-subgroup L.

The statement holds obviously when G is a \(\{2,3,5\}\)-group. Now assume that |G| has a prime divisor \(p > 5\). Let \(p_1,\ldots , p_m\) be all prime divisors of |G| with \(p_i>5\), and let \(P_i\) be a Sylow \(p_i\)-subgroup of G for each \(1\le i\le m\). By Corollary 4.6 (i), \(P_i\) is abelian. Consider the group \(M=P_i\langle g \rangle ,\) where \(g \in N_G(P_i)\) has order \(r^m\) for some prime r.

If \(r=p_i,\) then \(g \in P_i,\) and hence \(M=P_i\) is abelian. If \(r \ne p_i\), then M has a cyclic Sylow r-subgroup and Corollary 4.6 (ii) implies that the \(\{p_i,r\}\)-group M (\(p_i>5\)) is abelian. These yield that \(P_i \le Z(N_G(P_i)),\) and thus by Burnside’s p-complement theorem (see [10, Theorem 7.4.3]), \(G=K_iP_i,\) where \(K_i \unlhd G\) and \(K_i \cap P_i=1\). Set \(L=K_1\cap \cdots \cap K_m\). Then \(L\unlhd G\), and since |L| divides \(|K_i|\) and \(p_i\) does not divide \(|K_i|\) for each \(1\le i\le m\), L is a \(\{2,3,5\}\)-subgroup. Since \(G/L\lesssim G/K_1\times \cdots \times G/K_m\cong P_1\times \cdots \times P_m\) (see [10, Exercises 10, page 14]), G / L is solvable, and since G is non-solvable, L is non-solvable, as claimed. (Recall that, given two groups A and B,  we write \(A \lesssim B\) if A is isomorphic to a subgroup of B.)

Since L is non-solvable, it has a composition factor T which is a non-abelian simple \(\{2,3,5\}\)-group. By Corollary 3.3, in order to arrive at a contradiction, it is enough to show that there exists a subset \(S \subset T\) such that the Haar graph \(\mathrm{H}(T,S)\) is not vertex-transitive and \(S \ne Sx\) and \(S \ne xS\) for any non-identity element \(x \in T\).

By [13, Theorem I], T is isomorphic to one of the following groups: \(A_5, \, A_6\) and \(\mathrm{PSU}(4,2)\). It follows that \(A_4 < T\). This is obvious if \(T=A_5\) or \(A_6\) and can be checked for \(T=\mathrm{PSU}(4,2)\) in [6, p. 24].

Now, consider the graph \(\Gamma =\mathrm{H}(T,S)\) where \(S \subset A_4 < T,\)\(A_4=\big \langle x,y,z \mid x^2=y^2=z^3=1, [x,y]=1, x^z=y, y^z=xy \big \rangle \) and \(S=\{1,x,z,xyz\}\). Notice that the graph \(\mathrm{H}(\langle S \rangle ,S)\) appeared already in the proof of Theorem 1.2. We observed that \(\mathrm{H}(\langle S \rangle ,S)\) is not vertex-transitive, this can be checked by Magma [4], and also, \(S \ne Sx\) and \(S \ne xS\) for any \(x \in A_4, x \ne 1\). This implies at once that \(\Gamma \) is not vertex-transitive as well. Suppose that \(S=Sx\) or \(S=xS\) for \(x \in T\). Since \(1 \in S,\) it follows that \(x \in S \subset A_4,\) and hence \(x=1\). This completes the proof of the theorem. \(\square \)