Improving fermion mass hierarchy in grand gauge–Higgs unification with localized gauge kinetic terms

Abstract

Grand gauge–Higgs unification of five dimensional SU(6) gauge theory on an orbifold $S^1/Z_2$ with localized gauge kinetic terms is discussed. The Standard model (SM) fermions on one of the boundaries and some massive bulk fermions coupling to the SM fermions on the boundary are introduced, so that they respect an SU(5) symmetry structure. The SM fermion masses including top quark are reproduced by mild tuning the bulk masses and parameters of the localized gauge kinetic terms. Gauge coupling universality is not guaranteed by the presence of the localized gauge kinetic terms and it severely constrains the Higgs vacuum expectation value. Higgs potential analysis shows that the electroweak symmetry breaking occurs by introducing additional bulk fermions in simplified representations. The localized gauge kinetic terms enhance the magnitude of the compactification scale, which helps Higgs boson mass large. Indeed the observed Higgs boson mass 125 GeV is obtained.

1 Introduction

Gauge–Higgs unification (GHU) [1,2,3,4,5,6] is one of the candidates among the physics beyond the Standard Model (SM), which solves the hierarchy problem by identifying the SM Higgs field with one of the extra spatial component of the higher dimensional gauge field. In this scenario, the most appealing feature is that physical observables in Higgs sector are calculable and predictable regardless of its non-renormalizablity. For instance, the quantum corrections to Higgs mass and Higgs potential are known to be finite at one-loop [7,8,9,10,11] and two-loop [12,13,14] thanks to the higher dimensional gauge symmetry. Rich structures of the theory and its phenomenology have been investigated [15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39].

The hierarchy problem was originally addressed in grand unified theory (GUT) as a problem how the discrepancy between the GUT scale and the weak scale are kept and stable under quantum corrections. Therefore, the extension of GHU to grand unification is a natural direction to explore. One of the authors discussed a grand gauge–Higgs unification (GGHU) [40],¹ where the five dimensional SU(6) GGHU was considered and the SM fermions were embedded in zero modes of SU(6) multiplets in the bulk. This embedding was very attractive in that it was a minimal matter content without massless exotic fermions absent in the SM, namely a minimal anomaly-free matter content. However, a crucial drawback was found. The down-type Yukawa couplings and the charged lepton Yukawa couplings are not allowed since the left-handed $SU(2)_L$ doublets and the right-handed $SU(2)_L$ singlets in the down-type sector are embedded into different SU(6) multiplets. As a result, the down-type Yukawa coupling in GHU originated from the gauge coupling cannot be allowed. This feature seems to be generic in GHU as long as the SM fermions are embedded into the bulk fermions. Fortunately, another approach to generate Yukawa coupling in a context of GHU has been known [51, 52]. In this approach, the SM fermions are introduced on the boundaries (i.e. fixed point in an orbifold compactification). We also introduce massive bulk fermions, which couple to the SM fermions through the mass terms on the boundary. Integrating out these massive bulk fermions leads to non-local SM fermion masses, which are proportional to the bulk to boundary couplings and exponentially sensitive to their bulk masses. Then, the SM fermion mass hierarchy can be obtained by very mild tuning of bulk masses.

Along this line, we have improved an SU(6) grand GHU model [40] in our previous paper [53], where the SM fermion mass hierarchy except for top quark mass is obtained by introducing the SM fermions on the boundary as SU(5) multiplets, the four types of massive bulk fermions in SU(6) multiplets coupling to the SM fermions. Furthermore, we have shown that the electroweak symmetry breaking and an observed Higgs mass can be realized by introducing additional bulk fermions with large dimensional representation. In GHU, generation of top quark mass is difficult since Yukawa coupling is originally gauge coupling and fermion mass is W boson mass as it stands. The following is well known to overcome this problem that if top quark has a mixing with a four rank tensor representation, an enhancement of group theoretical factor helps a realization of top quark mass [54]. We have attempted to analyze for the cases of three and four rank tensor representations, but an observed top quark mass was not obtained.

As another known approach [52], introducing the localized gauge kinetic terms has enhancement effects on fermion masses. In this paper, we follow this approach. We consider an SU(6) GGHU model in our previous paper [53], where the SM fermions are localized 4D fields on the boundary and the four types of massive bulk fermion. The localized gauge kinetic terms on the boundaries are added to this model. Once the localized gauge kinetic terms are introduced, the zero mode wave functions of gauge fields are distorted and the gauge coupling universality is not guaranteed. We will find a parameter space where the gauge coupling constant between fermions and a gauge field, the cubic and the quartic self-coupling constants are almost universal. Then, we will show that the fermion mass hierarchy including top quark mass is indeed realized by appropriately choosing the bulk mass parameters and the size of the localized gauge kinetic terms. The correct pattern of electroweak symmetry breaking will be obtained by introducing extra bulk fermions as in our previous paper [53], but their representations become greatly simplified.

This paper is organized as follows. In the next section, we briefly describe the gauge and Higgs sectors of our model. In Sect. 3, the localized gauge kinetic terms are introduced and discuss the mass spectrum of gauge fields including their effects. In models with the localized gauge kinetic terms, the gauge coupling universality is not guaranteed. We will find a parameter space where the gauge couplings are almost universal. In Sect. 4, after briefly explaining the generation mechanism of the SM fermion masses, it is shown that the SM fermion masses including top quark can be reproduced by mild tuning of bulk masses and parameters of the localized gauge kinetic terms. One-loop Higgs potential is calculated and investigated in Sect. 5. We will show that the observed pattern of the electroweak symmetry breaking and Higgs boson mass are realized by introducing some extra bulk fermions. Final section is devoted to our conclusions.

2 Gauge and Higgs sector of our model

In this section, we briefly explain gauge and Higgs sectors of SU(6) GHU model [40]. We consider a five dimensional (5D) SU(6) gauge theory with an extra space compactified on an orbifold $S^1/Z_2$ whose radius and coordinate are denoted by R and y, respectively. The orbifold has fixed points at $y=0,\pi R$ and their $Z_2$ parities are given as follows.

$$\begin{aligned} P= & {} \text{ diag }(+,+,+,+,+,-) \, \, \text{ at }~y=0, \nonumber \\ P'= & {} \text{ diag }(+,+,-,-,-,-) \, \, \text{ at }~y=\pi R. \end{aligned}$$

(1)

We assign the $Z_2$ parity for the gauge field and the scalar field as $A_{\mu }(-y)=PA_{\mu }(y)P^\dag$, $A_y(-y)=-PA_y(y)P^\dag$, which implies that their fields have the following parities in components,

$$\begin{aligned} A_{\mu } = \left( \begin{array}{cc|ccc|c} (+,+) &{}(+,+) &{} (+,-) &{} (+,-) &{} (+,-) &{} (-,-) \\ (+,+) &{}(+,+) &{} (+,-) &{} (+,-) &{} (+,-) &{} (-,-) \\ \hline (+,-) &{}(+,-) &{} (+,+) &{} (+,+) &{} (+,+) &{} (-,+) \\ (+,-) &{}(+,-) &{} (+,+) &{} (+,+) &{} (+,+) &{} (-,+) \\ (+,-) &{}(+,-) &{} (+,+) &{} (+,+) &{} (+,+) &{} (-,+) \\ \hline (-,-) &{}(-,-) &{} (-,+) &{} (-,+) &{} (-,+) &{} (+,+) \end{array} \right) ,\nonumber \\ \end{aligned}$$

(2)

$$\begin{aligned} A_{y} = \left( \begin{array}{cc|ccc|c} (-,-) &{}(-,-) &{} (-,+) &{} (-,+) &{} (-,+) &{} (+,+) \\ (-,-) &{}(-,-) &{} (-,+) &{} (-,+) &{} (-,+) &{} (+,+) \\ \hline (-,+) &{}(-,+) &{} (-,-) &{} (-,-) &{} (-,-) &{} (+,-) \\ (-,+) &{}(-,+) &{} (-,-) &{} (-,-) &{} (-,-) &{} (+,-) \\ (-,+) &{}(-,+) &{} (-,-) &{} (-,-) &{} (-,-) &{} (+,-) \\ \hline (+,+) &{}(+,+) &{} (+,-) &{} (+,-) &{} (+,-) &{} (-,-) \end{array} \right) ,\nonumber \\ \end{aligned}$$

(3)

where $(+,-)$ means that $Z_2$ parity is even (odd) at $y=0~(y = \pi R)$ boundary, for instance. We note that only the fields with $(+, +)$ parity has a 4D massless zero mode since the wave function takes a form of $\cos (ny/R)$ after the Kaluza-Klein (KK) expansion. The $Z_2$ parity of $A_\mu$ indicates that SU(6) gauge symmetry is broken to $SU(3)_C \times SU(2)_L \times U(1)_Y \times U(1)_X$ by the combination of the symmetry breaking pattern at each boundary,

$$\begin{aligned}&SU(6)\rightarrow SU(5)\times U(1)_X \, \, \text{ at }~y=0, \end{aligned}$$

(4)

$$\begin{aligned}&SU(6)\rightarrow SU(2)\times SU(4) \, \, \text{ at }~y=\pi R. \end{aligned}$$

(5)

The hypercharge $U(1)_Y$ is contained in Georgi–Glashow SU(5) GUT, which is an upper-left $5 \times 5$ submatrix of $6 \times 6$ matrix. Thus, we have a relation of the gauge coupling

$$\begin{aligned} g_3 = g_2 = \sqrt{\frac{5}{3}}g_Y \end{aligned}$$

(6)

at the unification scale, which will not be so far from the compactification scale. $g_{3,2,Y}$ are the gauge coupling constants for $SU(3)_C, SU(2)_L, U(1)_Y$, respectively. This coupling relation implies that the weak mixing angle is the same as that of Georgi–Glashow SU(5) GUT model, $\sin ^2 \theta _W=3/8$ ($\theta _W:$weak mixing angle) at the unification scale.

The SM $SU(2)_L$ Higgs doublet field is identified with a part of an extra component of gauge field $A_y$ as shown below,

(7)

We suppose that a vacuum expectation value (VEV) of the Higgs field is taken to be in the 28-th generator of SU(6), $\langle A_y^a \rangle = \frac{2\alpha }{Rg}\delta ^{a\,28}$, where g is a 5D SU(6) gauge coupling constant and $\alpha$ is a dimensionless constant. The VEV of Higgs field is given by $\langle H \rangle = \frac{\sqrt{2}\alpha }{Rg}$. We note that the doublet-triplet splitting problem is solved by the orbifolding since the $Z_2$ parity of the colored Higgs field is $(+, -)$ and it become massive [55].

The Higgs couplings to the gauge bosons and the fermons are generated from the gauge interactions,

$$\begin{aligned}&-\frac{1}{4}{\mathcal {F}}^a_{MN}{\mathcal {F}}^{a\,MN} \supset -\frac{1}{2}{\mathcal {F}}^a_{\mu y}{\mathcal {F}}^{a\,\mu y} \supset -\frac{1}{2}A^a_{\mu }\nonumber \\&\quad \times \left( \partial _y+f^{adb}A_{y}^d \right) \left( \partial _y+f^{bec}A_{y}^e\right) A^{\mu \,c}, \end{aligned}$$

(8)

(9)

where M, $N=\{\mu ,y\}$ , $\mu =0,1,2,3$ , $y=5$ and subscript a,b, c, d, e denote the gauge index for SU(6). After the Higgs field has the VEV, Eqs. (8) and  (9) become the mass terms, which mass eigenvalues are $m_n(q\alpha )=\frac{n+\nu +q\alpha }{R}$, where n is KK mode, $\nu =0$ or 1/2. q is an integer charge and determined by the representation to which the field with coupling to Higgs field belongs. More precisely, the integer charge q is determined by the $SU(2)_L$ representation. In case where the field with coupling to Higgs field belongs to N+1 representation of $SU(2)_L$, the integer charge q is equal to N. For instance, in the case of 6 representation of SU(6), since the corresponding branching rule under $SU(6)\rightarrow SU(3)_C\times SU(2)_L$ is given by $\mathbf{6 } \rightarrow (3,1) \oplus (1,2) \oplus (1,1)$, this representation has four states with $q=0$ and a state with $q=1$.

Some comments on $U(1)_X$ gauge symmetry which remains unbroken by orbifolding are given. We first note that the $U(1)_X$ is in general anomalous since the massless fermions are only the SM fermions and their $U(1)_X$ charge assignments are not anomaly-free. It is easy to cancel the anomaly by adding appropriate number of the SM singlet fermions with some $U(1)_X$ charge. In our model, $U(1)_X$ is supposed to be broken by some mechanism.

3 Localized gauge kinetic term

As mentioned in the introduction, we introduce localized gauge kinetic terms at $y=0$ and $y=\pi R$ to reproduce a realistic top quark mass. Lagrangian for SU(6) gauge field is

$$\begin{aligned} {{\mathcal {L}}}_g= & {} \frac{1}{4} {\mathcal {F}}^{a\,MN}{\mathcal {F}}^a_{MN} - 2 \pi R c_1\delta (y) \frac{1}{4}{\mathcal {F}}^{b\,\mu \nu }{\mathcal {F}}^b_{\mu \nu } \nonumber \\&-2 \pi R c_2\delta (y-\pi R) \frac{1}{4}{\mathcal {F}}^{c\,\mu \nu }{\mathcal {F}}^c_{\mu \nu }, \end{aligned}$$

(10)

where the first term is the gauge kinetic term in the bulk and $M,N=0,1,2,3,5$. The second and the third terms are gauge kinetic terms localized at fixed points and $\mu , \nu = 0,1,2,3$. $c_{1,2}$ are dimensionless free parameters. The subscript a, b, c denote the gauge indices for $SU(6), SU(5)\times U(1), SU(2) \times SU(4)$. Note that the localized gauge kinetic terms have only to be invariant under an unbroken symmetry on each fixed point.

3.1 Mass spectrum in gauge sector

Because of the presence of localized gauge kinetic terms, the mass spectrum of the SM gauge field becomes very complicated. In particular, their effects for a periodic sector and an anti-periodic sector are different, where the periodic sector means the fields satisfying a condition $A(y+\pi R)=A(y)$ or those with parity $(P,P')=(+,+), (-,-)$, while the anti-periodic sector means the fields satisfying a condition $A(y+\pi R)=-A(y)$ or those with parity $(+,-), (-,+)$. This difference originates from the boundary conditions for wave functions with a definite charge q, $f_n(y; q \alpha )$. In a basis where 4D gauge kinetic terms are diagonal, they are found as $f_n(y+\pi R;q\alpha )=e^{2i\pi q\alpha }f_n(y;q\alpha )$ in periodic sector and $f_n(y+\pi R;q\alpha )=e^{2i\pi (q\alpha +1/2)}f_n(y;q\alpha )$ in anti-periodic sector. Moreover, the wave functions in the same basis satisfy

$$\begin{aligned}&\left[ \partial _y^2+m_n^2(q \alpha ) \left( 1+2\pi R c_1\delta (y)+2\pi R c_2 \delta (y-\pi R)\right) \right] \nonumber \\&\quad f_n(y;q \alpha )=0, \end{aligned}$$

(11)

where $m_n(q \alpha )$ is the KK mass. By solving Eq. (11) with the periodic (anti-periodic) boundary conditions, the wave functions and equations determining the KK mass spectrum are obtained [56]. Solving first Eq. (11) without boundary terms, we obtain

$$\begin{aligned}&f_n(y;q\alpha ) = {\mathcal {N}}_n(q\alpha +\nu )\nonumber \\&\quad {\left\{ \begin{array}{ll} \cos (m_n y) + \beta _n^- \sin (m_n y) \, ,\, y \in [-\pi R,0] \\ \cos (m_n y) - \beta _n^+ \sin (m_n y) \, ,\, y \in [0,\pi R]. \\ \end{array}\right. } \end{aligned}$$

(12)

where ${\mathcal {N}}_n$ is a normalization factor determined by $\int _{0}^{2\pi R}\left| f_n\right| ^2 dy=1$. $\beta _n^{\pm }$ are integration constants. Continuity conditions at $y=0, \pi R$ using the above solution $f_n(y; q \alpha )$ lead to

$$\begin{aligned} \beta _n^{\pm }= & {} e^{\pm i\pi (q\alpha +\nu )} \sec (\pi (q\alpha +\nu ))(\pi Rm_n)c_1\nonumber \\&\mp \, i \tan (\pi (q\alpha +\nu )) \cot (\pi Rm_n) \end{aligned}$$

(13)

and eliminating $\beta _n^{\pm }$ in the continuity conditions at $y=0, \pi R$, the equations determining the KK mass spectrum

$$\begin{aligned}&2\left( 1-c_1c_2\xi _n^2\right) \sin ^2\xi _n\nonumber \\&\quad +\left( c_1+c_2\right) \xi _n \sin 2\xi _n-2 \sin ^2(\pi (q\alpha +\nu ))=0 \end{aligned}$$

(14)

is obtained. $\nu$ is 0 (1/2) for the periodic (anti-periodic) sector, and $\xi _n=\pi R m_n$.

Since $m_0$ is around weak scale ($\sim 100$ GeV) and 1/R is more than 1 TeV, it is reasonable to suppose $\xi _0 \ll 1$. From this observation, we can find an approximate form of $\xi _0$ as

$$\begin{aligned} \xi _0 \sim \frac{\sin (\pi (q\alpha +\nu ))}{\sqrt{1+c_1+c_2}}. \end{aligned}$$

(15)

For instance, the W boson is the gauge boson whose q and $\nu$ are 1 and 0, respectively, therefore, the W boson mass $m_W$ is given by

$$\begin{aligned} m_W = \frac{\sin (\pi \alpha )}{\pi R\sqrt{1+c_1+c_2}}. \end{aligned}$$

(16)

Moreover, the copmactification scale 1/R is determined by this reration when VEV $\alpha$ is obtained (Fig. 1) and 1/R can be large by decreasing $\alpha$. Since the first KK particle has not observed, the copmactification scale should be larger than a few TeV. Thus, the following condition is needed:

$$\begin{aligned} \alpha \le 0.1. \end{aligned}$$

(17)

This condition is very rough estimation. We would like to emphasize here that $\alpha$ is very small, which will be important in fitting the fermion mass hierarchy.

Fig. 1

Higgs VEV $\alpha$ in the range $0 \le c_1+c_2 \le 40$ (horizontal axis) and $3000 \text{ GeV } \le 1/R \le 10000 \text{ GeV }$ (vertical axis)

Fig. 2

Ratio between $g_{\text { eff}}$ and the effective gauge coupling constant between the SM fermions and $q=1$ gauge boson in the range $0 \le c_1 \le 20$ (horizontal axis) and $0 \le c_2 \le 20$ (vertical axis). $\alpha$ is taken to be 0.3 (upper left), 0.1 (upper right) and 0.05 (lower left)

3.2 Gauge coupling universality

In the SM, the gauge coupling constant between fermions and a gauge boson, cubic and quartic self-interaction gauge couplings are universal. However, in our model, the universality of 4D gauge coupling is not maintained

since the wave functions for massive gauge bosons are distorted from the flat wave functions by the localized gauge kinetic terms and 4D gauge couplings depend on the integral of the wave functions. Therefore, we have to search for a parameter region where the universality is valid. The gauge coupling between the SM fermions localized at $y=0$ and a 4D gauge boson (KK zero mode: $n=0$) is given by

$$\begin{aligned} g_{4\,\text {gff}}(0;q) = g_5\frac{|f_0(0;q\alpha )|}{\sqrt{Z_0(q\alpha )}}. \end{aligned}$$

(18)

Similarly, the 4D cubic and quartic self-interaction gauge couplings are given by

$$\begin{aligned}&g_{4\,\text {ggg}}(q_i,q_j,q_k) = g_5\int dy [1+2\pi R c_1\delta (y)\nonumber \\&+2\pi R c_2\delta (y-\pi R)] \frac{|f_0(y;q_i \alpha )|}{\sqrt{Z_0(q_i \alpha )}} \frac{|f_0(y;q_j \alpha )|}{\sqrt{Z_0(q_j \alpha )}} \frac{|f_0(y;q_k \alpha )|}{\sqrt{Z_0(q_k \alpha )}}\nonumber \\ \end{aligned}$$

(19)

and

$$\begin{aligned}&g_{4\,\text {gggg}}(q_i,q_j,q_k,q_l) \nonumber \\&\quad = g_5 \left( \int dy [1+2\pi R c_1\delta (y)+2\pi R c_2\delta (y-\pi R)]\right. \nonumber \\&\qquad \left. \times \frac{|f_0(y;q_i \alpha )|}{\sqrt{Z_0(q_i \alpha )}} \frac{|f_0(y;q_j \alpha )|}{\sqrt{Z_0(q_j \alpha )}} \frac{|f_0(y;q_k \alpha )|}{\sqrt{Z_0(q_k \alpha )}} \frac{|f_0(y;q_l \alpha )|}{\sqrt{Z_0(q_l \alpha )}}\right) ^{1/2},\nonumber \\ \end{aligned}$$

(20)

where $Z_n(q \alpha )$ is a wave function renormalization factor for the gauge field with a charge q

$$\begin{aligned} Z_n(q\alpha ) {=} 1 {+} 2\pi R c_1 \left| f_n(0;q\alpha ) \right| ^2 {+} 2\pi R c_2 \left| f_n(\pi R;q\alpha ) \right| ^2.\nonumber \\ \end{aligned}$$

(21)

In the case of $q=0$ corresponding to the photon and the gluon in the SM, Eq. (18) is simplified. According to Eq. (14), we find $m_0(0)=0$, which implies $f_0(y;0) = {\mathcal {N}}_n(0) = \frac{1}{\sqrt{2 \pi R}}$ and $Z_0(0) = 1 + 2\pi R c_1 \left| f_0(0;0) \right| ^2 + 2\pi R c_2 \left| f_0(\pi R;0) \right| ^2 = 1 + c_1 +c_2$. Therefore, the gauge coupling universality is valid for $q=0$

$$\begin{aligned} g_{4\,\text {gff}}(0;0)= & {} g_{4\,\text {ggg}}(0,0,0) = g_{4\,\text {gggg}}(0,0,0,0) \nonumber \\= & {} \frac{g_5}{\sqrt{1+c_1+c_2}\sqrt{2 \pi R}}\equiv g_{\text { eff}}. \end{aligned}$$

(22)

Then, we have to search for the parameter space where the gauge coupling universality is kept for a nonvanishing charge q. Figure 2 shows the ratio between $g_{\text { eff}}$ and the gauge coupling constant between the SM fermions and the W boson ($q=1$). The free parameters for the localized gauge kinetic terms are taken in the range $0\le c_1+c_2 \le 40$. In the cases of $\alpha \le 0.1$ or $1/2< r = c_1/(c_1+c_2) < 1$, the ratio is almost unity in a good approximation. As for the cubic and quartic self-interaction gauge coupling constants, the ratio is also almost unity in the same parameters. Since $\alpha$ is restricted to the range $\alpha \le 0.1$ which is explained in Sect. 3.1, we do not consider the case $1/2< r < 1$ hereafter. After all, the universality of gauge coupling constants can be maintained in the range of $\alpha \le 0.1$.

4 Fermion masses

4.1 Generation mechanism of the SM fermion masses

In a paper [40], the SM fermions were embedded in SU(6) multiplets in the bulk, which was a minimal matter content without massless exotic fermions absent in the SM. However, the down-type and the charged lepton Yukawa couplings are not allowed from SU(6) gauge interactions since the left-handed $SU(2)_L$ doublets and the right-handed $SU(2)_L$ singlets in the down-type sector are embedded into different SU(6) multiplets. Because of this reason, we take another approach.

The SM quarks and leptons are embedded into SU(5) multiplets localized at $y=0$ boundary, which are three sets of decouplet, anti-quintet and singlet $\chi _{10}$, $\chi _{5^{*}}$ and $\chi _{1}$.

We also introduce three types of bulk fermions $\Psi$ and ${\tilde{\Psi }}$ (referred as “mirror fermions”) with opposite $Z_2$ parities each other shown in Table 1 and constant mass term such as $M{\bar{\Psi }} {\tilde{\Psi }}$ in the bulk to avoid exotic massless fermions. Without these mirror fermions and mass terms, we necessarily have extra exotic massless fermions with the SM charges after an orbifold compactification. In this setup, we have no massless chiral fermions from the bulk and its mirror fermions. The massless fermions are only the SM fermions and the gauge anomalies for the SM gauge groups are trivially canceled. In order to realize the SM fermion masses, the boundary localized mass terms between the SM fermions localized at $y=0$ and the bulk fermions are necessary. To allow such localized mass terms, we have to choose appropriate SU(6) representations for bulk fermions carefully. Note that the mirror fermions have no coupling to the SM fermions. Table 1 shows the representations for bulk and mirror fermions introduced in our model in addition to the SM fermions, which corresponds to the matter content for one generation. Totally, three copies of them are present in our model.

Table 1 Representation of bulk fermions and the corresponding mirror fermions. $P_i$ are parity of bulk fermion for i representation in SU(6) ($P_i=\pm 1$). R in $R^{(+,+)}$ means an SU(6) representation of the bulk fermion. $r_i$ in $r_1 \oplus r_2$ are SU(5) representations

Table 2 20 bulk fermion and SM fermions per a generation. $P_{20}$ is parity of bulk fermion for 20 ($P_{20}=\pm 1$). R in $R^{(+,+)}$ means an SU(6) representation of the bulk fermion. $r_{1,2}$ in $(r_1, r_2)_{a}$ are SU(3), SU(2) representations in the SM, respectively. a is U(1)$_Y$ charges

Table 3 15 bulk fermion and SM fermions per a generation. $P_{15}$ is parity of bulk fermion for 15 ($P_{15}=\pm 1$). $r_{1,2}$ in $(r_1, r_2)_{a}$ are SU(3), SU(2) representations in the SM, respectively. a is U(1)$_Y$ charges

Lagrangian for the fermions is given by

$$\begin{aligned} {{\mathcal {L}}}_{{\text { matter}}}= & {} \sum _{a=20, 15, 6} \left[ {\overline{\Psi }}_ai\Gamma ^MD_M\Psi _a \right. \nonumber \\&\left. +\overline{{\tilde{\Psi }}}_ai\Gamma ^MD_M{\tilde{\Psi }}_a + \left( \frac{ \lambda _a}{\pi R}{\overline{\Psi }}_a {\tilde{\Psi }}_a + {\text { h.c.}} \right) \right] \nonumber \\&+\delta (y)\left[ {\overline{\chi }}_{10}i\Gamma ^{\mu }D_{\mu }\chi _{10} +{\overline{\chi }}_{5^{*}}i\Gamma ^{\mu }D_{\mu }\chi _{5^{*}} +{\overline{\chi }}_1i\Gamma ^{\mu }D_{\mu }\chi _1\right. \nonumber \\&+\sqrt{\frac{2}{\pi R}}\left\{ \epsilon _{20} \left( {\overline{\chi }}_{10}\Psi _{10\subset 20} + {\overline{\chi }}_{10}^{c}\Psi _{10^{*} \subset 20} \right) \right. \nonumber \\&+\epsilon _{15} \left( {\overline{\chi }}_{10}\Psi _{10\subset 15} + {\overline{\chi }}_{5^{*}}^{c}\Psi _{5\subset 15} \right) \nonumber \\&\left. \left. +\epsilon _6 \left( {\overline{\chi }}_{5^{*}}\Psi _{5\subset 6} + {\overline{\chi }}_{1}\Psi _{1\subset 6} \right) +\text{ h.c. } \right\} \right] , \end{aligned}$$

(23)

where the five-dimensional gamma matrices $\Gamma ^M$ is given by $(\Gamma ^{\mu },\Gamma ^y)=(\gamma ^{\mu },i\gamma ^5)$.

The first line is Lagrangian for the bulk and mirror fermions, and the remaining terms are Lagrangian localized on $y=0$ boundary. Note that the subscript $``a''$ denotes the SU(6) representations of the bulk and mirror fermions. The bulk masses between the bulk and the mirror fermions are normalized by $\pi R$ and expressed by the dimensionless parameter $\lambda _a$. The last two lines are mixing mass terms between the bulk fermions and the SM fermions. $\Psi _{M\subset N}$ is a bulk fermion for M in SU(5) representation and N means SU(6) representation. $\epsilon _i$ are the strength of the mixing term between the bulk fermion and the SM fermion. Note that all of the boundary terms respect SU(5) GHU symmetry structure. Integrating out y-direction after KK expansion of bulk fermions leads to the following 4D effective Lagrangian.

(24)

where $\Psi ^{(n)}({\tilde{\Psi }}^{(n)})$ represents a n-th KK mode of bulk (mirror) fermion, and $\psi _{{\text { SM}}}$ is a SM fermion. $P_{L,R}$ are chiral projection operators and $\kappa _{L,R}$ are some constants, in this model $\kappa _{L,R}=0$ or 1. $m_n(q\alpha ) = \frac{n + \nu + q\alpha }{R}$ denotes the sum of the ordinary KK mass and the electroweak symmetry breaking mass proportional to the Higgs VEV $\alpha$ where $\nu =0$ or 1/2. The charge q is determined by the representation which the fermion belongs to. The mass spectrum of bulk and mirror fermions is totally given by $m_n^2 = \left( \frac{\lambda }{\pi R}\right) ^2 + m_n(q\alpha )^2$. Note that the Lagrangian (24) is illustrated for a particular bulk and mirror fermion as an example.

A comment on the bulk mass spectrum $m_n^2 = \left( \frac{\lambda }{\pi R}\right) ^2 + m_n(q\alpha )^2$ is given. This spectrum is not exactly correct in the case that the mixings between the bulk and the boundary fermions are large. Following the argument in [52], we also assume in this paper that the physical mass induced for the boundary fields is much smaller than the masses of the bulk fields. In this case, the effects of the mixing on the spectrum for the bulk fields can be negligible and the spectrum $m_n^2 = \left( \frac{\lambda }{\pi R}\right) ^2 + m_n(q\alpha )^2$ is a good approximation.

In order to derive the SM fermion masses, we need the quadratic terms in the effective Lagrangian for the SM fermion.

$$\begin{aligned} \mathcal{L}_{{\text { SM}}} \supset {\overline{\psi }}_{{\text { SM}}} \sum _i K_i \psi _{{\text { SM}}} \end{aligned}$$

(25)

with

(26)

where $x \equiv \pi R p$ and

$$\begin{aligned} f_0^{(+)}\left( \sqrt{x^2 + \lambda ^2}, q \alpha \right)\equiv & {} \sum ^{\infty }_{n = - \infty } \frac{1}{\sqrt{x^2 + \lambda ^2} + i \pi (n + q \alpha )} \nonumber \\= & {} \text{ coth }\left( \sqrt{x^2 + \lambda ^2} + i \pi q\alpha \right) , \end{aligned}$$

(27)

$$\begin{aligned} f_0^{(-)}(\sqrt{x^2 + \lambda ^2}, q \alpha )\equiv & {} \sum ^{\infty }_{n = - \infty } \frac{1}{\sqrt{x^2 + \lambda ^2} + i \pi (n+ 1/2 + q \alpha )} \nonumber \\= & {} \text{ tanh }\left( \sqrt{x^2 + \lambda ^2} + i \pi q\alpha \right) . \end{aligned}$$

(28)

In deriving ${{{\mathcal {L}}}}_{{\text { SM}}}$, we simply took the large bulk mass limit $\frac{\lambda ^2}{(\pi R)^2} \gg p^2$ so that the mixings of the SM fermions with non-zero KK modes become negligibly small.

Integrating out all massive bulk fermions and normalizing the kinetic term to be canonical, we obtain the physical mass for the SM fermions.

$$\begin{aligned} m^a_{{\text { phys}}}= & {} \left. \frac{m^a}{\sqrt{Z_L^a Z_R^a}} \right| _{x \ll \lambda } \nonumber \\\simeq & {} \sqrt{1+c_1+c_2} m_W e^{- \lambda }~(a=u, d, e, \nu ) \end{aligned}$$

(29)

where the bare mass and the wave function renormalization factors are

$$\begin{aligned} m^a= & {} \sum _i \frac{\epsilon _i^2\kappa _L^i\kappa _R^i}{\pi R} \text{ Im } f_0^{(\pm P_i)}\left( \sqrt{x^2 + \lambda ^2_i}, q_i \alpha \right) , \end{aligned}$$

(30)

$$\begin{aligned} Z^a_{L, R}= & {} 1+ \sum _i \frac{\epsilon _i^2 \kappa ^i_{L,R}}{\sqrt{x^2 + \lambda _i^2}} \text{ Re } f_0^{(\pm P_i)}\left( \sqrt{x^2 + \lambda _i^2}, q_i \alpha \right) . \nonumber \\ \end{aligned}$$

(31)

The summation in $Z^a_{L, R}$ is taken for all the bulk fields contributing to mass $m^a$ and its precise expressions are explicitly shown in the next section.

Table 4 6 bulk fermion and SM fermions per a generation. $P_{6}$ is parity of bulk fermion for 6 ($P_{6}=\pm 1$). $r_{1,2}$ in $(r_1, r_2)_{a}$ are SU(3), SU(2) representations in the SM, respectively. a is U(1)$_Y$ charges

In this model, these are explicitly given

$$\begin{aligned} m^{u}&= \frac{\epsilon _{{\text {20}}}^2}{\pi R} \left[ \text{ Im }f_0^{(P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},\alpha \right) \right. \nonumber \\&\quad \left. + \text{ Im }f_0^{(-P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},\alpha \right) \right] ,\nonumber \\ Z^{u}_L&= 1 +\frac{\epsilon _{{\text {20}}}^2}{\sqrt{x^2 + \lambda _{20}^2}} \left[ \text{ Re }f_0^{(P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},\alpha \right) \right. \nonumber \\&\quad \left. +\text{ Re }f_0^{(-P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},\alpha \right) \right] \nonumber \\&\quad +\frac{\epsilon _{{\text {15}}}^2}{\sqrt{x^2 + \lambda _{15}^2}} \text{ Re }f_0^{(-P_{15})}\left( \sqrt{x^2+\lambda _{15}^2},0\right) ,\end{aligned}$$

(32)

$$\begin{aligned} Z^{u}_R&= 1 +\frac{\epsilon _{{\text {20}}}^2}{\sqrt{x^2 + \lambda _{20}^2}} \left[ \text{ Re }f_0^{(P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},\alpha \right) \right. \nonumber \\&\quad \left. +\text{ Re }f_0^{(-P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},\alpha \right) \right] \nonumber \\&\quad +\frac{\epsilon _{{\text {15}}}^2}{\sqrt{x^2 + \lambda _{15}^2}} \text{ Re }f_0^{(P_{15})}\left( \sqrt{x^2+\lambda _{15}^2},0\right) ,\nonumber \\ m^{d}&= \frac{\epsilon _{{\text {15}}}^2}{\pi R}\text{ Im }f_0^{(-P_{15})}\left( \sqrt{x^2+\lambda _{15}^2},\alpha \right) ,\nonumber \\ Z^{d}_L&= 1 +\frac{\epsilon _{{\text {20}}}^2}{\sqrt{x^2 + \lambda _{20}^2}} \left[ \text{ Re }f_0^{(P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},0\right) \right. \nonumber \\&\quad \left. +\text{ Re }f_0^{(-P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},0\right) \right] \nonumber \\&\quad +\frac{\epsilon _{{\text {15}}}^2}{\sqrt{x^2 + \lambda _{15}^2}} \text{ Re }f_0^{(-P_{15})}\left( \sqrt{x^2+\lambda _{15}^2},\alpha \right) ,\\ Z^{d}_R&= 1 +\frac{\epsilon _{{\text {15}}}^2}{\sqrt{x^2 + \lambda _{15}^2}} \text{ Re }f_0^{(-P_{15})}\left( \sqrt{x^2+\lambda _{15}^2},\alpha \right) \nonumber \\&\quad +\frac{\epsilon _{{\text {6}}}^2}{\sqrt{x^2 + \lambda _6^2}} \text{ Re }f_0^{(P_{6})}\left( \sqrt{x^2+\lambda _6^2},0\right) ,\nonumber \\ m^{e}&= \frac{\epsilon _{{\text {15}}}^2}{\pi R}\text{ Im }f_0^{(P_{15})}\left( \sqrt{x^2+\lambda _{15}^2},\alpha \right) ,\nonumber \\ Z^{e}_L&= 1 +\frac{\epsilon _{{\text {15}}}^2}{\sqrt{x^2 + \lambda _{15}^2}} \text{ Re }f_0^{(P_{15})}\left( \sqrt{x^2+\lambda _{15}^2},\alpha \right) \nonumber \\&\quad +\frac{\epsilon _{{\text {6}}}^2}{\sqrt{x^2 + \lambda _6^2}} \text{ Re }f_0^{(-P_{6})}\left( \sqrt{x^2+\lambda _6^2},0\right) ,\end{aligned}$$

(33)

$$\begin{aligned} Z^{e}_R&= 1 +\frac{\epsilon _{{\text {20}}}^2}{\sqrt{x^2 + \lambda _{20}^2}} \left[ \text{ Re }f_0^{(P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},0\right) \right. \nonumber \\&\quad \left. +\text{ Re }f_0^{(-P_{20})}\left( \sqrt{x^2+\lambda _{20}^2},0\right) \right] \nonumber \\&\quad +\frac{\epsilon _{{\text {15}}}^2}{\sqrt{x^2 + \lambda _{15}^2}} \text{ Re }f_0^{(P_{15})}\left( \sqrt{x^2+\lambda _{15}^2},\alpha \right) ,\nonumber \\ m^{\nu }&= \frac{\epsilon _{{\text {6}}}^2}{\pi R}\text{ Im }f_0^{(-P_6)}\left( \sqrt{x^2+\lambda _6^2},\alpha \right) , \nonumber \\ Z^{\nu }_L&= 1 +\frac{\epsilon _{{\text {15}}}^2}{\sqrt{x^2 + \lambda _{15}^2}} \text{ Re }f_0^{(P_{15})}(\sqrt{x^2+\lambda _{15}^2},0)\nonumber \\&\quad +\frac{\epsilon _{{\text {6}}}^2}{\sqrt{x^2 + \lambda _6^2}} \text{ Re }f_0^{(-P_{6})}\left( \sqrt{x^2+\lambda _6^2},\alpha \right) ,\end{aligned}$$

(34)

$$\begin{aligned} Z^{\nu }_R&= 1 +\frac{\epsilon _{{\text {6}}}^2}{\sqrt{x^2 + \lambda _6^2}} \text{ Re }f_0^{(-P_6)}\left( \sqrt{x^2+\lambda _6^2},\alpha \right) .\nonumber \end{aligned}$$

(35)

Focusing on parity for 20, since $P_{20}=\pm 1$ have same contributions, we chose $P_{20}=1$ in this paper. In the following section, we determine the remaining parity of $P_{15}$.

4.2 Mass hierarchy between down type quark mass and charged lepton mass

In this section, we consider the mass hierarchy between down type quark mass and charged lepton mass. According to Sect. 3.1, the condition (17) is needed. Under this conditon, SM fermion mass (29) and W boson mass (16) approximately proportional to VEV $\alpha$. Therefore, the ratio of these masses does not depend on the Higgs VEV $\alpha$.

Fig. 3

Common logarithms of the ratio of physical neutrino mass and W mass $m^{\nu }_{\text {{phys}}}/m_W$. The vertical axis is $\text{ log}_{10}[\epsilon _6]$. The horizontal axis is $\lambda _{6}$. Parameters $c_1+c_2$, $\lambda _{15}$, $\epsilon _{15}$, $P_{15}$ and $P_{6}$ are taken to be 10, 1, 1, 1 and 1, respectively. Red lines mean the order of magnitude for $m^{\nu }_{\text {{phys}}}/m_W$. Blue region is an allowed region for neutrino mass

The mass hierarchy depend heavily on the parameters $P_{15}$, $P_{6}$, $\lambda _{15}$, $\lambda _{6}$, $\epsilon _{15}$ and $\epsilon _{6}$. Before the analysis of the mass hierarchy, we will obtain suitable regions for $\lambda _{6}$ and $\epsilon _{6}$ by analyzing the physical neutrino mass $m^{\nu }_{{\text {phys}}}$. The physical neutrino mass $m^{\nu }$ depend heavily on $\lambda _{6}$ and $\epsilon _{6}$. Figure 3 shows $\lambda _{6}$ and $\epsilon _{6}$ dependence on the neutrino mass $m^{\nu }$ with $c_1+c_2=10$, $\lambda _{15}=1$, $\epsilon _{15}=1$, $P_{15}=1$ and $P_{6}=1$. It is known that neutrino masses is smaller than 1 eV, so that the suitable region is $m^{\nu }_{{\text{ phys }}}/m_W < 10^{-11}$ (blue region in Fig. 3). In this region, contribution of the representation 6 are exponentially small, so that the contribution can be ignored to reproduce the SM fermion masses except for SM neutrinos. In this case, the mass hierarchy is converge to 1 by increasing $\lambda _{15}$ (Fig. 4). This figure indicates that down-type quark mass is smaller than charged leplon mass in the case of $P_{15}=1$, on the other hand, down-type quark mass is larger than charged leplon mass in the case of $P_{15}=-1$. In SM, the hierarchies are

$$\begin{aligned}&\frac{m_{{\text{ down }}}}{m_{{\text{ electron }}}} \sim 9.1>1,\nonumber \\&\frac{m_{{\text{ strange }}}}{m_{{\text{ muon }}}} \sim 0.9 <1, \frac{m_{{\text{ bottom }}}}{m_{{\text{ tauon }}}} \sim 2.3 >1, \end{aligned}$$

(36)

therefore the parity assignments of 15 representation for each generation have to be taken as

$$\begin{aligned} P_{15}^{{\text{1st }}} = -1, P_{15}^{{\text{2nd }}} = 1, P_{15}^{{\text{3rd }}} = -1. \end{aligned}$$

(37)

Fig. 4

The ratio of the physical down-type mass and the physical charged lepton mass $m^d_{\text {{phys}}}/m^e_{\text {{phys}}}$. Parameters $\epsilon _{15}$, $\lambda _{20}$ and $\epsilon _{20}$ are taken to be 1

4.3 Reproducing top quark mass

In our previous paper [53], the up-type quark masses could not be larger than W boson mass, although we had attempted some cases where top quark is embedded in higher rank representations whether the enhancement due to the group theoretical factor for the up-type quark masses can be obtained [53]. As another possibility, it is known that the sizable localized gauge kinetic terms enhances fermion masses, which might be possible to reproduce top quark mass [52]. We consider this possibility in this paper.

Figure 5 shows the bulk mass $\lambda _{20}$ dependence on the ratio of W boson mass and the physical up-type mass $m^u_{\text {{phys}}}$ given by Eq. (29). To reproduce the top quark mass, the maximum value of physical up-type quark mass has to be larger than the observed top quark mass 173 GeV. We have studied the behavior of the maximum value in the range from $c_1 + c_2 = 0$ to 20. It turns out that the conditions where $c_1+c_2$ is at least larger than 4 is necessary to reproduce the top quark mass.

Ignoring the contribution of representation 6, three masses ($m^u_{{\text {phys}}}$, $m^d_{{\text {phys}}}$, $m^e_{{\text {phys}}}$) are fitted by 4 parametars ($\lambda _{20}$, $\epsilon _{20}$, $\lambda _{15}$, $\epsilon _{15}$), so that a degree of freedom is remaind to be free. Here, we try to get suitable parameters $\lambda _{20}$, $\lambda _{15}$, $\epsilon _{15}$ by changing the value of $\epsilon _{20}$ at each generations (Fig. 6). We can find allowed parameter sets in a broad region. In the case of the first generation, the region that there is no point appears around $\epsilon _{20} \sim 0.8$ since it is difficult to reproduce the mass hierarchy between the down quark and the electron, which is lager than those of the second and the third generation. In the case of the second and the third generations, the region that there is no point appears when $\epsilon _{20}$ is below a certain value since the up-type quark mass can not be reproduced.

Fig. 5

The bulk mass $\lambda _{20}$ dependence on the ratio of the physical up-type mass and W boson mass $m^u_{\text {{phys}}}/m_W$ with $c_1+c_2=0$ (left side) and the maximum ratio in the range of $0 \le c_1 + c_2 \le 20$ (right side). Other parameters are taken to be 1. Dotted line means top quark mass: $m^u_{\text {{phys}}}/m_W=m_{\text {{top}}}/m_W\sim 2.15$

Fig. 6

Scatter plots of the parameters reproducing the SM fermion masses except for neutrino masses. The upper, middle and bottom plots show $\epsilon _{20}$ dependences of $\lambda _{20}$, $\lambda _{15}$ and $\epsilon _{15}$ for the first, the second and the third generation, respectively

5 Higgs effective potential

In this section, we calculate the effective potential for the Higgs field and study whether the electroweak symmetry breaking correctly occurs. Since the Higgs field is originally a gauge field, the potential is generated at one-loop by Coleman-Weinberg mechanism. The potential from the bulk fields is given by

$$\begin{aligned} V(\alpha ) = \sum _n (\pm g) \int \frac{d^4p_E}{(2\pi )^4} \log [p_E^2 + m_n^2] \equiv g {\mathcal {F}}^\pm (q\alpha ) \end{aligned}$$

(38)

with

$$\begin{aligned} {\mathcal {F}}^{\pm }(q\alpha ) = \pm \sum _n \int \frac{d^4p_E}{(2\pi )^4} \log [p_E^2+m_n^2], \end{aligned}$$

(39)

where overall signs $+(-)$ stand for fermion (boson) contributions, respectively. g means the spin degrees of freedom of the field running in the loop. The loop momentum $p_E$ is taken to be Euclidean.

For bulk fermions and mirror fermions, the mass spectrum is calculated as the following four types of form depending on the $Z_2$ parity and the bulk mass.

$$\begin{aligned}&m_n^2 = \frac{(n + q \alpha )^2}{R^2}, \nonumber \\&m_n^2 = \frac{(n + 1/2 + q \alpha )^2}{R^2}, \nonumber \\&m_n^2 = \frac{(n + q \alpha )^2}{R^2} + \left( \frac{\lambda }{\pi R} \right) ^2, \nonumber \\&m_n^2 = \frac{(n + 1/2 + q \alpha )^2}{R^2} + \left( \frac{\lambda }{\pi R} \right) ^2. \end{aligned}$$

(40)

The first (second) half of spectrum correspond to the spectrum of massless (massive) bulk fields. The first and third (the second and the last) types of spectrum correspond to the spectrum of the fields with (anti-)periodic boundary conditions. Using this information, we obtain the corresponding potentials [54].

$$\begin{aligned} {\mathcal {F}}^{\pm }(q\alpha )= & {} \mp \frac{3}{64\pi ^6 R^4} \sum _{k=1}^{\infty } \frac{\cos (2\pi q\alpha k)}{k^5}, \nonumber \\ {\mathcal {F}}^{\pm }_{1/2}(q\alpha )= & {} \mp \frac{3}{64\pi ^6 R^4} \sum _{k=1}^{\infty }(-1)^k \frac{\cos (2\pi q \alpha k)}{k^5}, \nonumber \\ {\mathcal {F}}^{\pm }_\lambda (q\alpha )= & {} \mp \frac{3}{64\pi ^6 R^4} \sum _{k=1}^{\infty } \frac{\cos (2\pi q \alpha k) e^{-2 k \lambda }}{k^3} \nonumber \\&\times \left[ \frac{(2\lambda )^3}{3} + \frac{2\lambda }{k}+\frac{1}{k^2}\right] ,\nonumber \\ {\mathcal {F}}^{\pm }_{1/2\lambda }(q\alpha )= & {} \mp \frac{3}{64\pi ^6 R^4}\sum _{k=1}^{\infty } (-1)^k\frac{\cos (2\pi q \alpha k)e^{-2k \lambda }}{k^3}\nonumber \\&\times \left[ \frac{(2\lambda )^3}{3}+\frac{2\lambda }{k}+\frac{1}{k^2}\right] . \end{aligned}$$

(41)

Table 5 lists the various potentials from bulk fermion and mirror fermion contributions. The coefficients in front of the each potential can be read from the branching rules in the decomposition of the SU(6) representation into $SU(3)_C \times SU(2)_L \times U(1)_Y \times U(1)_X$ representations listed in Appendix A of our previous paper [53].

Table 5 Bulk fermion and mirror fermion contribution to Higgs potential

Next, we calculate the gauge field loop contributions to the effective potential. As was mentioned in Sect. 3, the gauge boson mass spectrum is complicated because of the localized gauge kinetic terms. Here, we derive the effective potential without solving explicit KK mass spectrum of gauge fields. For the quantization of the gauge fields, we fix the gauge by using the background field method where the gauge fields are divided into the classical field $\overline{{\mathcal {{\mathcal {A}}}}}_M$ and quantum field ${\mathcal {{\mathcal {A}}}}_M$. The gauge fixing condition function is given by $G^a(x) ={\overline{D}}_M^{ab}{\mathcal {{\mathcal {A}}}}^{M\,b}(x) - \omega ^a(x)$ as usual, where $\overline{{\mathcal {{\mathcal {A}}}}}_M = \delta _{My}\frac{2\alpha }{Rg}T^{28}$, ${\overline{D}}^{M\,ab}$ is covariant derivative containing a classical field only ${\overline{D}}^{M\,ab} {\mathcal {{\mathcal {A}}}}^b_M = (\delta ^{ab}\partial ^M -g_5 f_0^{abc}\overline{{\mathcal {{\mathcal {A}}}}}^{M\,c}){\mathcal {{\mathcal {A}}}}_M^b$ and $\omega (x)$ is an arbitrary scalar function.

We note that the contributions from the gauge fields are different depending on the boundary conditions. In the periodic sector, the quadratic terms in Eq. (10) become

$$\begin{aligned} {\mathcal {L}}_{{\text { quadratic}}}= & {} \frac{1}{2}\mathcal {{\mathcal {A}}}^{a,\mu }g_{\mu \nu }({\overline{D}}_{P}{\overline{D}}^{P})^{ab}\mathcal {{\mathcal {A}}}^{b,\nu }\nonumber \\&+\frac{1}{2}\mathcal {{\mathcal {A}}}^a_y({\overline{D}}_{P}{\overline{D}}^{P})^{ab} \mathcal {{\mathcal {A}}}^b_y + \overline{\mathbf{c }}^a{\overline{D}}^{M,ab}{\overline{D}}^{bc}_{M}\mathbf{c }^c \nonumber \\&+[2\pi R c_1\delta (y)+2\pi R c_2\delta (y-\pi R)]\frac{1}{2}\mathcal {{\mathcal {A}}}^{a,\mu } \delta ^{ab}\nonumber \\&\times (g_{\mu \nu }{\overline{D}}_{\rho }{\overline{D}}^{\rho }-{\overline{D}}_{\mu }{\overline{D}}_{\nu })\mathcal {{\mathcal {A}}}^{b,\nu }, \end{aligned}$$

(42)

where $\mathbf{c }$ ($\overline{\mathbf{c }}$) denotes the ghost (anti-ghost) field. After the KK expansion of the gauge and the ghost fields and diagonalizing 4D gauge kinetic terms, the contribution to Higgs potential can be written down as

$$\begin{aligned} \int \frac{d^4 p}{(2\pi )^4} \frac{1}{2}\log \det \left[ \frac{K^{{\mathcal {A}}_M}}{(K^{{\text { ghost}}})^2}\right] , \end{aligned}$$

(43)

where

$$\begin{aligned} K^{{\mathcal {A}}_5,{\text { ghost}}}_{mn}= & {} \delta _{mn}(p^2+m_n^2), \end{aligned}$$

(44)

$$\begin{aligned} K^{{\mathcal {A}}_{\mu }}_{mn,\mu \nu }= & {} \delta _{mn}g_{\mu \nu }(p^2+m_n^2)\nonumber \\&+(c_1+(-1)^{m+n}c_2)(g_{\mu \nu }p^2-p_{\mu }p_{\nu }). \end{aligned}$$

(45)

Using the following determinant results

$$\begin{aligned}&\det _{(\mu \nu )}\left[ \delta _{nm}g_{\mu \nu }(p^2+m_n^2)+(c_1+(-1)^{m+n}c_2)(g_{\mu \nu }p^2-p_{\mu }p_{\nu })\right] \nonumber \\&\quad = -\delta _{nm}(p^2+m_n^2)\left[ \delta _{nm}(p^2+m_n^2)\right. \nonumber \\&\qquad \left. +(c_1+(-1)^{m+n}c_2)p^2\right] ^3, \nonumber \\&\det _{(nm)}\left[ \delta _{nm}(p^2+m_n^2)+(c_1+(-1)^{m+n}c_2)p^2\right] \end{aligned}$$

(46)

$$\begin{aligned}&\quad = \Pi _n(p^2+m_n^2)\left[ \Pi _{i=1}^2 \left( 1+c_i \sum _n \frac{p^2}{p^2+m_n^2}\right) \right. \nonumber \\&\qquad \left. -\Pi _{i=1}^2 \left( c_i \sum _n \frac{p^2(-1)^n}{p^2+m_n^2}\right) \right] \end{aligned}$$

(47)

where $(\mu \nu )$ denots the determinant over 4D spacetime and (nm) denotes the determinant over the KK mode, Eq. (43) is computed as follows.

$$\begin{aligned} \int \frac{d^4 p}{(2\pi )^4} \frac{1}{2}\log \det \left[ \frac{K^{{\mathcal {A}}_M}}{(K^{gh})^2}\right] ={\mathcal {F}}^+(q\alpha )+{\mathcal {F}}^c(q\alpha ), \end{aligned}$$

(48)

$$\begin{aligned} {\mathcal {F}}^c(q\alpha )= & {} \frac{3}{16\pi ^6 R^4}\int dx \, x^3 \log \left[ \Pi _{i=1}^2 \right. \nonumber \\&\times \left( 1+c_i \sum _n x \text{ Re }f_0^{(+)}(x,q\alpha ) \right) \nonumber \\&\left. -\Pi _{i=1}^2 \left( c_i \sum _n x \text{ Re }f_1(x,q\alpha ) \right) \right] \end{aligned}$$

(49)

with

$$\begin{aligned} f_1(x,q\alpha ) = \sum _{n=-\infty }^{\infty }\frac{(-1)^n}{|x|+i\pi (n+q\alpha )} =\sinh ^{-1}(|x|+i\pi q\alpha ). \end{aligned}$$

(50)

${\mathcal {F}}^+(q\alpha )$ and ${\mathcal {F}}^c(q\alpha )$ are contributions from the bulk gauge kinetic terms and the localized gauge kinetic terms, respectively. It is easy to check ${\mathcal {F}}^c(q\alpha )=0$ at $c_i=0$.

In the anti-periodic sector, a difference from the periodic sector is the following.

$$\begin{aligned} K^{{\mathcal {A}}_{\mu }}_{mn,\mu \nu } = \delta _{mn}g_{\mu \nu }(p^2+m_n^2) +c_1(g_{\mu \nu }p^2-p_{\mu }p_{\nu }). \end{aligned}$$

(51)

Therefore, substituting $c_2=0$ in Eq. (49), we easily obtain contributions from the localized gauge kinetic terms with anti-periodic boundary condition.

$$\begin{aligned}&{\mathcal {F}}_{1/2}^c(q\alpha ) = \frac{3}{16\pi ^6 R^4}\int dx \, x^3\nonumber \\&\quad \log \left[ 1+c_1 \sum _n x \text{ Re }f_0^{(-)}(x,q\alpha )\right] . \end{aligned}$$

(52)

Table 6 lists a Higgs potential from gauge field contributions.

Table 6 Gauge field contributions to Higgs potential

Finally, we need the contributions from the SM fermion localized at $y=0$ to the Higgs potential. The results are obtained from the expression below [53].

$$\begin{aligned}&V_{a} = -\frac{1}{4\pi ^6 R^4} \int dx \, x^3\nonumber \\&\quad \log \left[ Z_L^a Z_R^a +\frac{\pi R}{x}m^a\right] \, (a=u,d,e,\nu ). \end{aligned}$$

(53)

Fig. 7

Total Higgs potential in the case of $c_1+c_2=10$ (left side) and 20 (right side). The horizontal axis is the VEV of Higgs field

Fig. 8

Potential for a set of 15 periodic bulk and mirror fermions with extra bulk mass $\lambda _{\text {{ext}}}=0$ (left side) and potential for a set of 6 periodic bulk and mirror fermions with extra bulk mass $\lambda _{\text {{ext}}}=0$ (right side)

In calculation of the potential from the both bulk and boundary contributions, we have subtracted the $\alpha$ independent part of the potential since it corresponds to the divergent vacuum energy and is irrelevant to the electroweak symmetry breaking.

Total potential is $V(\alpha )=V_{{\text { gauge}}}+V_{{\text { bulk}}}+V_{{\text { boundary}}}$, where $V_{{\text { gauge}}}$, $V_{{\text { bulk}}}$ and $V_{{\text { boundary}}}$ are the contributions from the gauge field, the bulk and mirror fermions and the mixing between the bulk fermions and the SM fermions respectively. In this model, 6 bulk mass $\lambda _6$ is lager than other bulk muss to reprodce small neutrino mass as mentioned in Sect. 4.2, so that 6 bulk and mirror fermion contribution to Higgs potential and mixing contribution $V_{\nu }$ are negligibly small. As a result, our analysis is independent of parameters $\lambda _6$, $\epsilon _6$ and $P_{6}$. The plots of the potential are shown in Fig. 7. It is reasonable to fix a sum $c_1+c_2$ to constant value $c = c_1 + c_2$ since the bulk mass $\lambda$ depends on c. Then, the behavior of potential can be considered by changing a ratio $r=c_1/c=1-c_2/c$ $(0\le r \le 1)$.

From the requirement of large compactification scale in Sect. 3.1, $\alpha$ has to be smaller than 0.1. This implies that the electroweak symmetry is not broken in our model as it stands since the minimum is $\alpha =0.5$. Therefore, we need to extend our model and introduce extra fermions to obtain $0< \alpha < 0.1$ for a successful electroweak symmetry breaking.

In this paper, we introduce three sets of 15 or five sets of 6 periodic bulk and mirror fermions. In Table 5, the contribution of a set of 15 and 6 periodic bulk and mirror fermions to Higgs potential are given and its plot of the potential is shown in Fig. 8. The strategy of introducing such a set of bulk and mirror fermions is as follows. Since the total potential without extra fermions has a minimum at $\alpha =0.5$, the contribution of extra potential with a minimum at $\alpha < 0.1$ is needed. As can be seen from Fig. 8, the contribution from massless 15 fermions and 6 fermions to the potential has a minimum at $\alpha =0$ where the correct electroweak symmetry breaking does not occur. On the other hand, the contribution of massive 15 and 6 is suppressed by the bulk mass $\lambda _{\text {{ext}}}$, then $\alpha$ in the total potential with extra matter contributions becomes small by increasing $\lambda _{\text {{ext}}}$. This opens a possibility that $0< \alpha < 0.1$ required for the correct pattern of the electroweak symmetry breaking can be realized.

Fig. 9

Bulk mass of extra fermion and r dependence for the 4D gauge coupling for introduced three sets of 15 bulk fermion (upper side) and five sets of 6 bulk fermion (lower side). Horizontal and vertical axis are $\log _{10}[r]$ and $\lambda _{{\text {ext}}}$ respectively

Higgs mass is obtained from the second derivative of total potential as

$$\begin{aligned} m_H = \frac{g_{\mathrm{eff}} R \sqrt{1+c}}{2}\sqrt{V''(\alpha )} \sim \text{125 } \text{ GeV }. \end{aligned}$$

(54)

The compactification scale 1/R is defined by Eq. (16)

$$\begin{aligned} \frac{1}{R}=\frac{\pi \sqrt{1+c}\times 80.3 \text{ GeV }}{\sin (\pi \alpha )}. \end{aligned}$$

(55)

The compactification scale can be large by increasing c and decreasing $\alpha$ (or equivalently increasing $\lambda _{\text {{ext}}}$). Figure 9 shows a plot of 4D gauge coupling $g_{\text { eff}}$ for $-1.3< \log _{10}[r] < 1$ and $0.20 \le \lambda _{\text {{ext}}} \le 0.90$ in the case of three sets of 15 fermion and for $-1.0< \log _{10}[r] < 0$ and $0.20 \le \lambda _{\text {{ext}}} \le 0.60$ in the case of five sets of 6 fermion. In this plot, $\alpha$ and the compactification scale are also displayed. We can find an allowed region of parameter space in our model, where the gauge coupling universality is kept ($\alpha < 0.1$), the top quark ($c > 4$), Higgs boson masses and the realistic electroweak symmetry breaking ($0<\alpha <0.5$) are obtained. As representative samples of our solutions, we list $g_{\text { eff}}$ and 1/R are 0.349 and 8 TeV with $\lambda _{\text {{ext}}}=0.71$, $r=1$, $c=7$ and $\epsilon _{20}^i$=(0.1,1,1) in the case of three sets of 15 fermion and $g_{\text { eff}}$ and 1/R are 0.371 and 16.2 TeV with $\lambda _{\text {{ext}}}=0.71$, $r=1$, $c=7$ and $\epsilon _{20}^i$=(0.1,1,1) in the case of three sets of 6 fermion Note that the compactification scale in the present paper becomes larger by the effects of the localized gauge kinetic terms, which is compared to the slightly small compactification scale ($\sim 0.8$ TeV) in our previous paper [53].

We give some comments on the extra bulk fermions which are required for the realistic electroweak symmetry breaking. First, their representations are very simplified. Although the representation is a totally four rank symmetric tensor of SU(6) in our previous paper, the representation is at most two rank symmetric tensor in the present analysis. Second, it is natural to ask whether there are any allowed region of parameters in cases with one or two sets of extra fermions in the 15 representation. We have also tried the potential analysis for those cases, but could not obtain an observed Higgs boson mass. The same is true for 6 representation.

6 Conclusions

In this paper, we have discussed the fermion mass hierarchy in SU(6) GGHU with localized gauge kinetic terms. The SM fermions are introduced in the SU(5) multiplets on the boundary at $y=0$. We also introduced massive bulk fermions in three types of SU(6) representations coupling to the SM fermions on the boundary. Once the localized gauge kinetic terms are present, the zero mode wave functions are distorted and the gauge coupling universality is not guaranteed. We have investigated the constraints where the gauge coupling constant between the SM fermions and a SM gauge field, the cubic and the quartic self-interaction gauge coupling constants are almost universal. It turns out that the gauge coupling universality can be preserved if the dimensionless Higgs VEV is smaller than 0.1 ($\alpha < 0.1$).

We have shown that the SM fermion masses including top quark can be reproduced by mild tuning of bulk masses and the parameters of the localized gauge kinetic terms. As for top quark mass, we have investigated a dependence of the maximum of fermion mass on a parameter c of the localized gauge kinetic terms. $c > 4$ is found for obtaining top quark mass in Fig. 5

We have also calculated additional contributions to one-loop Higgs potentials from the localized gauge kinetic terms. It was found as in our previous paper [53] that the electroweak symmetry breaking does not occur for the fermion matter content mentioned above. To overcome this problem, we have shown that the electroweak symmetry breaking happened by introducing additional three sets of bulk and mirror fermions in $\mathbf{15}$ representation or five sets of bulk and mirror fermions in $\mathbf{6}$ representation. Note that the representation was very simplified comparing to our previous case where it was the $\mathbf{126}$ representation. The effects of localized gauge kinetic terms enhanced the compactification scale, which is compared to the small compactification scale ($\sim 16$ TeV) in our previous paper [53]. This enhancement of the compactification scale also helps Higgs boson mass large. The observed SM Higgs boson mass 125 GeV was indeed obtained in our analysis.

There are issues to be explored in a context of GUT scenario. First one is the gauge coupling unification. It is well known that the gauge coupling running in (flat) extra dimensions is the power dependence on energy scale [57, 58] not the logarithmic one. Therefore, the GUT scale is likely to be very small comparing to the conventional 4D GUT. It is therefore very nontrivial whether the unified SU(6) gauge coupling at the GUT scale is perturbative since many bulk fields are introduced, which might lead to Landau pole below the GUT scale. Second one is proton decay. X, Y gauge boson masses are likely to be extremely light comparing to the conventional GUT scale. Therefore, proton decays very rapidly and our model is immediately excluded by the constraints from the Super Kamiokande data as it stands. Dangerous baryon number violating operators must be forbidden by some symmetry (see [59] for UED case) for the proton stability. If $U(1)_X$ is broken to some discrete symmetry which plays an role for it, it would be very interesting.

These issues are remained for our future work.