Fermion mass hierarchy and mixing in simplified grand gauge–Higgs unification

Gauge–Higgs unification (GHU) [1] is a concept in physics that goes beyond the Standard Model (SM). It solves the hierarchy problem by associating the SM Higgs field with one of the extra spatial components of the higher-dimensional gauge field. In this scenario, the physical observables in the Higgs sector are calculable and predictable regardless of its non-renormalizability. For instance, the quantum corrections to Higgs mass and Higgs potential are known to be finite at one-loop [2] and two-loop [3] levels thanks to the higher-dimensional gauge symmetry.

The hierarchy problem originally arose in the context grand unified theory (GUT), where the discrepancy between the GUT scale and the weak scale are preserved and stable under quantum corrections. Therefore, it is logical to explore the extension of GHU to grand unification. One of the authors discussed grand gauge–Higgs unification (GGHU) [4], ⁵ where five-dimensional (5D) SU(6) GGHU was considered and the SM fermions were embedded into zero modes of SU(6) multiplets in the bulk. This setup was very attractive because of its minimal matter content without massless exotic fermions being absent in the SM, meaning that it was anomaly-free matter content. However, a crucial drawback was found. The down-type Yukawa couplings and the charged lepton Yukawa couplings in GHU originating from the gauge interaction cannot be allowed since the left-handed ${SU}{\left(2\right)}_{L}$ doublets and the right-handed ${SU}{\left(2\right)}_{L}$ singlets are embedded into different SU(6) multiplets. ⁶ Fortunately, an alternative approach to generate Yukawa coupling in the context of GHU was found [7, 8], in which the SM fermions are introduced on the boundaries (i.e. a fixed point in an orbifold compactification). We also introduced massive bulk fermions, which couple to the SM fermions through the mass terms on the boundary. Integrating out these massive bulk fermions leads to non-local SM fermion masses, which are proportional to the bulk-to-boundary couplings and exponentially sensitive to their bulk masses. Then, the SM fermion mass hierarchy can be obtained by very mild tuning of bulk masses.

Along this line, we have improved the SU(6) GGHU model from [4] in [9], where the SM fermion mass hierarchy except for top quark mass is obtained by introducing it on the boundary as SU(5) multiples, with the four types of massive bulk fermions in SU(6) multiplets coupling to the SM fermions. Furthermore, we have shown that electroweak symmetry breaking and an observed Higgs mass can be realized by introducing additional bulk fermions with a large-dimensional representation. In GHU on a flat spacetime, where the top quark is embedded into the bulk field, generating the top quark mass becomes difficult. This is because Yukawa coupling is originally gauge coupling and fermion mass is at most an order of W boson mass in this scenario. As a useful approach [7], introducing the localized gauge kinetic terms on the boundary is known to enhance fermion mass. In our previous paper [10], we followed this approach in order to realize the SM fermion mass hierarchy including the top quark. Once the localized gauge kinetic terms are introduced, the zero-mode wave functions of gauge fields are distorted and the gauge coupling universality is not guaranteed. We found a parameter space where the gauge coupling constant between fermions and a gauge field, the cubic and the quartic self-coupling constants are almost universal. Then, we showed that the fermion mass hierarchy including top quark mass was indeed realized by appropriately choosing the bulk mass parameters and the size of the localized gauge kinetic terms. The correct pattern of electroweak symmetry breaking was obtained by introducing extra bulk fermions, as in our previous paper [10], but their representations have been greatly simplified. However, unfortunately, generation mixing could not be generated in the previous model [10] because each type of bulk fermion was introduced per generation.

The main motivation of this paper is to reproduce the generation mixing as well as fermion masses, which were not realized previously [10]. For this purpose, we will reduce the number of bulk fermions coupling to the SM fermions on the boundaries and modify their coupling. This reduction inevitably introduces inter-generational couplings and makes reproducing the quark and lepton mixing angles highly nontrivial, but it finally enables us to reproduce the flavor-mixing angles and a charge-parity (CP) phase. After the analysis, we find allowed parameter sets to reproduce the SM fermion masses and mixing.

This paper is organized as follows. In the next section, we briefly describe the gauge and Higgs sectors of our model with the localized gauge kinetic terms, and discuss the mass spectrum of gauge fields including their effects. In section 3, we explain how our model has been changed as well as the generation mechanism of the SM fermion masses and mixing. Then, it is shown that the SM fermion masses and mixing can be reproduced by mild tuning of bulk masses and parameters of the localized gauge kinetic terms. The final section is devoted to our conclusions.

In this section, we briefly explain our model [10], which is a 5D SU(6) gauge theory compactified on an orbifold S¹/Z₂ where the radius is denoted as R. Since two fixed points are located at y = 0, π R in the fifth dimension, Z₂ parities must be imposed and given as follows.

$$\begin{eqnarray}\begin{array}{rcl}P & = & {\rm{diag}}(+,+,+,+,+,-)\,{\rm{at}}\,y=0,\\ P^{\prime} & = & {\rm{diag}}(+,+,-,-,-,-)\,{\rm{at}}\,y=\pi R.\end{array}\end{eqnarray} \tag{ 1 }$$

Accordingly, we assign the Z₂ parity to the gauge field and the scalar field as A_μ ( − y) = PA_μ (y)P^†, A_y ( − y) = − PA_y (y)P^†. We note that only the field with parity (+, + ) has a 4D massless zero mode, where (+, + ) means that Z₂ parity is even at the y = 0 (y = π R) boundary. This is because the wave functions of (+, + ) are $\cos ({ny}/R)\,(n=0,\cdots ,\infty )$ after the Kaluza–Klein (KK) expansion. The Z₂ parity of A_μ tells us that SU(6) gauge symmetry is broken into ${SU}{\left(3\right)}_{C}\times {SU}{\left(2\right)}_{L}\times U{\left(1\right)}_{Y}\times U{\left(1\right)}_{X}$ by the combination of the symmetry-breaking pattern at each boundary,

$$\begin{eqnarray}&&{SU}(6)\to {SU}(5)\times U{\left(1\right)}_{X}\hspace{1.6cm}{\rm{at}}\,y=0,\end{eqnarray} \tag{ 2 }$$

$$\begin{eqnarray}&&{SU}(6)\to {SU}(2)\times {SU}(4)\times U(1)^{\prime} \,{\rm{at}}\,y=\pi R.\end{eqnarray} \tag{ 3 }$$

The hypercharge $U{\left(1\right)}_{Y}$ is contained in the Georgi–Glashow SU(5) GUT, which implies that the weak mixing angle is ${\sin }^{2}{\theta }_{{\rm{W}}}=3/8$ (θ_W: weak mixing angle) at the unification scale.

The problem of the remaining extra $U{\left(1\right)}_{X}$ is easily resolved by introducing a 4D $U{\left(1\right)}_{X}$ charged scalar field localized on a fixed point and constructing the quadratic and quartic scalar potential with negative mass squared. The scalar field will develop a vacuum expectation value (VEV) and the $U{\left(1\right)}_{X}$ gauge field is massive.

The SM ${SU}{\left(2\right)}_{L}$ Higgs doublet field is identified with part of an extra component of gauge field A_y . The VEV of the Higgs field is assumed to be taken in the 28th SU(6) generator as $\langle {A}_{y}^{a}\rangle =\tfrac{2\alpha }{{Rg}}{\delta }^{a\,28}$, where g is a 5D SU(6) gauge coupling constant and α is a dimensionless constant. The VEV of the Higgs field is expressed by $\langle H\rangle =\tfrac{\sqrt{2}\alpha }{{Rg}}$. In this setup, the doublet-triplet splitting problem is solved by the orbifolding since the colored Higgs has a Z₂ parity (+, − ) and correspondingly becomes massive [11].

The Higgs couplings to the gauge bosons and the fermions are included in the gauge interactions of their kinetic terms,

$$\begin{eqnarray}&&-\displaystyle \frac{1}{4}{{ \mathcal F }}_{{MN}}^{a}{{ \mathcal F }}^{a\,{MN}}\supset -\displaystyle \frac{1}{2}{{ \mathcal F }}_{\mu y}^{a}{{ \mathcal F }}^{a\,\mu y}\supset -\displaystyle \frac{1}{2}{A}_{\mu }^{a}({\partial }_{y}+{f}^{{adb}}{A}_{y}^{d})({\partial }_{y}+{f}^{{bec}}{A}_{y}^{e}){A}^{\mu \,c},\end{eqnarray} \tag{ 4 }$$

$$\begin{eqnarray}&&\overline{{\rm{\Psi }}}i\rlap{/}{D}{\rm{\Psi }}\supset \overline{{\rm{\Psi }}}{{iD}}_{y}{{\rm{\Gamma }}}^{y}{\rm{\Psi }}=-\overline{{\rm{\Psi }}}({\partial }_{y}+{A}_{y}){\gamma }^{5}{\rm{\Psi }},\end{eqnarray} \tag{ 5 }$$

where M, N = {μ, y}, μ = 0, 1, 2, 3, y = 5 and subscripts a, b, c, d and e denote the gauge indices for SU(6). Γ^y in (5) is the fifth component of the 5D gamma matrices Γ^M = (Γ^μ , Γ^y ) = (γ^μ , i γ⁵). Equations (4) and (5) become the mass terms after the Higgs field takes the VEV. The mass eigenvalues are obtained as ${m}_{n}(q\alpha )=\tfrac{n+\nu +q\alpha }{R}$, where n is KK mode. ν = 0 or 1/2 depending on the boundary condition. ν is 0 for a periodic sector where A(y + π R) = A(y) or parity of fields are $(P,{P}^{{\prime} })=(+,+),(-,-)$, or 1/2 for an anti-periodic sector where A(y + π R) = − A(y) or parity of fields are $(P,{P}^{{\prime} })=(+,-),(-,+)$. q is an integer charge fixed by the SU(2) representation to which the field coupled to a Higgs field belongs. If the field belongs to an N + 1 representation of ${SU}{\left(2\right)}_{L}$, the q is equal to N. For instance, 6 representation of SU(6) has the branching rule under ${SU}(6)\to {SU}{\left(3\right)}_{C}\times {SU}{\left(2\right)}_{L}$ like 6 → (3, 1) ⊕ (1, 2) ⊕ (1, 1), which implies that this representation has four states with q = 0 and one state with q = 1.

In order to reproduce the top quark mass, we introduce additional gauge kinetic terms localized at y = 0 and y = π R. The Lagrangian of the SU(6) gauge kinetic term is

$$\begin{eqnarray}&&{{ \mathcal L }}_{g}=-\displaystyle \frac{1}{4}{{ \mathcal F }}^{a\,{MN}}{{ \mathcal F }}_{{MN}}^{a}-2\pi {{Rc}}_{1}\delta (y)\displaystyle \frac{1}{4}{{ \mathcal F }}^{b\,\mu \nu }{{ \mathcal F }}_{\mu \nu }^{b}-2\pi {{Rc}}_{2}\delta (y-\pi R)\displaystyle \frac{1}{4}{{ \mathcal F }}^{c\,\mu \nu }{{ \mathcal F }}_{\mu \nu }^{c},\end{eqnarray} \tag{ 6 }$$

where the first term is the 5D bulk gauge kinetic term and the second and third terms are gauge kinetic terms localized at fixed points. c_1,2 are dimensionless free parameters. The superscripts a, b and c denote the gauge indices for SU(6), SU(5) × U(1) and ${SU}(2)\times {SU}(4)\times U(1)^{\prime}$.

The mass spectrum of the SM gauge field becomes very complicated due to these localized gauge kinetic terms. In particular, their effects on the periodic sector and the anti-periodic sector are different. In a basis where the 4D gauge kinetic terms are diagonal, the boundary conditions for wave functions are found as f_n (y + π R; q α) = e^{2iπ q α} f_n (y; q α) in the periodic sector and f_n (y + π R; q α) = e^{2iπ(q α+1/2)} f_n (y; q α) in the anti-periodic sector. The wave functions in the same basis satisfy the following equation

$$\begin{eqnarray}&&\left[{\partial }_{y}^{2}+{m}_{n}^{2}(q\alpha )\left(1+2\pi {{Rc}}_{1}\delta (y)+2\pi {{Rc}}_{2}\delta (y-\pi R)\right)\right]{f}_{n}(y;q\alpha )=0,\end{eqnarray} \tag{ 7 }$$

where m_n (q α) is the KK mass. By solving equation (7) with the (anti-)periodic boundary conditions, the equations determining the KK mass spectrum are obtained [12].

$$\begin{eqnarray}&&2(1-{c}_{1}{c}_{2}{\xi }_{n}^{2}){\sin }^{2}{\xi }_{n}+({c}_{1}+{c}_{2}){\xi }_{n}\sin 2{\xi }_{n}-2{\sin }^{2}(\pi (q\alpha +\nu ))=0\end{eqnarray} \tag{ 8 }$$

where ξ_n = π Rm_n .

Taking into account that m₀ is around weak scale (∼100 GeV) and 1/R is larger than 1 TeV, it is reasonable to suppose ξ₀ ≪ 1. Then, we can find an approximate form of ξ₀ as

$$\begin{eqnarray}&&{\xi }_{0}\sim \displaystyle \frac{\sin (\pi (q\alpha +\nu ))}{\sqrt{1+{c}_{1}+{c}_{2}}}.\end{eqnarray} \tag{ 9 }$$

For instance, the W boson mass m_W is given by

$$\begin{eqnarray}&&{m}_{W}=\frac{\sin (\pi \alpha )}{\pi R\sqrt{1+{c}_{1}+{c}_{2}}}\end{eqnarray} \tag{ 10 }$$

since the W boson is the gauge boson with q = 1 and ν = 0.

In the previous paper [10], the SM fermions were embedded into SU(5) multiplets localized at the y = 0 boundary, where three sets of decouplet, anti-quintet and singlet ${\chi }_{10},{\chi }_{{5}^{* }},{\chi }_{1}$ are introduced. We also introduced three types of bulk fermions Ψ and $\tilde{{\rm{\Psi }}}$ (referred as 'mirror fermions') with opposite Z₂ parities to each other per generation and a constant mass term of $M\bar{{\rm{\Psi }}}\tilde{{\rm{\Psi }}}$ in the bulk to avoid exotic 4D massless fermions. Without these mirror fermions and mass terms, we necessarily have extra exotic 4D massless fermions with SM charges after an orbifold compactification. As a result, we have no massless chiral fermions from the bulk and its mirror fermions. The massless fermions are only the SM fermions and the gauge anomalies for the SM gauge groups are trivially canceled.

In this setup, we could not generate the generation mixing of quarks and leptons [10]. In this section, we will show how the model has been changed, discuss the mechanism generating the SM mass and the generation mixing of weak interaction, and demonstrate the setup of our model and the results.

3.1. Boundary fermion mass

The boundary fermions χ^i=1,2,3 localized on the boundaries ${r}_{1}^{i},{r}_{2}^{i}$ are equal to either 0 or π R and have kinetic mixing terms between the bulk and mirror fermions. These interaction terms are

$$\begin{eqnarray}\begin{array}{ccl}{{ \mathcal L }}_{4} & = & \int {\rm{d}}y\displaystyle \sum _{i}\sqrt{\frac{2}{\pi R}}\left\{\delta (y-{r}_{1}^{i}){\epsilon }_{L}^{i}\bar{{\psi }_{L}^{i}}(x)A(x,y)+\delta (y-{r}_{2}^{i}){\epsilon }_{R}^{i}\bar{{\psi }_{R}^{i}}(x)B(x,y)+\mathrm{h.c.}\right\}\\ & = & \displaystyle \sum _{i}\displaystyle \sum _{n}\frac{1}{\pi R}\left\{{\xi }_{{r}_{1}^{i}}^{a}{\epsilon }_{L}^{i}\bar{{\psi }_{L}^{i}}(x){a}_{n}(x)+{\xi }_{{r}_{2}^{i}}^{b}{\epsilon }_{R}^{i}\bar{{\psi }_{R}^{i}}(x){b}_{n}(x)+\mathrm{h.c.}\right\},\end{array}\end{eqnarray} \tag{ 11 }$$

where A and B are the bulk and mirror fermions. a_n and b_n are the corresponding 4D fields:

$$\begin{eqnarray}&&A(x,y)=\displaystyle \sum _{n}{f}_{n}(y){a}_{n}(x),\qquad B(x,y)=\displaystyle \sum _{n}{f}_{n}(y){b}_{n}(x).\end{eqnarray} \tag{ 12 }$$

${\xi }_{{r}_{n}^{i}}=\sqrt{\pi R}{f}_{n}({r}_{n}^{i})$ is a factor from mode functions f_n (y) and takes 1 if ${r}_{1,2}^{i}=0$ or ${\left(-1\right)}^{n}$ if ${r}_{1,2}^{i}=\pi R$.

The generation mixing of the boundary fermion is generated by integrating out the bulk or mirror fermions, which can be seen from the diagram shown in figure 1,

$$\begin{eqnarray}&&\displaystyle \sum _{i,j}\left[-i\overline{{\psi }_{L}^{i}}{\rlap{/}{p}}_{E}{Z}_{L}^{{ij}}{\psi }_{L}^{j}-i\overline{{\psi }_{R}^{i}}{\rlap{/}{p}}_{E}{Z}_{R}^{{ij}}{\psi }_{R}^{j}+\left(\overline{{\psi }_{L}^{i}}{M}^{{ij}}{\psi }_{R}^{j}+{\rm{h.c.}}\right)\right],\end{eqnarray} \tag{ 13 }$$

where the subscript E means Euclidean, namely Wick rotation is performed. Furthermore, all terms can be expressed in terms of the bulk and mirror fermion propagator ${\left\langle ...\right\rangle }_{E}$

$$\begin{eqnarray}\begin{array}{rcl}{\rlap{/}{p}}_{E}{Z}_{L}^{{ij}}{P}_{L} & = & \displaystyle \frac{-i}{{\pi }^{2}{R}^{2}}\displaystyle \sum _{n}{\xi }_{{r}_{1}^{i}}^{a}{\xi }_{{r}_{2}^{j}}^{b}{\epsilon }_{L}^{i}{\epsilon }_{L}^{j* }{P}_{R}{\left\langle {a}_{n}(x){\overline{a}}_{n}(x)\right\rangle }_{E}{P}_{L},\\ {\rlap{/}{p}}_{E}{Z}_{R}^{{ij}}{P}_{R} & = & \displaystyle \frac{-i}{{\pi }^{2}{R}^{2}}\displaystyle \sum _{n}{\xi }_{{r}_{1}^{i}}^{a}{\xi }_{{r}_{2}^{j}}^{b}{\epsilon }_{R}^{i}{\epsilon }_{R}^{j* }{P}_{L}{\left\langle {b}_{n}(x){\overline{b}}_{n}(x)\right\rangle }_{E}{P}_{R},\\ {M}^{{ij}}{P}_{R} & = & \displaystyle \frac{1}{{\pi }^{2}{R}^{2}}\displaystyle \sum _{n}{\xi }_{{r}_{1}^{i}}^{a}{\xi }_{{r}_{2}^{j}}^{b}{\epsilon }_{L}^{i}{\epsilon }_{R}^{j* }{P}_{R}{\left\langle {a}_{n}(x){\overline{b}}_{n}(x)\right\rangle }_{E}{P}_{R}.\end{array}\end{eqnarray} \tag{ 14 }$$

We notice that these mixing effects are new bulk contributions, which are absent in the previous paper [10] where the same set of bulk and mirror fermions are introduced per generation. By construction, we have no mixing from the bulk sector. In this paper, however, generation mixings are inevitable due to the reduction of the number of bulk and mirror fermions, which makes our analysis very complicated and also obtaining realistic mixing parameters nontrivial.

Zoom In Zoom Out Reset image size

We can evaluate these terms (14) by computing the bulk and mirror fermion propagator ${\left\langle ...\right\rangle }_{E}$ and the summation of the KK modes. The methods are summarized in appendices A and B. The results are classified into three cases depending on the cases where a and b are the bulk fermion or the mirror fermion, and can be rewritten by the following functions,

$$\begin{eqnarray}{f}_{\delta }^{(T)}(x,q\alpha )=\left\{\begin{array}{cc}\coth (x+{\rm{i}}\pi \alpha ) & :\delta =0,T=+1,(\nu =0)\\ \sinh {\left(x+{\rm{i}}\pi \alpha \right)}^{-1} & :\delta =1,T=+1,(\nu =0)\\ \tanh (x+{\rm{i}}\pi \alpha ) & :\delta =0,T=-1,(\nu =1/2)\\ \cosh {\left(x+{\rm{i}}\pi \alpha \right)}^{-1} & :\delta =1,T=-1,(\nu =1/2)\end{array}\right.,\end{eqnarray} \tag{ 15 }$$

where x = π Rp_E and $\delta ={\left(\pi R\right)}^{-1}| {r}_{1}^{i}-{r}_{2}^{j}|$ describes the distance between r₁ ⁱ and r₂ ^j . δ takes 0 if ${r}_{1}^{i}={r}_{2}^{j}$) and 1 if ${r}_{1}^{i}\ne {r}_{2}^{j}$. T denotes the periodicity of the fields. T = 1 corresponds to $(P,P^{\prime} )=(\pm ,\pm )$ and T = − 1 corresponds to $(P,P^{\prime} )=(\pm ,\mp )$ in tables 2, 3 and 4.

Table 1. Representation of bulk fermions and the corresponding mirror fermions. P_i is the parity of bulk fermion for i representation in SU(6). R in R^(+,+) means an SU(6) representation of the bulk fermion. r_i in r₁ ⊕ r₂ are SU(5) representations.

Bulk fermion SU(6) → SU(5)	Mirror fermion
20(+,+)=10 ⊕ 10*	20(−,−)
15(+,+)=10 ⊕ 5	15(−,−)
$15{{\prime} }^{(+,-)}=10^{\prime} \oplus 5^{\prime}$	$15{{\prime} }^{(-,+)}$
6(−,−)=5 ⊕ 1	6(+,+)
$6{{\prime} }^{(+,+)}=5^{\prime} \oplus 1^{\prime}$	$6{{\prime} }^{(-,-)}$

Table 2.  20 bulk and SM fermions. r_1,2 in ${\left({r}_{1},\,{r}_{2}\right)}_{a}$ are SU(3) and SU(2) representations in the SM, respectively. a is $U{\left(1\right)}_{Y}$ charges.

Bulk fermion ${SU}(5)\to {SU}{\left(3\right)}_{C}\times {SU}{\left(2\right)}_{L}\times U{\left(1\right)}_{Y}$	SM fermion coupling to bulk
$10={Q}_{20}{\left(3,2\right)}_{1/6}^{(+,+)}\oplus {U}_{20}^{* }{\left({3}^{* },1\right)}_{-2/3}^{(+,-)}\oplus {E}_{20}^{* }{\left(1,1\right)}_{1}^{(+,-)}$	${q}_{L}{\left(\mathrm{3,2}\right)}_{1/6},\,{u}_{R}^{c}{\left({3}^{* },1\right)}_{-2/3},\,{e}_{R}^{c}{\left(\mathrm{1,1}\right)}_{1}$
${10}^{* }={Q}_{20}^{* }{\left({3}^{* },2\right)}_{-1/6}^{(-,-)}\oplus {U}_{20}{\left(3,1\right)}_{2/3}^{(-,+)}\oplus {E}_{20}{\left(1,1\right)}_{-1}^{(-,+)}$	${q}_{L}^{c}{\left({3}^{* },2\right)}_{-1/6},\,{u}_{R}{\left(\mathrm{3,1}\right)}_{2/3},\,{e}_{R}{\left(\mathrm{1,1}\right)}_{-1}$

Table 3. Upper (Lower) table shows 15 (${\bf{15}}^{\prime}$) bulk and SM fermions. r_1,2 in ${\left({r}_{1},\,{r}_{2}\right)}_{a}$ are SU(3) and SU(2) representations in the SM, respectively. a is $U{\left(1\right)}_{Y}$ charges.

Bulk fermion ${SU}(5)\to {SU}{\left(3\right)}_{C}\times {SU}{\left(2\right)}_{L}\times U{\left(1\right)}_{Y}$	SM fermion coupling to bulk
$10={Q}_{15}{\left(3,2\right)}_{1/6}^{(+,-)}\oplus {U}_{15}^{* }{\left({3}^{* },1\right)}_{-2/3}^{(+,+)}\oplus {E}_{15}^{* }{\left(1,1\right)}_{1}^{(+,+)}$	${q}_{L}{\left(\mathrm{3,2}\right)}_{1/6},\,{u}_{R}^{c}{\left({3}^{* },1\right)}_{-2/3},\,{e}_{R}^{c}{\left(\mathrm{1,1}\right)}_{1}$
$5={D}_{15}{\left(3,1\right)}_{-1/3}^{(-,+)}\oplus {L}_{15}^{* }{\left(1,2\right)}_{1/2}^{(-,-)}$	${d}_{R}{\left(\mathrm{3,1}\right)}_{-1/3},\,{l}_{L}^{c}{\left(\mathrm{1,2}\right)}_{1/2}$
Bulk fermion ${SU}(5)\to {SU}{\left(3\right)}_{C}\times {SU}{\left(2\right)}_{L}\times U{\left(1\right)}_{Y}$	SM fermion coupling to bulk
$10^{\prime} ={Q}_{15^{\prime} }{\left(3,2\right)}_{1/6}^{(+,+)}\oplus {U}_{15^{\prime} }^{* }{\left({3}^{* },1\right)}_{-2/3}^{(+,-)}\oplus {E}_{15^{\prime} }^{* }{\left(1,1\right)}_{1}^{(+,-)}$	${q}_{L}{\left(\mathrm{3,2}\right)}_{1/6},\,{u}_{R}^{c}{\left({3}^{* },1\right)}_{-2/3},\,{e}_{R}^{c}{\left(\mathrm{1,1}\right)}_{1}$
$5^{\prime} ={D}_{15^{\prime} }{\left(3,1\right)}_{-1/3}^{(-,-)}\oplus {L}_{15^{\prime} }^{* }{\left(1,2\right)}_{1/2}^{(-,+)}$	${d}_{R}{\left(\mathrm{3,1}\right)}_{-1/3},\,{l}_{L}^{c}{\left(\mathrm{1,2}\right)}_{1/2}$

Table 4. Upper (Lower) table shows 6 (${\bf{6}}^{\prime}$) bulk and SM fermions. r_1,2 in ${\left({r}_{1},\,{r}_{2}\right)}_{a}$ are SU(3) and SU(2) representations in the SM, respectively. a is $U{\left(1\right)}_{Y}$ charges.

Bulk fermion ${SU}(5)\to {SU}{\left(3\right)}_{C}\times {SU}{\left(2\right)}_{L}\times U{\left(1\right)}_{Y}$	SM fermion coupling to bulk
$5={D}_{6}{\left(3,1\right)}_{-1/3}^{(-,+)}\oplus {L}_{6}^{* }{\left(1,2\right)}_{1/2}^{(-,-)}$	${d}_{R}{\left(\mathrm{3,1}\right)}_{-1/3},\,{l}_{L}^{c}{\left(\mathrm{1,2}\right)}_{1/2}$
$1={N}_{6}^{* }{\left(1,1\right)}_{0}^{(+,+)}$	${\nu }_{R}^{c}{\left(\mathrm{1,1}\right)}_{0}$
Bulk fermion ${SU}(5)\to {SU}{\left(3\right)}_{C}\times {SU}{\left(2\right)}_{L}\times U{\left(1\right)}_{Y}$	SM fermion coupling to bulk
$5^{\prime} ={D}_{6^{\prime} }{\left(3,1\right)}_{-1/3}^{(+,-)}\oplus {L}_{6^{\prime} }^{* }{\left(1,2\right)}_{1/2}^{(+,+)}$	${d}_{R}{\left(\mathrm{3,1}\right)}_{-1/3},\,{l}_{L}^{c}{\left(\mathrm{1,2}\right)}_{1/2}$
$1^{\prime} ={N}_{6^{\prime} }^{* }{\left(1,1\right)}_{0}^{(-,-)}$	${\nu }_{R}^{c}{\left(\mathrm{1,1}\right)}_{0}$

The first case is that both are bulk fermions:

$$\begin{eqnarray}\begin{array}{rcl}{Z}_{L}^{{ij}} & = & \displaystyle \frac{{\epsilon }_{L}^{i}{\epsilon }_{L}^{j* }}{\sqrt{{x}^{2}+{\lambda }^{2}}}{\rm{Re}}{f}_{\delta }^{(T)}\left(\sqrt{{x}^{2}+{\lambda }^{2}},q\alpha \right),\\ {Z}_{R}^{{ij}} & = & \displaystyle \frac{{\epsilon }_{R}^{i}{\epsilon }_{R}^{j* }}{\sqrt{{x}^{2}+{\lambda }^{2}}}{\rm{Re}}{f}_{\delta }^{(T)}\left(\sqrt{{x}^{2}+{\lambda }^{2}},q\alpha \right),\\ {M}^{{ij}} & = & \displaystyle \frac{{\epsilon }_{L}^{i}{\epsilon }_{R}^{j* }}{\pi R}{\rm{Im}}{f}_{\delta }^{(T)}\left(\sqrt{{x}^{2}+{\lambda }^{2}},q\alpha \right),\end{array}\end{eqnarray} \tag{ 16 }$$

where λ = π RM.

The second case is that they are bulk and mirror fermions, respectively:

$$\begin{eqnarray}\begin{array}{rcl}{Z}_{L}^{{ij}} & = & 0,\\ {Z}_{R}^{{ij}} & = & 0,\\ {M}^{{ij}} & = & -\displaystyle \frac{{\epsilon }_{L}^{i}{\epsilon }_{R}^{j* }\lambda }{\pi R\sqrt{{x}^{2}+{\lambda }^{2}}}{\rm{Re}}{f}_{\delta }^{(T)}\left(\sqrt{{x}^{2}+{\lambda }^{2}},q\alpha \right).\end{array}\end{eqnarray} \tag{ 17 }$$

The third case is that both are mirror fermions:

$$\begin{eqnarray}\begin{array}{rcl}{Z}_{L}^{{ij}} & = & \displaystyle \frac{{\epsilon }_{L}^{i}{\epsilon }_{L}^{j* }}{\sqrt{{x}^{2}+{\lambda }^{2}}}{\rm{Re}}{f}_{\delta }^{(T)}\left(\sqrt{{x}^{2}+{\lambda }^{2}},q\alpha \right),\\ {Z}_{R}^{{ij}} & = & \displaystyle \frac{{\epsilon }_{R}^{i}{\epsilon }_{R}^{j* }}{\sqrt{{x}^{2}+{\lambda }^{2}}}{\rm{Re}}{f}_{\delta }^{(T)}\left(\sqrt{{x}^{2}+{\lambda }^{2}},q\alpha \right),\\ {M}^{{ij}} & = & -\displaystyle \frac{{\epsilon }_{L}^{i}{\epsilon }_{R}^{j* }}{\pi R}{\rm{Im}}{f}_{\delta }^{(T)}\left(\sqrt{{x}^{2}+{\lambda }^{2}},q\alpha \right).\end{array}\end{eqnarray} \tag{ 18 }$$

If some bulk and mirror fermions are introduced, all of the contributions to the kinetic and mass mixing must be summed in equation (14),

$$\begin{eqnarray}&&{\tilde{Z}}_{L}^{{ij}}\equiv {\delta }^{{ij}}+\displaystyle \sum _{a}{Z}_{L}^{a,{ij}},\end{eqnarray} \tag{ 19 }$$

$$\begin{eqnarray}&&{\tilde{Z}}_{{ij}}^{R}\equiv {\delta }^{{ij}}+\displaystyle \sum _{a}{Z}_{R}^{a,{ij}},\end{eqnarray} \tag{ 20 }$$

$$\begin{eqnarray}&&{\tilde{M}}^{{ij}}\equiv \displaystyle \sum _{a}{M}^{a,{ij}}.\end{eqnarray} \tag{ 21 }$$

In the expression above, the superscript 'a' in ${Z}_{L(R)}^{a,{ij}}$ and M^a,ij denotes the SU(6) representations and ${Z}_{L(R)}^{a,{ij}}$ and M^a,ij show the contributions from the corresponding bulk and mirror fermions.

These kinetic mixing can be diagonalized by unitary matrices ${U}_{{Z}_{L,R}}$

$$\begin{eqnarray}&&{\tilde{Z}}_{L}={U}_{{Z}_{L}}^{\dagger }{Z}_{L}^{\mathrm{diag}}{U}_{{Z}_{L}},\end{eqnarray} \tag{ 22 }$$

$$\begin{eqnarray}&&{\tilde{Z}}_{R}={U}_{{Z}_{R}}^{\dagger }{Z}_{R}^{\mathrm{diag}}{U}_{{Z}_{R}},\end{eqnarray} \tag{ 23 }$$

where ${Z}_{L,R}^{\mathrm{diag}}$ are diagonal matrices. After rewriting the mass term in terms of the new basis diagonalizing the kinetic mixings and normalizing the kinetic term, we obtain the following mass matrix

$$\begin{eqnarray}&&M^{\prime} =\sqrt{{\left({Z}_{L}^{\mathrm{diag}}\right)}^{-1}}{U}_{{Z}_{L}}\tilde{M}{U}_{{Z}_{R}}^{\dagger }\sqrt{{\left({Z}_{R}^{\mathrm{diag}}\right)}^{-1}}.\end{eqnarray} \tag{ 24 }$$

Next, we perform unitary transformations in order to move on to the mass basis:

$$\begin{eqnarray}&&\psi {{\prime} }_{R}={U}_{R}{\psi }_{R},\hspace{30pt}\psi {{\prime} }_{L}={U}_{L}{\psi }_{L}.\end{eqnarray} \tag{ 25 }$$

In this new basis, the mass matrix is diagonalized:

$$\begin{eqnarray}&&M^{\prime} ={U}_{L}^{\dagger }{M}_{\mathrm{diag}}{U}_{R}.\end{eqnarray} \tag{ 26 }$$

This expression allows us to compute the SM fermion masses.

3.2. Weak interaction for the boundary fermion

As in the previous section, taking into account the mixing effects between the bulk and mirror fermions and the boundary fermions, additional mixing in the charged weak interaction of the boundary fermion, which is not present in the SM, also seems to be generated. The corresponding interactions generating the additional mixings in the charged weak interaction are shown below,

$$\begin{eqnarray}&&\begin{array}{l}{{ \mathcal L }}_{4}\supset \displaystyle \sum _{i}\left[{W}_{\mu }^{+}\overline{{u}_{L}^{i}}{\gamma }^{\mu }{d}_{L}^{i}+\displaystyle \sum _{n}\displaystyle \frac{1}{\pi R}{\epsilon }_{L}^{i}\left({\xi }_{{r}_{1}^{i}}^{u}\overline{{u}_{L}^{i}}{a}_{n}^{u}+{\xi }_{{r}_{2}^{i}}^{d}\overline{{d}_{L}^{i}}{a}_{n}^{d}\right)\right]\\ \quad +\displaystyle \sum _{n}{W}_{\mu }^{+}\overline{{\chi }_{n}^{u}}{\gamma }^{\mu }{\chi }_{n}^{d}+\displaystyle \sum _{n}{W}_{\mu }^{+}\overline{{\tilde{\chi }}_{n}^{u}}{\gamma }^{\mu }{\tilde{\chi }}_{n}^{d}+{\rm{h.c.}},\end{array}\end{eqnarray} \tag{ 27 }$$

where W is the weak boson, u is up-type quark or electron and d is a down-type quark or neutrino. χ and $\tilde{\chi }$ are the bulk and mirror fermions, respectively. Note that a_n was defined as the corresponding 4D fields of the bulk or mirror fermions and we have not specified whether a_n represents bulk or mirror fermions here. We will consider three cases as in the previous subsection and specify what a_n is.

The additional generation mixing in the charged weak interaction is generated by integrating out the bulk and mirror fermions and can be seen from the diagram shown in figure 2,

$$\begin{eqnarray}&&\displaystyle \sum _{{ij}}{W}_{\mu }^{+}\overline{{u}_{L}^{i}}{\gamma }^{\mu }{M}_{{wi}}^{{ij}}{d}_{L}^{j}+{\rm{h.c.}},\end{eqnarray} \tag{ 28 }$$

where

$$\begin{eqnarray}&&{\gamma }^{\mu }{M}_{{wi}}^{{ij}}{P}_{L}={P}_{R}\displaystyle \sum _{n}\displaystyle \frac{{\xi }_{{r}_{1}^{i}}^{u}{\xi }_{{r}_{2}^{j}}^{d}{\epsilon }_{L}^{i}{\epsilon }_{L}^{j* }}{{\left(\pi R\right)}^{2}}\left\{\left\langle {a}_{n}^{u}\overline{{\chi }_{n}^{u}}\right\rangle {\gamma }^{\mu }\left\langle \overline{{a}_{n}^{d}}{\chi }_{n}^{d}\right\rangle +\left\langle {a}_{n}^{u}\overline{{\tilde{\chi }}_{n}^{u}}\right\rangle {\gamma }^{\mu }\left\langle \overline{{a}_{n}^{d}}{\tilde{\chi }}_{n}^{d}\right\rangle \right\}{P}_{L}.\end{eqnarray} \tag{ 29 }$$

Similarly, this contribution can be evaluated by using the method in appendices A and B. Here, we consider three cases where we specify whether a_n is the bulk fermion χ_n or the mirror fermion $\tilde{{\chi }_{n}}$. As a result, we find the additional mixings in the charged weak interaction to be the same as the kinetic mixing for the left-handed fermions localized on the boundary,

$$\begin{eqnarray}&&{M}_{{wi}}^{{ij}}={Z}_{L}^{{ij}}.\end{eqnarray} \tag{ 30 }$$

This property ensures that no additional mixing is needed, as will be shown below. In the case where some bulk and mirror fermions are introduced, the kinetic and mass mixing are the summation of all contributions in equation (29),

$$\begin{eqnarray}&&{\tilde{M}}_{{wi}}^{{ij}}={\delta }^{{ij}}+\displaystyle \sum _{a}{M}_{{wi},a}^{{ij}}={\tilde{Z}}_{L}^{{ij}}.\end{eqnarray} \tag{ 31 }$$

After rewriting the charged weak interaction in terms of the new basis in equation (25), the Cabibbo–Kobayashi–Maskawa (CKM) and Pontecorvo–Maki–Nakagawa–Sakat (PMNS) matrices are given by

$$\begin{eqnarray}\begin{array}{rcl}{V}_{\mathrm{CKM}} & = & {U}_{L}^{u}\sqrt{{\left({Z}_{L}^{\mathrm{diag},u}\right)}^{-1}}{U}_{{Z}_{L}}^{u}{\tilde{M}}_{{wi}}^{{ud}}{U}_{{Z}_{L}}^{d\dagger }\sqrt{{\left({Z}_{L}^{\mathrm{diag},d}\right)}^{-1}}{U}_{L}^{d\dagger }={U}_{L}^{u}{U}_{L}^{d\dagger },\\ {V}_{\mathrm{PMNS}} & = & {U}_{L}^{e}\sqrt{{\left({Z}_{L}^{\mathrm{diag},e}\right)}^{-1}}{U}_{{Z}_{L}}^{e}{\tilde{M}}_{{wi}}^{e\nu }{U}_{{Z}_{L}}^{\nu \dagger }\sqrt{{\left({Z}_{L}^{\mathrm{diag},\nu }\right)}^{-1}}{U}_{L}^{\nu \dagger }={U}_{L}^{e}{U}_{L}^{\nu \dagger },\end{array}\end{eqnarray} \tag{ 32 }$$

where u, d, e and ν denote the up-type quarks, down-type quarks, charged leptons and neutrinos of the SM fermion, respectively. We utilized the fact that the contributions of the left-handed SU(2) doublets to the kinetic mixing are the same, for instance, ${\tilde{Z}}_{L}^{u}={\tilde{Z}}_{L}^{d}$ and ${\tilde{Z}}_{L}^{e}={\tilde{Z}}_{L}^{\nu }$. These expressions allow us to calculate the weak mixing angles and a CP phase.

Zoom In Zoom Out Reset image size

3.3. Fermion sector of our model and results

In the setup of our model, we introduce five bulk fermions ${{\rm{\Psi }}}_{20},{{\rm{\Psi }}}_{15},{{\rm{\Psi }}}_{15^{\prime} },{{\rm{\Psi }}}_{6}\mathrm{and}{{\rm{\Psi }}}_{6^{\prime} }$ and the corresponding mirror fermions ${\tilde{{\rm{\Psi }}}}_{20},{\tilde{{\rm{\Psi }}}}_{15},{\tilde{{\rm{\Psi }}}}_{15^{\prime} },{\tilde{{\rm{\Psi }}}}_{6}\mathrm{and}{\tilde{{\rm{\Psi }}}}_{6^{\prime} }$ shown in table 1. Note that the number of bulk fermions has been reduced from nine in [10] to five, which necessarily introduces generational mixings through the coupling of bulk fermions to boundary fermions. The Lagrangian for the bulk and mirror fermions is given by

$$\begin{eqnarray}&&{{ \mathcal L }}_{\mathrm{bulk}+\mathrm{mirror}}=\displaystyle \sum _{a=20,15,15^{\prime} ,6,6^{\prime} }\left[{\bar{{\rm{\Psi }}}}_{a}{\mathrm{i\Gamma }}^{M}{D}_{M}{{\rm{\Psi }}}_{a}+{\bar{\tilde{{\rm{\Psi }}}}}_{a}{\mathrm{i\Gamma }}^{M}{D}_{M}{\tilde{{\rm{\Psi }}}}_{a}+\left(\frac{{\lambda }_{a}}{\pi R}{\bar{{\rm{\Psi }}}}_{a}{\tilde{{\rm{\Psi }}}}_{a}+\mathrm{h.c.}\right)\right],\end{eqnarray} \tag{ 33 }$$

where the subscript a denotes the SU(6) representations of the bulk and mirror fermions. The bulk masses between the bulk and the mirror fermions are normalized by π R and expressed by the dimensionless parameter λ_a .

The SM quarks and leptons for the first and the second generation are embedded into SU(5) multiplets localized at the y = 0 boundary, which are two sets of decouplets, anti-quintets and singlets ${\chi }_{10},{\chi }_{{5}^{* }},{\chi }_{1}$. On the other hand, those for the third generation are embedded into ${SU}{\left(3\right)}_{C}\times {SU}{\left(2\right)}_{L}\times U{\left(1\right)}_{Y}$ multiplets localized at the y = π R boundary. The reason why such a configuration of the SM fermions is adopted is to avoid massless SM quarks and leptons. If three generations of the SM fermions are localized on the y = 0 boundary, we found that the rank of the mass matrices for the SM quarks and leptons is at most two, which means that at least one massless quark or lepton is inevitable. Therefore, the Lagrangian for the SM fermions ${{ \mathcal L }}_{\mathrm{SM}}$ is expressed by

$$\begin{eqnarray}\begin{array}{ccc}{{ \mathcal L }}_{\mathrm{SM}}^{j=1,2} & = & \delta (y)[{\bar{\chi }}_{10}^{j}{\mathrm{i\Gamma }}^{\mu }{D}_{\mu }{\chi }_{10}^{j}+{\bar{\chi }}_{{5}^{* }}^{j}{\mathrm{i\Gamma }}^{\mu }{D}_{\mu }{\chi }_{{5}^{* }}^{j}+{\bar{\chi }}_{1}^{j}{\mathrm{i\Gamma }}^{\mu }{D}_{\mu }{\chi }_{1}^{j}],\\ {{ \mathcal L }}_{\mathrm{SM}}^{j=3} & = & \delta (y-\pi R)[{\bar{q}}_{L}^{3}{\mathrm{i\Gamma }}^{\mu }{D}_{\mu }{q}_{L}^{3}+{\bar{u}}_{R}^{3}{\mathrm{i\Gamma }}^{\mu }{D}_{\mu }{u}_{R}^{3}+{\bar{d}}_{R}^{3}{\mathrm{i\Gamma }}^{\mu }{D}_{\mu }{d}_{R}^{3}\\ & & +{\bar{l}}_{L}^{3}{\mathrm{i\Gamma }}^{\mu }{D}_{\mu }{l}_{L}^{3}+{\bar{e}}_{R}^{3}{\mathrm{i\Gamma }}^{\mu }{D}_{\mu }{e}_{R}^{3}+{\bar{\nu }}_{R}^{3}{\mathrm{i\Gamma }}^{\mu }{D}_{\mu }{\nu }_{R}^{3}].\end{array}\end{eqnarray} \tag{ 34 }$$

Here, the superscript 'j' denotes the generation of the SM fermions and ${\bar{\chi }}_{b}(b=10,5,1)$ is the Dirac conjugate of χ_b .

In order to realize the SM fermion masses, the boundary localized mass terms between the SM fermions localized at the boundaries and the bulk fermions are necessary. To allow such localized mass terms, we have to choose appropriate SU(6) representations for bulk fermions carefully. Note that, for simplicity, the mirror fermions have no coupling to the SM fermions in this setup. ${{ \mathcal L }}_{\mathrm{SM}+\mathrm{bulk}}$ are boundary mass terms between the bulk fermions and the SM fermions, which are defined as follows.

$$\begin{eqnarray}\begin{array}{rcl}{{ \mathcal L }}_{\mathrm{SM}+\mathrm{bulk}}^{j=1,2} & = & \delta (y)\sqrt{\displaystyle \frac{2}{\pi R}}[{\epsilon }_{20}^{j}({\bar{\chi }}_{10}^{j}{{\rm{\Psi }}}_{10\subset 20}+{\bar{\chi }}_{10}^{j,c}{{\rm{\Psi }}}_{{10}^{* }\subset 20})\\ & & +{\epsilon }_{15}^{j}({\bar{\chi }}_{10}^{j}{{\rm{\Psi }}}_{10\subset 15}+{\bar{\chi }}_{{5}^{* }}^{j,c}{{\rm{\Psi }}}_{5\subset 15})+{\epsilon }_{15^{\prime} }^{j}({\bar{\chi }}_{10}^{j}{{\rm{\Psi }}}_{10^{\prime} \subset 15^{\prime} }+{\bar{\chi }}_{{5}^{* }}^{j,c}{{\rm{\Psi }}}_{5^{\prime} \subset 15^{\prime} })\\ & & +{\epsilon }_{6}^{j}({\bar{\chi }}_{{5}^{* }}^{j,c}{{\rm{\Psi }}}_{5\subset 6}+{\bar{\chi }}_{1}^{j}{{\rm{\Psi }}}_{1\subset 6})+{\epsilon }_{6^{\prime} }^{j}({\bar{\chi }}_{{5}^{* }}^{j,c}{{\rm{\Psi }}}_{5^{\prime} \subset 6^{\prime} }+{\bar{\chi }}_{1}^{j}{{\rm{\Psi }}}_{1^{\prime} \subset 6^{\prime} })+{\rm{h.c.}}],\end{array}\end{eqnarray} \tag{ 35 }$$

$$\begin{eqnarray}\begin{array}{rcl}{{ \mathcal L }}_{\mathrm{SM}+\mathrm{bulk}}^{j=3} & = & \delta (y-\pi R)\sqrt{\displaystyle \frac{2}{\pi R}}[{\epsilon }_{20e}({\bar{e}}_{R}^{3}{E}_{20}+{\bar{u}}_{R}^{3}{U}_{20})+{\epsilon }_{20q}{\bar{q}}_{L}^{3}{Q}_{20}\\ & & +{\epsilon }_{15u}{\bar{u}}_{R}^{3,c}{U}_{15}^{* }+{\epsilon }_{15e}({\bar{e}}_{R}^{3,c}{E}_{15}^{* }+{\bar{l}}_{L}^{3,c}{L}_{15}^{* })+{\epsilon }_{15^{\prime} d}({\bar{q}}_{L}^{3}{Q}_{15^{\prime} }+{\bar{d}}_{R}^{3}{D}_{15^{\prime} })\\ & & +{\epsilon }_{6\nu }({\bar{l}}_{L}^{3,c}{L}_{6}^{* }+{\bar{\nu }}_{R}^{3,c}{N}_{6}^{* })+{\epsilon }_{6^{\prime} d}{\bar{d}}_{R}^{3}{D}_{6^{\prime} }+{\rm{h.c.}}],\end{array}\end{eqnarray} \tag{ 36 }$$

where Ψ_M⊂N is a bulk fermion for M in SU(5) representation and N means SU(6) representation. is the strength of the mixing term between the bulk fermion and the SM fermion and should be a complex number to avoid the problem where the determinant of the mass matrix in equation (18) equals zero. In other words, some SM fermions become a massless state. The decomposition of the introduced bulk fermions in the 20, 15 (${\bf{15}}^{\prime}$), 6 (${\bf{6}}^{\prime}$) representations into the SM gauge group and the corresponding the SM fermions to be coupled on the boundary are summarized in tables 2, 3 and 4, respectively.

Based on the discussion above, our total Lagrangian for the fermions is given as follows:

$$\begin{eqnarray}&&{{ \mathcal L }}_{\mathrm{matter}}={{ \mathcal L }}_{\mathrm{bulk}+\mathrm{mirror}}+{{ \mathcal L }}_{\mathrm{SM}}+{{ \mathcal L }}_{\mathrm{SM}+\mathrm{bulk}}.\end{eqnarray} \tag{ 37 }$$

Solving the exact KK spectrum of the bulk fermions from this Lagrangian is very difficult because of the complicated bulk and boundary system. We assume in this paper that the physical mass induced for the boundary fields is much smaller than the masses of the bulk fields [7]. This is reasonable since the compactification scale and the bulk mass mainly determines the KK mass spectrum of the bulk fields is larger than the mass for the boundary fields whose typical scale is given by the Higgs VEV. In this case, the effects of the mixing on the spectrum for the bulk fields can be negligible and the spectrum ${m}_{n}^{2}={\left(\tfrac{\lambda }{\pi R}\right)}^{2}+{m}_{n}{\left(q\alpha \right)}^{2}$ is a good approximation [7].

Thanks to the reduction of bulk fermions, we can reproduce the quark and lepton mixing angles in addition to the SM fermion masses. For example, the discussion in section 3.1 allows us to obtain up-type quark mass M^u in the following form

$$\begin{eqnarray}{M}^{u}={m}_{W}\sqrt{1+c}\left(\begin{array}{ccc}2{\epsilon }_{20}^{1}{\epsilon }_{20}^{1* }{{\rm{\Lambda }}}_{11} & ({\epsilon }_{20}^{1}{\epsilon }_{20}^{2* }+{\epsilon }_{20}^{2}{\epsilon }_{20}^{1* }){{\rm{\Lambda }}}_{12} & ({\epsilon }_{20}^{1}{\epsilon }_{20u}^{* }+{\epsilon }_{20u}{\epsilon }_{20}^{1* }){{\rm{\Lambda }}}_{13}\\ ({\epsilon }_{20}^{2}{\epsilon }_{20}^{1* }+{\epsilon }_{20}^{1}{\epsilon }_{20}^{2* }){{\rm{\Lambda }}}_{21} & 2{\epsilon }_{20}^{2}{\epsilon }_{20}^{2* }{{\rm{\Lambda }}}_{22} & ({\epsilon }_{20}^{2}{\epsilon }_{20u}^{* }+{\epsilon }_{20u}{\epsilon }_{20}^{2* }){{\rm{\Lambda }}}_{23}\\ ({\epsilon }_{20q}{\epsilon }_{20}^{1* }+{\epsilon }_{20}^{1}{\epsilon }_{20q}^{* }){{\rm{\Lambda }}}_{31} & ({\epsilon }_{20q}{\epsilon }_{20}^{2* }+{\epsilon }_{20}^{2}{\epsilon }_{20q}^{* }){{\rm{\Lambda }}}_{32} & ({\epsilon }_{20q}{\epsilon }_{20u}^{* }+{\epsilon }_{20u}{\epsilon }_{20q}^{* }){{\rm{\Lambda }}}_{33}\end{array}\right),\end{eqnarray} \tag{ 38 }$$

where c = c₁ + c₂, and ${\epsilon }_{20}^{i}(i=1,2),{\epsilon }_{20q},{\epsilon }_{20u}$ are the input parameters that appeared in equations (35) and (36). Λ_ij (i, j = 1, 2, 3) is defined as ${{\rm{\Lambda }}}_{{ij}}\,={\coth }^{2}| {\lambda }_{20}| +{\tanh }^{2}| {\lambda }_{20}| -2(i,j=1,2)$, ${{\rm{\Lambda }}}_{i3}={\lambda }_{3i}=\coth | {\lambda }_{20}| \mathrm{cosec}| {\lambda }_{20}| (i=1,2)$ and ${{\rm{\Lambda }}}_{33}={\coth }^{2}| {\lambda }_{20}| -1$. Mass matrices for other types of quarks and leptons are given in a similar way. After following the method discussed in section 3.2, we can also calculate the flavor-mixing angles and a CP phase and fit them to their experimental values. This is because we have now changed the way the bulk fermions couple to the SM fermions. In the previous model, each type of bulk fermion was introduced per generation so that the mass matrices were essentially diagonal.

Finally, we have identified allowed parameter sets capable of reproducing the SM fermion masses and mixing. Sample data sets depending on the parameter of the localized gauge kinetic terms c are shown in tables 5, 6 and 7. Note that we use experimental data in our analysis and one of the standard conventions for the CKM and PMNS matrices shown by the Particle Data Group [13]. In the analysis of the neutrino sector, we assume a normal hierarchy, although it is not a mandatory requirement. As can be seen from tables 5 and 6, our results are in almost good agreement with the experimental data. ⁷ This is a very remarkable result since it introduces new generation mixing in the bulk resulting from the reduction of the number of bulk fermions, which makes, in particular, reproducing the quark and lepton mixing angles highly nontrivial. Our model turned out to be a good starting point for constructing a realistic model of GGHU.

Table 5. Our results of parameter fitting in the quark sector for some parameters 1/R and c = c₁ + c₂. The up, down and strange quark masses are the $\overline{\mathrm{MS}}$ masses at the scale μ=2 GeV. The charm and bottom quark masses are the $\overline{\mathrm{MS}}$ masses renormalized at the $\overline{\mathrm{MS}}$ mass, i.e. $\bar{m}=\bar{m}(\mu =\bar{m})$. The top quark mass is extracted from direct measurements.

1/R	c	mu	mc	mt
10 TeV	80	2.163 MeV	1.217 GeV	166.294 GeV
10 TeV	90	2.320 MeV	1.229 GeV	167.931 GeV
15 TeV	80	2.316 MeV	1.214 GeV	165.300 GeV
15 TeV	90	2.156 MeV	1.225 GeV	166.89 GeV
Data		${2.16}_{-0.26}^{+0.49}$ MeV	1.27 ± 0.02 GeV	172 ± 0.30 GeV

1/R	c	md	ms	mb
10 TeV	80	5.583 MeV	75.7 MeV	4.155 GeV
10 TeV	90	5.505 MeV	75.8 MeV	4.321 GeV
15 TeV	80	5.522 MeV	75.5 MeV	4.201 GeV
15 TeV	90	5.545 MeV	75.1 MeV	4.183 GeV
Data		${4.67}_{-0.17}^{+0.48}$ MeV	${93}_{-5}^{+11}$ MeV	${4.18}_{-0.02}^{+0.13}$ GeV

1/R	c	$\sin {\theta }_{12}$	$\sin {\theta }_{13}$	$\sin {\theta }_{23}$	δ
10 TeV	80	0.191 797	0.003 537	0.041 430	1.1560
10 TeV	90	0.195 857	0.003 510	0.039 893	1.2424
15 TeV	80	0.190 839	0.003 556	0.041 459	1.1831
15 TeV	90	0.192 085	0.003 518	0.040 088	1.1750
Data		0.226 50 ± 0.00048	${0.00361}_{-0.00009}^{+0.00011}$	${0.04053}_{-0.00061}^{+0.00083}$	${1.196}_{-0.043}^{+0.045}$

Table 6. Our results of parameter fitting in the lepton sector for some parameters 1/R and c = c₁ + c₂. In the neutrino sector, a normal hierarchy is assumed.

1/R	c	me	mμ	mτ
10 TeV	80	0.5136 MeV	98.750 MeV	1687.12 MeV
10 TeV	90	0.5140 MeV	98.188 MeV	1689.56 MeV
15 TeV	80	0.5135 MeV	98.776 MeV	1695.46 MeV
15 TeV	90	0.5139 MeV	98.610 MeV	1687.59 MeV

Data		0.510 998 946 1(31) MeV	105.658 374 5(24) MeV	1776.86(12) MeV
1/R	c	${\rm{\Delta }}{m}_{21}^{2}$	${\rm{\Delta }}{m}_{32}^{2}$ (Normal)	δ
10 TeV	80	7.7306 × 10−5 eV2	2.4524 × 10−3 eV2	1.539π rad
10 TeV	90	7.7087 × 10−5 eV2	2.4367 × 10−3 eV2	1.536π rad
15 TeV	80	7.8054 × 10−5 eV2	2.3895 × 10−3 eV2	1.531π rad
15 TeV	90	7.6544 × 10−5 eV2	2.4577 × 10−3 eV2	1.536π rad

Data		(7.53 ± 0.18) × 10−5 eV2	(2.453 ± 0.033) × 10−3 eV2	${1.36}_{-0.16}^{+0.20}\pi$ rad
1/R	c	${\sin }^{2}{\theta }_{12}$	${\sin }^{2}{\theta }_{13}$	${\sin }^{2}{\theta }_{23}$ (Normal)
10 TeV	80	0.3313	2.240 × 10−2	0.5161
10 TeV	90	0.3294	2.155 × 10−2	0.5187
15 TeV	80	0.3505	2.094 × 10−2	0.5069
15 TeV	90	0.3308	2.123 × 10−2	0.5161
Data		0.307 ± 0.013	(2.20 ± 0.07) × 10−2	0.546 ± 0.021

Table 7. Input parameters of the parameter fitting where c = c₁ + c₂. ${\lambda }_{a}(a=20,15,15^{\prime} ,6,6^{\prime} )$ are the bulk masses between the bulk and the mirror fermions which are normalized by π R. is the strength of the mixing term between the bulk fermion and the SM fermion, ∣∣ is its absolute value and θ is a phase of the corresponding . Only ${\epsilon }_{a}^{1}(a=20,15,15^{\prime} ,6,6^{\prime} )$ can be taken as a real number without loss of generality. This is because we have not shown ${\theta }_{a}^{1}(a=20,15,15^{\prime} ,6,6^{\prime} )$ in this table.

1/R	c	λ20	λ15	${\lambda }_{15^{\prime} }$	λ6	${\lambda }_{6^{\prime} }$
10 TeV	80	0.697 103	0.379 299	1.866 68	13.404	12.0452
10 TeV	90	0.707 565	0.382 020	1.875 16	13.383	12.0376
15 TeV	80	0.727 571	0.421 544	1.885 62	13.390	11.0374
15 TeV	90	0.737 359	0.428 899	1.896 30	13.355	11.0374

1/R	c	$| {\epsilon }_{20}^{1}|$	$| {\epsilon }_{15}^{1}|$	$| {\epsilon }_{15^{\prime} }^{1}|$	$| {\epsilon }_{6}^{1}|$	$| {\epsilon }_{6^{\prime} }^{1}|$
10 TeV	80	1	0.061 893 8	0.398 107	0.000 724 436	0.020 417 4
10 TeV	90	1.000 05	0.062 408 1	0.398 146	0.000 668 344	0.019 724 2
15 TeV	80	0.999 60	0.062 089 9	0.398 107	0.000 031 622 8	0.006 683 44
15 TeV	90	0.999 994	0.062 593	0.398 095	0.000 029 512 1	0.006 531 31

1/R	c	$| {\epsilon }_{20}^{2}|$	$| {\epsilon }_{15}^{2}|$	$| {\epsilon }_{15^{\prime} }^{2}|$	$| {\epsilon }_{6}^{2}|$	$| {\epsilon }_{6^{\prime} }^{2}|$
10 TeV	80	0.190 546	0.014 150 8	0.079 432 8	0.000 426 58	0.010 115 8
10 TeV	90	0.190 545	0.014 177 3	0.079 433 3	0.000 380 189	0.009 885 53
15 TeV	80	0.190 546	0.014 159 3	0.079 432 8	0.000 028 510 2	0.004 954 5
15 TeV	90	0.190 546	0.014 197 0	0.079 433 1	0.000 026 607 3	0.004 841 72

1/R	c	${\theta }_{20}^{2}$	${\theta }_{15}^{2}$	${\theta }_{15^{\prime} }^{2}$	${\theta }_{6}^{2}$	${\theta }_{6^{\prime} }^{2}$
10 TeV	80	0.007 368 13	0.258 731	0.070 371 2	5.956 450	3.783 11
10 TeV	90	0.007 449 81	0.253 490	0.071 090 4	5.956 450	3.783 11
15 TeV	80	0.007 758 07	0.268 290	0.073 149 7	0.712 979	4.940 97
15 TeV	90	0.007 744 13	0.263 562	0.074 399 5	0.752 089	4.940 97

1/R	c	∣20e ∣	∣15u ∣	$| {\epsilon }_{15^{\prime} d}|$	∣6ν ∣	$| {\epsilon }_{6^{\prime} d}|$
10 TeV	80	0.054 238 6	0.000 873 495	0.008 777 01	0.000 01	0.000 01
10 TeV	90	0.052 238 6	0.000 847 602	0.008 491 88	0.000 010 115 8	0.000 01
15 TeV	80	0.054 210 9	0.000 878 758	0.008 885 08	0.000 213 796	0.000 01
15 TeV	90	0.051 966 0	0.000 853 353	0.008 584 60	0.000 211 349	0.000 01

1/R	c	θ20e	θ15u	${\theta }_{15^{\prime} d}$	θ6ν	${\theta }_{6^{\prime} d}$
10 TeV	80	4.251 55	4.147 07	4.213 75	3.180 33	0.000 01
10 TeV	90	4.226 78	4.165 61	4.214 89	3.180 33	0.000 01
15 TeV	80	4.289 15	4.111 34	4.223 85	4.560 62	0.000 01
15 TeV	90	4.286 32	4.129 57	4.2136	4.560 62	0.000 01

1/R	c	∣20q ∣	∣15e ∣	θ20q	θ15e
10 TeV	80	0.011 359 9	0.001 225 72	2.155 76	2.163 23
10 TeV	90	0.010 911 6	0.001 965 32	2.137 19	2.595 40
15 TeV	80	0.011 597 6	0.001 235 53	2.193 91	2.572 35
15 TeV	90	0.011 416 7	0.001 407 87	2.175 91	2.651 10

Note that these parameters are evaluated at the compactification scale. In order to compare these values with the experimental data, we have to solve the renormalization group equation (RGE) from the compactification scale (${ \mathcal O }(10)$ TeV) to the weak scale, but we expect that these RGE effects do not affect our results.

In this paper, we have discussed the fermion mass hierarchy and mixing in SU(6) GGHU with localized gauge kinetic terms. The SM fermions are introduced on the boundaries. We also introduced massive bulk fermions in three types of SU(6) representations coupling to the SM fermions on the boundaries. Their number has been reduced in order to achieve generation mixings of quarks and leptons, which greatly changed the coupling of the bulk fermions to the SM fermions on the boundaries and led to the appearance of additional generation mixings in the bulk sector. This feature makes our analysis of the SM fermion masses and mixing angles highly complicated and nontrivial. We have shown that the SM fermion masses and mixing can be almost reproduced by mild tuning of bulk masses and the parameters of the localized gauge kinetic terms. Some parameter sets of our results are listed. The model discussed in this paper turned out to be a good starting point for constructing a realistic model of GGHU.

For our future work, it is important to calculate the effective potential for the Higgs field and study whether the electroweak symmetry breaking occurs correctly. Since the Higgs field is originally a gauge field, the potential is generated at the one-loop level by the Coleman–Weinberg mechanism. It is not easy to obtain the observed Higgs mass 125 GeV because the effects of localized gauge kinetic terms enhance the compactification scale and Higgs boson mass. From the viewpoint of gauge coupling unification, we have little room to introduce extra bulk fields to adjust the Higgs mass, which also makes the analysis of the electroweak symmetry breaking difficult. It is possible to introduce a Majorana neutrino on the boundary in order to obtain the Higgs boson mass, which relaxes the constraint for the bulk mass parameters. We would like to investigate the electroweak symmetry breaking and Higgs boson mass along this line. The predictions from this model could be also studied based on these analyses.

This work was supported by JST SPRING, Grant Number JPMJSP2139 (Y Y ).

All data that support the findings of this study are included within the article (and any supplementary files).

The bulk and mirror fermions ${\rm{\Psi }}(x,y),\tilde{{\rm{\Psi }}}(x,y)$ in five dimensions and their KK decomposition is given by

$$\begin{eqnarray}\begin{array}{rcl}{\rm{\Psi }}(x,y) & = & \displaystyle \sum _{n}{f}_{n}(y){\chi }_{n}(x),\\ \tilde{{\rm{\Psi }}}(x,y) & = & \displaystyle \sum _{n}{f}_{n}(y){\tilde{\chi }}_{n}(x),\end{array}\end{eqnarray} \tag{ A1 }$$

where ${\chi }_{n}(x),{\tilde{\chi }}_{n}(x)$ are 4D fields and f_n (y) is a mode function. The quadratic terms for ${\chi }_{n}(x),{\tilde{\chi }}_{n}(x)$ can be expressed in momentum space by using the above KK decomposition as

$$\begin{eqnarray}({\chi }_{n}^{\dagger },{\tilde{\chi }}_{n}^{\dagger })\left(\begin{array}{cc}\rlap{/}{p}-{m}_{n} & M\\ M & \rlap{/}{p}+{m}_{n}\end{array}\right)\left(\begin{array}{c}{\chi }_{n}\\ {\tilde{\chi }}_{n}\end{array}\right),\end{eqnarray} \tag{ A2 }$$

which leads to the propagator in Minkowski spacetime

$$\begin{eqnarray}{{\rm{\Delta }}}_{\chi }=\frac{{\rm{i}}}{{p}^{2}-{m}_{n}^{2}-{M}^{2}}\left(\begin{array}{cc}\rlap{/}{p}+{m}_{n} & -M\\ -M & \rlap{/}{p}-{m}_{n}\end{array}\right),\end{eqnarray} \tag{ A3 }$$

or in Euclidean spacetime

$$\begin{eqnarray}{{\rm{\Delta }}}_{\chi }^{E}={\mathrm{i\Delta }}_{\chi }=\frac{1}{{p}_{E}^{2}+{m}_{n}^{2}+{M}^{2}}\left(\begin{array}{cc}i{\rlap{/}{p}}_{E}-{m}_{n} & M\\ M & i\rlap{/}{p}+{m}_{n}\end{array}\right).\end{eqnarray} \tag{ A4 }$$

In order to evaluate equation (14), we need to calculate the summation of KK mode contributions. The summation of KK mode contributions can be rewritten as

$$\begin{eqnarray}\begin{array}{ccl}{f}_{\delta }^{(T)}(x,q\alpha ) & = & \displaystyle \sum _{n}\frac{{\left(-1\right)}^{\delta n}}{x+{\rm{i}}\pi (n+\nu +\alpha )}\\ & = & \left\{\begin{array}{cc}\coth (x+{\rm{i}}\pi \alpha ) & :\delta =0,T=+1,(\nu =0)\\ \sinh {\left(x+{\rm{i}}\pi \alpha \right)}^{-1} & :\delta =1,T=+1,(\nu =0)\\ \tanh (x+{\rm{i}}\pi \alpha ) & :\delta =0,T=-1,(\nu =1/2)\\ \cosh {\left(x+{\rm{i}}\pi \alpha \right)}^{-1} & :\delta =1,T=-1,(\nu =1/2)\end{array}\right.,\end{array}\end{eqnarray} \tag{ B1 }$$

where δ = 0(1) for the case where the left-handed and right-handed boundary fermions are localized on the same (opposite) boundary. T is the periodicity for bulk and mirror fermions. The factor n + ν + q α in the denominator comes from the KK mass spectrum

$$\begin{eqnarray}&&{m}_{n}=\frac{n+\nu +q\alpha }{R},\end{eqnarray} \tag{ B2 }$$

where α is a dimensionless parameter corresponding to the VEV of the Higgs field.

Furthermore, it is useful to express the real and imaginary part of these functions.

$$\begin{eqnarray}\begin{array}{ccc}\mathrm{Re}{f}_{\delta }^{(T)}(x,q\alpha ) & = & \frac{1}{2}\displaystyle \sum _{n}\left(\frac{{\left(-1\right)}^{\delta n}}{x+{\rm{i}}\pi (n+\nu +\alpha )}+\frac{{\left(-1\right)}^{\delta n}}{x-{\rm{i}}\pi (n+\nu +\alpha )}\right)\\ & = & \displaystyle \sum _{n}\frac{x{\left(-1\right)}^{\delta n}}{{x}^{2}+{\pi }^{2}{\left(n+\nu +\alpha \right)}^{2}},\end{array}\end{eqnarray} \tag{ B3 }$$

$$\begin{eqnarray}\begin{array}{ccc}\mathrm{Im}{f}_{\delta }^{(T)}(x,q\alpha ) & = & \frac{1}{2{\rm{i}}}\displaystyle \sum _{n}\left(\frac{{\left(-1\right)}^{\delta n}}{x+{\rm{i}}\pi (n+\nu +\alpha )}-\frac{{\left(-1\right)}^{\delta n}}{x-{\rm{i}}\pi (n+\nu +\alpha )}\right)\\ & = & \displaystyle \sum _{n}\frac{-{\left(-1\right)}^{\delta n}\pi (n+\nu +\alpha )}{{x}^{2}+{\pi }^{2}{\left(n+\nu +\alpha \right)}^{2}},\end{array}\end{eqnarray} \tag{ B4 }$$

therefore we obtain

$$\begin{eqnarray}\begin{array}{rcl}\displaystyle \sum _{n}\displaystyle \frac{{\left(-1\right)}^{\delta n}}{{x}^{2}+{\left(\pi {{Rm}}_{n}\right)}^{2}} & = & \displaystyle \frac{1}{x}{\rm{Re}}{f}_{\delta }^{(T)}(x,q\alpha ),\\ \displaystyle \sum _{n}\displaystyle \frac{{\left(-1\right)}^{\delta n}{m}_{n}}{{x}^{2}+{\left(\pi {{Rm}}_{n}\right)}^{2}} & = & -\displaystyle \frac{1}{\pi R}{\rm{Im}}{f}_{\delta }^{(T)}(x,q\alpha ).\end{array}\end{eqnarray} \tag{ B5 }$$