1 Introduction

Moiré materials that exhibit flat bands such as twisted bilayer graphene (tBG) or certain van der Waals heterostructures such as trilayer graphene on hexagonal boron nitride (TLG/h-BN) have recently been established as novel, highly tunable platforms for the study of strongly correlated electrons. Relative to an almost vanishing bandwidth, residual interactions in these materials can induce a plethora of different many-body phenomena ranging from the formation of correlated insulators [1,2,3,4] and superconductors [5,6,7] to anomalous quantum Hall effects [8]. However, a microsopic description of these phenomena is a formidable challenge as the number of of low-energy degrees of freedom is often increased [9,10,11] in comparison to conventional Mott insulators.

Fig. 1
figure 1

a Moiré pattern emerging in two stacked layers of graphene with a relative twist angle \(\theta \). Clearly visible are the different regions with AA, AB, and BA stacking leading to a triangular super-lattice structure. b Construction of the two degenerate mini Brillouin zones from the difference of the K (or K’) points of the two layers of graphene. In addition to the spin degree of freedom, indicated by the grey arrows, the electrons obtain a valley degree of freedom due to the possibility of being in either one of the mini Brillouin zones at the two valleys (at the K and K’ points) of the graphene band structure

More specifically, it has been argued [12, 13] that multi-orbital Hubbard models can describe the flat band physics in, e.g., TLG/h-BN within the topologically trivial regime, where fully symmetric Wannier states may be constructed [14]. The proposed interaction terms for the corresponding Hamiltonians usually include a generalized Hubbard U [12, 13, 15] as well as Hund’s type couplings. Performing a strong coupling expansion where one treats the interactions as the dominant energy scale, these extended Hubbard models can then be mapped to \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(4)Footnote 1 spin–valley Hamiltonians that may be used as a starting point to investigate the nature of the correlated insulating states. The so-derived \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(4) models bear a close resemblance to Kugel–Khomskii models [16] that have a long history in the study of transition metal oxides, where they are used to capture the Jahn–Teller physics of intertwined spin and orbital degrees of freedom. Increasing the number of relevant microscopic degrees of freedom (in comparison to conventional quantum magnets) has been particularly appreciated to boost quantum fluctuations independent of, e.g., lattice geometries [17], which has made Kugel–Khomskii models a recurring target in the search for unusual many-body states such as quantum spin–orbital liquids [18,19,20,21]. As such, one might expect the \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(4) spin–valley physics relevant to the correlated insulating states of moiré materials to hold similar promise for the observation of spin–valley liquid states with macroscopic entanglement and potentially long-range, topological order.

In this manuscript, we present a powerful numerical scheme to analyze such \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(4) spin–valley (or spin–orbital) models based on a functional renormalization group (FRG) technique. Our approach is based on the pseudo-fermion FRG (pf-FRG) [22], approximating the elementary spin operators of the six-dimensional, self-conjugate representation of \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(4) by auxiliary complex fermions combined with an on-average constraint on the number of particles per site. Our approach allows to go beyond mean-field level by treating competing instabilities in different interaction channels on equal footing, and is able to capture both, long-ranged spin and/or valley ordered states as well as spin–valley liquid phases. In expanding previous work (by some of us) [21], we extend the range of applicability of this approach to models with off-diagonal interactions in either spin or valley space by formulating an efficient vertex parametrization derived from a meticulous symmetry analysis. We demonstrate the feasibility of this method by studying a spin–valley Hamiltonian with SU(2)\(_{\text {spin}}\) \(\otimes \) U(1)\(_{\text {valley}}\) symmetry where we identify a plethora of spin and valley orderded phases from a state-of-the-art numerical implementation of pf-FRG [23, 24].

The remainder of this manuscript is structured as follows. To begin with, we introduce the spin–valley Hamiltonian of interest on a general level and discuss its specific form for TLG/h-BN as a concrete example in Sect. 2. We will continue by reviewing the pf-FRG approach (Sect. 3), its generalization for \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(4) models as well as the implementation of model specific symmetries (Sect. 4). Finally, numerical results for the phase diagram of a SU(2)\(_{\text {spin}}\) \(\otimes \) U(1)\(_{\text {valley}}\) model on the triangular lattice are presented and examined in Sect. 5.

2 Model

Microscopically, the SU(4) models of interest in this manuscript can be cast in terms of a general Hamiltonian

$$\begin{aligned} {\mathcal {H}} = \frac{1}{8} \sum _{\langle ij \rangle } J (1 + \varvec{\sigma }_i \varvec{\sigma }_j) (1 + \varvec{\tau }_i \varvec{\tau }_j) \,, \end{aligned}$$
(1)

which couples two elementary \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(2) degrees of freedom, captured by the operators \(\sigma \) and \(\tau \), which might denote a spin and valley (or oribtal) degree of freedom. The overall SU(4) symmetry of the Hamiltonian arises from the balanced couplings of equal strength in both degrees of freedom, i.e., J is identical for the Heisenberg-like coupling of spins \(\varvec{\sigma }_i \varvec{\sigma }_j\) on sites i and j (with \(\varvec{\sigma }_i = (\sigma _j^x, \sigma _j^y, \sigma _j^z)^T\)) and a similar interaction of the valley degrees of freedom \(\varvec{\tau }_i \varvec{\tau }_j\). Such valley degrees of freedom arise, in the context of tBG and related moiré materials, from the Dirac cones in the original graphene bands, which hybridize between the two layers upon twisting and thereby add an extra index [25] to the moiré bands, as illustrated in Fig. 1. Before drawing broad attention in the context of moiré materials, the spin–orbital variant of this model has been widely studied as Kugel–Khomskii model [16], often in connection with Jahn–Teller physics in transition metal oxides where spin and orbital ordering are intertwined [26]. We note that while we will frame our discussion of the SU(4) model (1) in the language of spin-valley physics relevant to moiré materials, the presented pf-FRG approach is equally applicable in the study of such spin–orbital models. We will return to this point in the discussion section at the end.

In what we will discuss in the following, we will put a focus on the self-conjugate representation of \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(4), where the spin–valley operators can be represented in terms of fermionic creation and annihilation operators as

$$\begin{aligned} \sigma _{i}^{\mu } \tau _{i}^{\kappa }&\equiv \sigma _{i}^{\mu } \otimes \tau _{i}^{\kappa } = f_{i s l}^{\dagger } \theta _{s s'}^{\mu } \theta ^{\kappa }_{l l'} f^{{\dagger }}_{i s' l'} \nonumber \\ \sigma _i^{\mu }&\equiv \sigma _i^{\mu } \otimes \tau _i^0 = f_{i s l}^{\dagger } \theta _{s s'}^{\mu } f^{{\dagger }}_{i s' l} \nonumber \\ \tau _i^\kappa&\equiv \sigma _i^0 \otimes \tau _i^{\kappa } = f_{i s l}^{\dagger } \theta _{l l'}^{\kappa } f^{{\dagger }}_{i s l'} \,, \end{aligned}$$
(2)

with a local half-filling constraint

$$\begin{aligned} f_{i sl}^{\dagger } f_{i sl}^{{\dagger }} = 2 \end{aligned}$$
(3)

subject to every lattice site, where summation over repeated spin indices s and valley indices l is implied. Here, \(\theta ^{\mu }\) denotes a Pauli matrix with \(\mu \in \{0, 1, 2, 3\}\) and \(\theta ^{0} = {\mathbbm {1}}\). Allowing also for more generic, i.e., SU(4) breaking, interactions, any bilinear spin–valley Hamiltonian can be written as

(4)

where \(J_{s, ij}^{\mu \nu } \otimes J_{v, ij}^{\kappa \lambda }\) is understood as the Kronecker product of the spin and valley exchange matrices and summation over repeating \(\mu ,\nu ,\kappa \) or \(\lambda \) is again implied. Here, \({\hat{n}}_i\) is the density operator \({\hat{n}}_i \equiv \sigma _i^0 \tau _i^0 = f^\dagger _{isl}f^{{\dagger }}_{isl}\), and the term proportional to the coupling \(I_{ij}\) is needed to potentially cancel the density term \(\sim \sigma _i^0 \tau _i^0 J^{00}_{s, ij}J^{00}_{v, ij} \sigma _j^0 \tau _j^0\), which does not appear in pure \({{\mathfrak {s}}}{{\mathfrak {u}}}(4)\) spin models as, e.g., the SU(4) symmetric Hamiltonian in Eq. (1).

To keep the numerical effort for employing our pf-FRG approach at a manageable level, we assume a specific form of the exchange matrices, namely, that both, the spin and the valley exchange only couple bilinears of spin/valley or density operators and that the spin exchange is \({\mathbb {Z}}_2 \times {\mathbb {Z}}_2 \times {\mathbb {Z}}_2\) symmetric, and thus

$$\begin{aligned} \begin{aligned} J_{s, ij}&= \begin{pmatrix} J_{s,ij}^d &{} 0 &{} 0 &{} 0 \\ 0 &{} J_{s,ij}^{x} &{} 0 &{} 0 \\ 0 &{} 0 &{} J_{s,ij}^{y} &{} 0 \\ 0 &{} 0 &{} 0 &{} J_{s,ij}^{z} \end{pmatrix}\\ J_{v, ij}&= \begin{pmatrix} J_{v,ij}^d &{} 0 &{} 0 &{} 0 \\ 0 &{} J_{v,ij}^{xx} &{} J_{v,ij}^{xy} &{} J_{v,ij}^{xz} \\ 0 &{} J_{v,ij}^{yx} &{} J_{v,ij}^{yy} &{} J_{v,ij}^{yz} \\ 0 &{} J_{v,ij}^{zx} &{} J_{v,ij}^{zy} &{} J_{v,ij}^{zz} \end{pmatrix} \,. \end{aligned} \end{aligned}$$
(5)

This form, although it spoils the generality of Eq. (4) it is nevertheless relevant to certain practical applications. For instance, the effective Hamiltonian for TLG/h-BN [11] can be recast to this form. Originally, the former is often given as

$$\begin{aligned} {\mathcal {H}}= & {} \frac{J_1}{8} \sum _{\langle ij \rangle } (1 + \varvec{\sigma }_i \varvec{\sigma }_j)(1 + \varvec{\tau }_i \varvec{\tau }_j) \nonumber \\&+ \frac{J_2}{8} \sum _{\langle \langle ij \rangle \rangle } (1 + \varvec{\sigma }_i \varvec{\sigma }_j)(1 + \varvec{\tau }_i \varvec{\tau }_j) \nonumber \\&+ \frac{1}{8} \sum _{\langle ij \rangle } J^1_{p;ij}(1 + \varvec{\sigma }_i \varvec{\sigma }_j)(\tau ^x_i \tau ^x_j +\tau ^y_i \tau ^y_j)\nonumber \\&+ \frac{1}{8} \sum _{\langle ij \rangle } J^2_{p;ij}(1 + \varvec{\sigma }_i \varvec{\sigma }_j)(\tau ^x_i \tau ^y_j - \tau ^y_i \tau ^x_j)+ O\left( \!\frac{t^3}{U^2}\!\right) ,\nonumber \\ \end{aligned}$$
(6)

which, in addition to SU(4) symmetric nearest-neighbour (\(\sim J_1\)) and next-nearest-neighbour (\(\sim J_2\)) interactions, contains both diagonal \(\sim J^1_{p, ij}\) and off-diagonal \(\sim J^2_{p,ij}\) valley exchange that breaks the SU(4) symmetry down to an SU(2)\(_\mathrm {spin}\) \(\otimes \) U(1)\(_\mathrm {valley}\) symmetry. Comparing this model to the form of the general spin–valley Hamiltonian defined in Eq. (4), the nearest-neighbour exchange matrices can be written as

$$\begin{aligned} \begin{aligned} J_{s, ij}&= {\mathbbm {1}} \\ J_{v, ij}&= \begin{pmatrix} J_1 &{} 0 &{} 0 &{} 0 \\ 0 &{} J_1 + J^1_{p;ij} &{} J^2_{p;ij} &{} 0 \\ 0 &{} -J^2_{p;ij} &{} J_1 + J^1_{p;ij} &{} 0 \\ 0 &{} 0 &{} 0 &{} J_1 \end{pmatrix}\, , \end{aligned} \end{aligned}$$
(7)

and the next-nearest-neighbour exchange is fully SU(4) symmetric, showing that they are indeed captured by the exchange matrices defined in Eq. (5).

3 pf-FRG for spin–valley models: an overview

We now proceed to the core methodological advancement of this manuscript, which will be laid out in this section—the extension of the conventional pf-FRG to spin–valley models described by Hamiltonians of the form given in Eq. (4), with general, diagonal, and off-diagonal couplings as defined by Eq. (5). To set the stage, we will first revisit the flow equations of the conventional pf-FRG approach for \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(2) spins and explain how the numerical solution of the flow equations can be used to determine whether and what type of magnetic order forms for a particular spin Hamiltonian at zero temperatures. We then proceed to the adapted pf-FRG approach for spin–valley models, for which we derive an efficient parametrization of the self-energy and two-particle vertex in what is a direct extension of the parametrization for \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models with generic two-spin interactions [27]. Our particular focus is on constraints that symmetries of the spin–valley Hamiltonian pose on the parametrized vertex functions—very similar to the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case but with slight differences which we especially highlight. To put these equations into numerical practice, we discuss our implementation of the spin–valley pf-FRG approach and its algorithmic scaling. This section is intended as an overview stating the main results of our study important for the implementation of the pf-FRG for spin–valley models. Readers looking for a more detailed discussion of how the symmetries of the Hamiltonian lead to the parametrization and symmetry constraints are referred to Sect. 4.

3.1 Pseudo-fermion functional renormalization group

Let us set the stage by revisiting some of the conceptual steps of the pseudo-fermion FRG, which has originally been formulated for bilinear \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models [22] with generic (diagonal and off-diagonal) interactions [27] and later generalized to SU(N) Heisenberg models [28], in the context of the spin–valley models at hand. By going to a pseudo-fermion representation of the original degrees of freedom, one arrives at a fermionic representation of the original model (with an additional half-filling constraint) as outlined in the previous section. One can then proceed to apply the well-established methods of the fermionic FRG [29, 30].

An important distinction to electronic systems is that the pseudo-fermion Hamiltonian exhibits only a quartic interaction term and no quadratic kinetic terms. This readily implies that the free propagator is diagonal in all its arguments and takes the simple form

$$\begin{aligned} G_0(1', 1) = G_0(\omega _1) \delta _{i_{1'} i_1} \delta _{s_{1'} s_1} \delta _{l_{1'}l_1} \delta _{\omega _{1'} \omega _1}, \end{aligned}$$
(8)

with \(G_0(\omega ) = (i\omega )^{-1}\). The multi-index \(1=(i_1, s_1, l_1, \omega _1)\) consists of a lattice site index \(i_1\), a spin index \(s_1\), a fermionic Matsubara frequency \(\omega _1\) and, for spin–valley models, the additional valley index \(l_1\). To implement the RG scale, or cut-off, \(\varLambda \), we multiply a regulator to the free propagator

$$\begin{aligned} G_0^\varLambda (\omega ) = G_0(\omega ) (1-e^{-\omega ^2/\varLambda ^2}), \end{aligned}$$
(9)

where we choose a smooth regulator for improved numerical stability. The pf-FRG flow equations are then given as a special case of the general fermionic FRG equations by assuming that the flowing self-energy is, just as the free propagator, diagonal in all its arguments. This assumption is true for arbitrary spin-model bilinear in \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin operators [27]. For spin–valley Hamiltonians, however, we will show in Sect. 4 that this is only the case if the couplings are diagonal in either the spin or valley sector. That is why, in this work, we always consider couplings diagonal in the spin sector as stated in Eq. (5). In the context of moiré materials, most physically relevant spin–valley models are indeed of this form. This additional assumption, therefore, leaves our method still generally applicable to most models of interest.

In the original implementation of the pf-FRG [22] and most works since, then the flow equations are truncated using the Katanin truncation scheme [31], which we also adapt hereFootnote 2. In the Katanin truncation, only the self-energy \(\varSigma ^\varLambda \) and the two-particle vertex \(\varGamma ^\varLambda \) are considered, while higher order vertex functions are neglected. The flow equations are then given by

$$\begin{aligned} \frac{d}{d\varLambda }\varSigma ^\varLambda (1', 1) = - \frac{1}{2\pi } \sum _{2} \varGamma ^\varLambda (1', 2, 1, 2) S^\varLambda (\omega _2) \end{aligned}$$
(10)

for the self-energy and

$$\begin{aligned}&\frac{d}{d\varLambda } \varGamma ^\varLambda (1', 2', 1, 2) \nonumber \\&\quad = -\frac{1}{2\pi } \sum _{3, 4} \bigg [ \varGamma ^\varLambda (1', 2', 3, 4)\varGamma ^\varLambda (3, 4, 1, 2)\nonumber \\&\qquad -\varGamma ^\varLambda (1', 4, 1, 3)\varGamma ^\varLambda (3, 2', 4, 2) - (3 \leftrightarrow 4)\nonumber \\&\qquad + \varGamma ^\varLambda (2', 4, 1, 3)\varGamma ^\varLambda (3, 1', 4, 2) + (3 \leftrightarrow 4) \bigg ]\nonumber \\&\qquad \times G^\varLambda (\omega _3)\partial _\varLambda G^\varLambda (\omega _4), \end{aligned}$$
(11)

for the two-particle vertex. Here, the single-scale propagator is defined as \(S^{\varLambda } \equiv -\partial _\varLambda G^{\varLambda }|_{\varSigma ^{\varLambda } = \text {const.}}\). Note that the flow equations are formulated in the \(T \rightarrow 0\) limit and the sums should therefore be understood as \(\sum _{1} \equiv \sum _{i_1 s_1 l_1} \int d\omega _1\).

Fig. 2
figure 2

Flow of the spin and valley structure factor in a magnetically ordered phase (a) and a paramaganetic phase (b) for different values of the vertex truncation length L. All structure factors are shown at the momentum at which they are maximal. The insets zoom into the flow at small cut-offs. In the magnetically ordered phase, we clearly see a breakdown of the flow in the valley sector, which manifests as a peak for small L and a more clear divergence when increasing L. In the paramagnetic phase, the flow is smooth and convex down to about \(\varLambda /J = 0.02\), which is the smallest scale for which our calculations are numerically reliable

The fermionic representation of the spin–valley operators, as presented in the previous section, necessitates the enforcement of a local half-filling constraint \((f^\dagger _{isl}f^{{\dagger }}_{isl} = 2)\) to determine the dimensionality of the local Hilbert space. To this end, we employ the same technique used for \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) models, where the constraint is fulfilled only on average by explicitly implementing particle-hole symmetry on the level of the vertex functions [22,23,24], as will be discussed in detail in Sect. 4. Numerically, computing the expectation value \(\langle f^\dagger _{isl}f^{{\dagger }}_{isl} \rangle \) from the self-energy and two-particle vertex, we confirm that the average constraint is indeed fulfilled during the entire pf-FRG flow. Although particle-number fluctuations violating the exact constraint have been shown to be sizeable, recent studies suggest that they leave the physical results obtained from the pf-FRG qualitatively unaffected [21, 23, 34]. We note that in contrast to \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models, the physically relevant filling for spin–valley models depends on the considered material and may also be at quarter-filling \((f^\dagger _{isl}f^{{\dagger }}_{isl} = 1)\) [11, 35], corresponding to the fundamental representation of \({{\mathfrak {s}}}{{\mathfrak {u}}}(4)\). At quarter-filling, however, the spin–valley Hamiltonian is no longer particle–hole symmetric and the constraint cannot be enforced in the same efficient manner. In this work, we therefore limit ourselves to half-filling.

To identify the ground state of a model of interest, we numerically solve the flow equations (as discussed in more detail below in Sect. 3.3) and thereby calculate the flow of various correlation functions from the flow of the vertex functions. In its most general we define a spin-valley-spin-valley correlation function

$$\begin{aligned} \chi _{ij}^{\mu \nu \kappa \lambda }(\omega ) = \int _0^\infty d\tau e^{i\omega \tau } \left\langle T_\tau (\sigma ^\mu _i \otimes \tau _i^\kappa )\nonumber \right. (\tau ) \\ \left. \times (\sigma ^\nu _j \otimes \tau _j^\lambda )(0)\right\rangle , \end{aligned}$$
(12)

where \(T_{\tau }\) is the time-ordering operator. From this general definition, we can then read off the form of spin–spin correlations

$$\begin{aligned} \chi _{ij}^{s, \mu \nu } \equiv \chi _{ij}^{\mu \nu 00} \sim \left\langle \sigma _i^\mu \sigma _j^\nu \right\rangle , \end{aligned}$$
(13)

as well as valley–valley correlations

$$\begin{aligned} \chi _{ij}^{v, \kappa \lambda } \equiv \chi _{ij}^{00\kappa \lambda } \sim \left\langle \tau _i^\kappa \tau _j^\lambda \right\rangle . \end{aligned}$$
(14)

A thermal phase transition to long-range, symmetry-breaking order in the spin or valley sector at some finite temperature can formally be detected by a divergence in the RG flow of the corresponding correlation at some breakdown scale \(\varLambda _c\) [28], as shown in Fig. 2a. Due to finite numerical resolution, however, they often manifest as a kink or a peak in the susceptibility. The momentum space profile of the dominant structure factor close to the breakdown scale, i.e., the Fourier transform of the static correlation \(\chi ^{\varLambda s/v}_{ij}(\omega = 0)\), then indicates the type of symmetry-breaking. Since the solution of the flow equation below the breakdown scale \(\varLambda _c\) is no longer physical, this only allows us to detect the phase transition that occurs at the largest breakdown scale if there are multiple subsequent transitions. This might be the case when spin and valley degrees of freedom exhibit different ordering transitions at two distinct energy scales. If, in this scenario, the spin sector orders at the larger of the two energy scales, we cannot directly determine the ground-state order of the valley sector from the flow of the valley–valley correlations. Instead, we need to fall back to, for instance, mean-field arguments as proposed in [21] to determine the most likely valley order. If, on the other hand, the correlations show no flow breakdown, both spin and valley degrees of freedom do not order, indicative of a ground state that remains paramagnetic or exhibits spin–valley liquid behavior.

These two scenarios are illustrated in Fig. 2a, b. Both panels show the flow of the structure factor at the dominant momentum for a magnetically ordered phase with dominant valley order (a) and the paramagnetic state at the SU(4) point [36] (b) where the spin–valley Hamiltonian corresponds to Eq. (1). In the magnetically ordered phase of panel (a), we see a clear flow breakdown in the valley structure factor \(\chi ^{\varLambda v}\), which manifests as a peak or divergence, depending on the vertex truncation length L (further discussed in Sect. 3.3). The spin structure factor \(\chi ^{\varLambda s}\) shown by the purple lines is strongly suppressed. At the SU(4) point, on the other hand, the flow of the structure factor is smooth and convex down to the lowest energy scale we can reliable calculate (\(\varLambda = 0.02 J\)), indicating a paramagnetic ground state. Here, spin and valley correlations are identical due to the global SU(4) symmetry of the Hamiltonian (and indistinguishable in our plot).

3.2 Vertex parametrization and symmetry constraints

To make the solution of the flow equations numerically feasible, one needs to keep the overall number of differential equations needed to capture the flow equations as small as possible. Practically, this can be achieved by eliminating redundant calculations through implementing the symmetry constraints which the Hamiltonian poses on the self-energy and the two-particle vertex. A comprehensive symmetry analysis of this sort has been carried out for generic \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models [27], which here will be generalized to the spin–valley Hamiltonians of interest. Details of this symmetry analysis will be discussed in Sect. 4, while we will report its main findings in the following.

The first important finding is that symmetries dictate that the self-energy is completely diagonal and can be parametrized by a single function \(\varSigma (\omega )\) as

$$\begin{aligned} \varSigma (1', 1) = \varSigma (\omega ) \delta _{s' s} \delta _{l' l} \delta _{i' i} \delta _{\omega ' \omega }. \end{aligned}$$
(15)

We emphasize again that this is only the case if the interactions remain diagonal in either the spin or valley sector. For Hamiltonians with off-diagonal interactions in both sectors, the self-energy will not be diagonal in the spin and valley indices, greatly increasing the numerical cost for the solution of the flow equations. The two-particle vertex can be parametrized as

$$\begin{aligned}&\varGamma (1', 2', 1, 2) \nonumber \\&\quad = \Big [ \varGamma ^{\mu \kappa \lambda }_{i_1 i_2}(s, t, u)\ \theta _{s_{1'}s_1}^\mu \theta _{s_{2'}s_2}^\mu \theta _{l_{1'}l_1}^\kappa \theta _{l_{2'}l_2}^\lambda \ \delta _{i_{1'}i_1} \delta _{i_{2'}i_2} \nonumber \\&\qquad - (1' \leftrightarrow 2') \Big ]\ \delta _{\omega _{1'} + \omega _{2'} - \omega _1 - \omega _{2}}, \end{aligned}$$
(16)

with the three bosonic transfer frequencies

$$\begin{aligned} \begin{aligned} s&=\omega _{1'}+\omega _{2'} \\ t&=\omega _{1'}-\omega _{1} \\ u&=\omega _{1'}-\omega _{2}. \end{aligned} \end{aligned}$$
(17)

This parametrization is of the same form as for \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models—apart from an increased number of components due to the valley sector \(\sim \theta _{l_{1'}l_1}^\kappa \theta _{l_{2'}l_2}^\lambda \) with the corresponding indices \(\kappa \) and \(\lambda \). If we assume the Hamiltonian to be diagonal in the spin sector, we will only need to consider components diagonal in the spin \(\sim \theta _{s_{1'}s_1}^\mu \theta _{s_{2'}s_2}^\mu \), with the corresponding index \(\mu \) (and vice versa for a system with a diagonal valley Hamiltonian). The basis functions of the parametrization are constrained by the symmetries of the Hamiltonian as

$$\begin{aligned} \varSigma (\omega )&\in i{\mathbb {R}} \nonumber \\ \varSigma (\omega )&= -\varSigma (-\omega ) \end{aligned}$$
(18)
$$\begin{aligned} \varGamma ^{\mu \kappa \lambda }_{i_1 i_2}(s, t, u)&\in {\left\{ \begin{array}{ll} {\mathbb {R}} &{} \text {if } \xi (\kappa ) \xi (\lambda ) = 1\\ i{\mathbb {R}} &{} \text {if } \xi (\kappa ) \xi (\lambda ) = -1\\ \end{array}\right. }\nonumber \\ \varGamma ^{\mu \kappa \lambda }_{i_1 i_2}(s, t, u)&= \varGamma ^{\mu \lambda \kappa }_{i_2 i_1}(-s, t, u)\nonumber \\ \varGamma ^{\mu \kappa \lambda }_{i_1 i_2}(s, t, u)&= \xi (\kappa ) \xi (\lambda ) \varGamma ^{\mu \kappa \lambda }_{i_1 i_2}(s, -t, u)\nonumber \\ \varGamma ^{\mu \kappa \lambda }_{i_1 i_2}(s, t, u)&= \xi (\kappa ) \xi (\lambda ) \varGamma ^{\mu \lambda \kappa }_{i_2 i_1}(s, t, -u), \end{aligned}$$
(19)

where we defined the sign function

$$\begin{aligned} \xi (\kappa ) = {\left\{ \begin{array}{ll} \ 1 &{} \text {if } \kappa = 0\\ -1 &{} \text {otherwise} \end{array}\right. } \ . \end{aligned}$$
(20)

These are the same relations as for the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case, apart from a missing constraint relating the s und u frequencies in the two-particle vertex (c.f. Eq. (14) in Ref. [27]). This is a consequence of the Hamiltonian only being invariant under a global particle–hole symmetry instead of the local particle–hole symmetry under which the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) Hamiltonian is invariant. We discuss this in more detail in Sect. 4. The missing relation, however, does not change the key implications of the constraints, namely that the basis functions are either completely real or imaginary, and that values of the vertex functions at negative transfer frequencies can be inferred from the positive frequency axes.

The parametrization of the two-particle vertex using the three transfer frequencies in Eq. (17) is convenient for deriving the flow equations and symmetry constraints. However, to better capture the asymptotic frequency dependence of the two-particle vertex one can further refine the frequency parametrization [23, 24, 37]. The first step is to group the contributions in the flow equation of the two-particle vertex given in Eq. (11) into three channels according to their two-particle irreducibility. This results in a particle–particle (pp), direct particle–hole (dph), and crossed particle–hole (cph) channel, which correspond to the three contributions on the right-hand side (RHS) of Eq. (11), in the respective ordering. In these terms, the flow equation for the two-particle vertex can be written as

$$\begin{aligned} \frac{d}{d \varLambda } \varGamma ^{\varLambda } ={\dot{g}}_{p p}^{\varLambda } +{\dot{g}}_{d p h}^{\varLambda } +{\dot{g}}_{c p h}^{\varLambda }, \end{aligned}$$
(21)

and the vertex is parametrized (stating only the frequency dependence) as

$$\begin{aligned} \varGamma ^\varLambda (s, t, u) = \varGamma ^{\varLambda \rightarrow \infty } + \sum _c g_c^\varLambda (\omega _c, v_c, v'_c), \end{aligned}$$
(22)

where \(\varGamma ^{\varLambda \rightarrow \infty }\) is the bare two-particle vertex at infinite cut-off. Each channel \(g_c(\omega _c, v_c, v^\prime _c)\) is parametrized by one bosonic transfer frequency \(\omega _c\) and two additional fermionic frequencies \(v^{{\prime }}_c, v^\prime _c\). The precise definition of the frequencies can be chosen in numerous ways. It is, however, advantageous to choose them, so that the symmetry constraints of the two-particle vertex given in Eq. (19) result in equally simple relations for each channel in the new parametrization. Here, we adapt the choice of Ref. [24]

$$\begin{aligned} \begin{aligned} \omega _{pp}&= s&v_{pp}&= \omega _1 - \frac{s}{2}&v'_{pp}&= \frac{s}{2} - \omega _{1'} \\ \omega _{dph}&= t&v_{dph}&= \omega _1 + \frac{t}{2}&v'_{dph}&= \omega _{1'} - \frac{t}{2} \\ \omega _{cph}&= u&v_{cph}&= \omega _1 - \frac{u}{2}&v'_{cph}&= \omega _{1'} - \frac{u}{2}, \end{aligned} \end{aligned}$$
(23)

and give the resulting symmetry constraints for the channels in A. Compared to \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models, no constraint relating the particle–particle and crossed particle–hole channel with each other is present, which can be traced back to the missing symmetry constraint relating the s and u frequency dependenceFootnote 3.

To complete the discussion, we still need to state the initial conditions of the flow equations corresponding to the self-energy and two-particle vertex in the limit \(\varLambda \rightarrow \infty \), which are given by

$$\begin{aligned} \begin{aligned} \varSigma ^{\varLambda \rightarrow \infty }(\omega )&= 0 \\ \varGamma _{i_1i_2}^{\varLambda \rightarrow \infty \mu \kappa \lambda }(s, t, u)&= \frac{1}{8}J_{s,i_1i_2}^\mu J_{v, i_1i_2}^{\kappa \lambda }, \end{aligned} \end{aligned}$$
(24)

with the couplings \(J^\mu _{s,i_1i_2}\) and \(J^{\kappa \lambda }_{i_1i_2}\) defined in Eq. (5).

3.3 Numerical implementation

The numerical solution of the pf-FRG flow equations poses several challenges and necessitates further approximations to be made. To overcome these challenges, we employ the state-of-the-art numerical implementation of Refs. [23, 24], where additional details of the implementation are discussed. Here, we only give a short overview and discuss some slight technical differences in the implementation for spin–valley models.

First, one has to truncate the infinite lattice geometry by a finite lattice graph. Employing the symmetries of the lattice geometry for which the spin–valley model is formulated and the local U(1) symmetry present in all pseudo-fermion Hamiltonians, the spatial dependence of the two-particle vertex can be reduced to just one site index j and one arbitrary fixed reference site \(i_0\), as will be derived in Sect. 4. To obtain a finite number of vertex components \(\varGamma ^\varLambda _{i_0 j}\) (considering only the lattice site dependence), we define a finite length scale L and truncate the vertex \(\varGamma ^\varLambda _{i_0, j}\) for bond distances \(d(i_0, j) > L\), effectively enforcing a maximal correlation length. The finite-size effect of this truncation can be observed in Fig. 2, where several calculations with increasing values of L were performed for a magnetically ordered and a paramagnetic phase. In the ordered phase, the flow breakdown sharpens from a relatively broad peak for low values of L to a clear divergence for larger values of L, which is a typical observation. The paramagnetic phase is, in contrast, not affected by the increase of L (at least qualitatively). From an algorithmic point of view, the asymptotic scaling of the computation time is quadratic in the number of lattice points \(N_L \sim L^d\), where d is the number of spatial dimensions. This is due to the fact that the number of vertex components as well as the sum over all lattice sites included in the flow equations scale linearly with \(N_L\). In this work, we typically perform calculations at \(L=9\), above which the breakdown scale does not significantly change anymore and the numerical effort is still reasonable.

Since the pf-FRG approach is formulated at zero temperature, another point we need to address is how to discretize the continuous Matsubara frequencies. To accurately resolve all features of the two-particle vertex, it turns out that particular care needs to be taken in the choice of frequency meshes [23, 24]. To this end, the frequencies are discretized on adaptive, hybrid linear-logarithmic meshes, which are updated using a scanning routine between each step of the ordinary differential equation (ODE) solver. In addition to continuous Matsubara frequencies, the flow equations at \(T = 0\) include frequency integrals which have to be performed numerically. To calculate these integrals, we employ an adaptive quadrature which takes both the relevant features around the origin and the algebraic decay for large frequencies into account. Values of the vertex for frequencies not lying on the discrete frequency meshes are obtained by multi-linear interpolation. The computation time asymptotically scales with the number of (positive) bosonic frequencies \(N_\varOmega \) and (positive) fermionic frequencies \(N_\nu \) as \({\mathcal {O}}(N_\varOmega \cdot N_\nu ^2)\). A typical setup for which the two-particle vertex is sufficiently well resolved is \(N_\varOmega = 40\) and \(N_\nu =30\), which we use for all calculations in this work. The computational effort to compute the self-energy is, compared to the vertex, negligible, as it only depends on one frequency. Here, we choose a frequency mesh with \(N_\varSigma = 250\) frequencies. In the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case, only positive frequencies were required, as the symmetry constraints map all negative frequency components to some positive counterpart. For spin–valley models, however, due to the missing symmetry constraint relating the particle–particle and crossed particle–hole channel (discussed in Sect. 3.2), we have to also consider negative frequencies for either \(\nu _c\) or \(\nu '_c\). This results in an additional factor of two in computation time compared to \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models.

The adaptive frequency meshes and integration routine allow for an efficient evaluation of the RHS of the flow equations. For the solution of the ODEs themselves, we choose the Bogacki–Shampine method [38], which is a third-order Runge–Kutta method with adaptive step size control. We find that this method is a good compromise between computational cost and numerical precision.

Although the asymptotic scaling of the computation time with the number of lattice points and frequencies is the same as for the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case, more complex spin–valley models usually require a much larger numerical effort, as the extra valley index greatly increases the number of independent two-particle vertex components \(\varGamma ^{\varLambda , \mu \kappa \lambda }_{i_1i_2}\), in which the computation time scales linearly. With the coupling matrices given in Eq. (5), there would be \(N_\varGamma = 4 \cdot 4^2 = 64\) independent vertex components (only considering the spin–valley dependence). In comparison, the parametrization for generic \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) models only has \(N_\varGamma = 4^2 = 16\) components. Fortunately, in almost all physical models, extra symmetries in the spin and valley space will greatly reduce the number of independent components. Considering, e.g., an SU(2) symmetry in the spin space and a U(1) symmetry in valley space, which is present in several models for moiré materials [11, 35], the number is already reduced to \(N_\varGamma = 2 \cdot 6 = 12\). For these models, the numerical effort is similar to \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) models with off-diagonal interactions and even allows for computations of relatively large-phase diagrams as will be presented in Sect. 5.

4 Symmetry classification

To proof the validity of the parametrization and the symmetry constraints presented in the previous section, we repeat the symmetry analysis of Ref. [27], where the pseudo-fermion Hamitonian for \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models with generic diagonal and off-diagonal interactions is considered, but for the spin–valley Hamiltonian given in Eq. (4). We show that most of the symmetries of the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) pseudo-fermion Hamiltonian are either also present in the spin–valley Hamiltonian, or can be generalized in a straightforward fashion. There are, however, some differences that we will highlight in the following. Most notably, we show that, even at the SU(4) point, the spin–valley model does not posses a local particle–hole symmetry that is present in the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case, but only the corresponding global symmetry. Consequently, it is also not present in generalizations of the SU(2) Heisenberg model to SU(N), which might not have been clearly stated before. This is the reason for the missing symmetry constraint for the two-particle vertex as presented in the previous section.

4.1 Local U(1) symmetry

The first symmetry transformation we consider, a local U(1) transformation, directly follows from the form of the spin–valley operator given by Eq. (2). It acts on the fermionic Hilbert space at site i by multiplying a local phase \(\varphi _i \in [0, 2\pi )\) to the fermionic operators as

$$\begin{aligned} g_{\varphi _i} \begin{pmatrix} f^\dagger _{isl} \\ f^{{\dagger }}_{isl} \end{pmatrix} g^{-1}_{\varphi _i} = \begin{pmatrix} e^{i\varphi _i} f^\dagger _{isl} \\ e^{-i\varphi _i} f^{{\dagger }}_{isl} \end{pmatrix}, \end{aligned}$$
(25)

which clearly leaves all spin–valley operators invariant. Interpreting the spin–valley Hamiltonian as a fermionic representation of an \({{\mathfrak {s}}}{{\mathfrak {u}}}(4)\) spin model, it is simply a consequence of the choice for the fermionic representation of the spin operators. It is therefore also present in all conventional pf-FRG implementations using the standard pseudo-fermion representation. In that sense, it is sometimes also referred to as a gauge redundancy instead of a symmetry, as it is not a symmetry of the original spin Hamiltonian, but only of the pseudo-fermion representation. For our functional renormalization group approach, we are interested in the implication of the symmetry on the functional formFootnote 4 of the one-particle correlation function

$$\begin{aligned} \begin{aligned} G(1', 1)&\equiv - \langle f^{\dagger }_{1'}f^{\dagger }_{1}\rangle \\&= - \int d \tau ' d \tau e^{i \tau ' \omega '-i \tau \omega }\left\langle f_{i' \tau ' s' l'}^{{\dagger }} f^{\dagger }_{i \tau s l}\right\rangle \end{aligned}\nonumber \\ \end{aligned}$$
(26)

and the two-particle correlation function

$$\begin{aligned}&G(1', 2', 1, 2) := \langle f^{\dagger }_{1'} f^{\dagger }_{2'} f^{\dagger }_{2} f^{\dagger }_{1} \rangle \nonumber \\&\quad = \int d \tau _{1'} d \tau _{2'} d \tau _{1} d \tau _{2} e^{i(\tau _{1'} \omega _{1'}+\tau _{2'} \omega _{2'}-\tau _{1} \omega _{1}-\tau _{2} \omega _{2})}\nonumber \\&\qquad \times \left\langle f_{i_{1'} \tau _{1'} s_{1'} l_{1'}}^{{\dagger }} f_{i_{2'} \tau _{2'} s_{2'} l_{2'}}^{{\dagger }} f^{\dagger }_{i_{2} \tau _{2} s_2 l_2} f^{\dagger }_{i_{1} \tau _{1} s_1 l_1}\right\rangle ,\nonumber \\ \end{aligned}$$
(27)

where we suppress the time-ordering operator as it becomes trivial in the path integral framework that the function renormalization group is formulated in. Acting with the local U(1) transformation given in Eq. (25) on the definition of the correlation functions and demanding their invariance leads to the corresponding symmetry constraint. It directly implies that we can restrict ourselves to a local one-particle correlation function

$$\begin{aligned} G(1', 1) = G(1', 1) \delta _{i_{1'}i_1}, \end{aligned}$$
(28)

which only depends on one lattice site \(i_1\), and a bi-local two-particle correlation function

$$\begin{aligned} G(1', 2', 1, 2) =\&G(1', 2', 1, 2) \delta _{i_{1'}i_1} \delta _{i_{2'}i_2}\nonumber \\ -\&G(2', 1', 1, 2) \delta _{i_{2'}i_1} \delta _{i_{1'}i_2}, \end{aligned}$$
(29)

which only depends on the two lattices sites \(i_1\) and \(i_2\).

4.2 Global particle–hole symmetry

In the pf-FRG for \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models, spin operators \(S_i^a\) are represented using fermions with one spin index \(\alpha = \pm 1\) as

$$\begin{aligned} S^a = \frac{1}{2}f^\dagger _{i \alpha } \theta ^a_{\alpha \alpha '} f^{{\dagger }}_{i \alpha '}, \end{aligned}$$
(30)

with \(a \in \{1, 2, 3\}\). In addition to the U(1) gauge redundancy, there exists another redundancy in this representation that can be formulated as a local particle–hole symmetry [27]. It acts on the fermionic Hilbert space as

$$\begin{aligned} g_i \begin{pmatrix} f_{i \alpha }^{\dagger } \\ f_{i \alpha }^{{\dagger }} \end{pmatrix} g_i^{-1}= \begin{pmatrix} \alpha f_{i {\bar{\alpha }}}^{{\dagger }} \\ \alpha f_{i {\bar{\alpha }}}^{\dagger } \end{pmatrix}, \end{aligned}$$
(31)

with \({\bar{\alpha }}\equiv -\alpha \). It leaves the fermionic representation of the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin operators invariant and is therefore a symmetry of the pseudo-fermion Hamiltonian. We note that this symmetry is not anti-unitary and therefore does not correspond to the usual physical particle–hole symmetry [27]. Instead, it is again a consequence of the representation of the spin operators. The natural extension for spin–valley models with spin index \(s = \pm 1\) and valley index \(l = \pm 1\) is the transformation

$$\begin{aligned} g_i \begin{pmatrix} f^\dagger _{isl} \\ f_{isl}^{{\dagger }} \end{pmatrix} g_i^{-1} = \begin{pmatrix} slf_{i{\bar{s}}{\bar{l}}}^{{\dagger }}\\ slf^\dagger _{i{\bar{s}}{\bar{l}}} \end{pmatrix}, \end{aligned}$$
(32)

under which the spin–valley operator transforms as

$$\begin{aligned} g_i\ \sigma _i^\mu \otimes \tau _i^\kappa \ g_i^{-1} = - \xi (\mu ) \xi (\kappa ) \sigma _i^\mu \otimes \tau _i^\kappa , \end{aligned}$$
(33)

which can be shown straightforwardly using the anti-commutation relations of the fermionic operators and the identity

$$\begin{aligned} {{\bar{\alpha }}} {{\bar{\alpha }}} ' \theta ^\mu _{\alpha \alpha '}= \xi (\mu ) \theta ^\mu _{{{\bar{\alpha }}} ' {{\bar{\alpha }}}}. \end{aligned}$$
(34)

Spin–valley operators with either the spin index \(\mu \) or the valley index \(\kappa \) set to zero—which correspond to the individual spin and valley operators as defined in Eq. (2)—are invariant under this transformation. General spin–valley operators, on the other hand, may change their sign. The Hamiltonian is, therefore, not invariant under the local particle–hole symmetry that acts on the Hilbert space of just one lattice site. Spin–valley operators, however, only appear in pairs in the spin–valley Hamiltonian. Performing the local particle–hole symmetry transformation on all lattice sites, such a pair of spin–valley operators transform as

$$\begin{aligned} g (\sigma _i^\mu \otimes \tau _i^\kappa ) (\sigma _j^\nu \otimes \sigma _j^\lambda ) g^{-1}= & {} \xi (\mu )\xi (\kappa )\xi (\nu )\xi (\lambda ) \nonumber \\&\times (\sigma _i^\mu \otimes \tau _i^\kappa ) (\sigma _j^\nu \otimes \sigma _j^\lambda ).\nonumber \\ \end{aligned}$$
(35)

If an odd number of spin and valley indices is set to zero, this again implies a sign change. Recalling the definition of the spin–valley Hamiltonian in Eq. (4) and the following definition of the exchange matrices in Eq. (5), such terms are not included in our definition of the Hamiltonian. All terms that do appear in the Hamiltonian are indeed invariant. The main difference to the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) pseudo-fermion Hamiltonian is, therefore, that the spin–valley Hamiltonian is invariant only under the global transformation, while the former was invariant under the local transformation. For the local single-particle correlation function, the global particle–hole symmetry implies

$$\begin{aligned} G(1', 1) \delta _{i' i}= -s s' l l' G(i-\omega {\bar{s}}{\bar{l}}, i -\omega ' {\bar{s}}'{\bar{l}}') \delta _{i' i}, \end{aligned}$$
(36)

and for the bi-local two-particle correlator, it implies

$$\begin{aligned}&G(1', 2', 1, 2) \delta _{i_{1'}i_1} \delta _{i_{2'}i_2} \nonumber \\&\quad =s_{1'}s_1l_{1'}l_1 s_{2'}s_2l_{2'}l_2\delta _{i_{1'}i_1} \delta _{i_{2'}i_2}G(i_1-\omega _1 {\bar{s}}_1{\bar{l}}_1, i_2\nonumber \\&\qquad - \omega _2 {\bar{s}}_2{\bar{l}}_2, i_1-\omega _{1'} {\bar{s}}_{1'}{\bar{l}}_{1'}, i_2 -\omega _{2'} {\bar{s}}_{2'}{\bar{l}}_{2'}). \end{aligned}$$
(37)

These relations are, apart form the extra factors of valley indices, the same as for the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case when considering the global transformation. The invariance under the local transformation would yield additional constraints on the two-particle correlator acting only on multi-indices with the same lattice site (\(i_1\) or \(i_2\)). For the parametrized two-particle vertex, these result in a constraint relating the s and u dependence or, in the asymptotic frequency parametrization defined in Eqs. (22, 23), the particle–particle and crossed particle–hole channel with each other. As already discussed in Sect. 3, this constraint is, consequently, missing for spin–valley models.

4.3 Generalized time-reversal symmetry

For \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models, a genuinely physical symmetry is the invariance under time-reversal. In this setting, time-reversal reverses the sign of all spin operators \(S^a \rightarrow - S^a\) and, as it is an anti-unitary symmetry, additionally applies complex conjugation to all complex numbers. Hamiltonians with real couplings in which spin operators only appear in pairs are therefore always invariant under time-reversal. On the Hilbert space of the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) pseudo-fermions, it can be represented as

$$\begin{aligned} g\begin{pmatrix} f_{i \alpha }^{\dagger } \\ f_{i \alpha }^{{\dagger }} \end{pmatrix} g^{-1}=\begin{pmatrix} e^{i \pi \alpha / 2} f_{i {\bar{\alpha }}}^{\dagger } \\ e^{-i \pi \alpha / 2} f_{i {\bar{\alpha }}}^{{\dagger }} \end{pmatrix}. \end{aligned}$$
(38)

We again consider a straightforward generalization of the transformation to spin–valley operators, which we define as the anti-unitary transformation

$$\begin{aligned} g \begin{pmatrix} f^\dagger _{isl} \\ f_{isl}^{{\dagger }} \end{pmatrix} g^{-1} = \begin{pmatrix} e^{i\pi s/2}e^{i\pi l/2} f^\dagger _{i{\bar{s}}{\bar{l}}} \\ e^{-i\pi s/2}e^{-i\pi l/2} f^{{\dagger }}_{i {\bar{s}}{\bar{l}}} \end{pmatrix}. \end{aligned}$$
(39)

Using the relation \(e^{i\pi (\alpha - \alpha ')/2} = \alpha \alpha '\) and Eq. (34), it is straightforward to show that the spin–valley operator transforms as

$$\begin{aligned} g\ \sigma _i^\mu \otimes \tau _i^\kappa \ g^{-1} = \xi (\mu ) \xi (\kappa ) \sigma _i^\mu \otimes \tau _i^\kappa , \end{aligned}$$
(40)

which, up to a minus sign, is the same transformation behavior as for the particle–hole symmetry in Eq. (33). As only pairs of spin–valley operators appear in the spin–valley Hamiltonian, for which the minus sign is irrelevant, the arguments for the invariance of Hamiltonian given there, consequently, also apply here. Applying this generalized version of time-reversal to the local one-particle correlator implies

$$\begin{aligned} G(1', 1) \delta _{i', i} = s s' l l' G(i-\omega '{\bar{s}}'{\bar{l}}', i -\omega {\bar{s}}{\bar{l}})^*\delta _{i', i}, \end{aligned}$$
(41)

where the complex conjugation stems from the fact that the transformation is anti-unitary. For the bi-local two-particle correlation function, it implies

$$\begin{aligned}&G(1', 2', 1, 2) \delta _{i_{1'}i_1} \delta _{i_{2'}i_2}\nonumber \\&\quad = s_{1'}s_1l_{1'}l_1 s_{2'}s_2l_{2'}l_2 \delta _{i_{1'}i_1} \delta _{i_{2'}i_2} G(i_1-\omega _{1'}{\bar{s}}_{1'}{\bar{l}}_{1'},\nonumber \\&\qquad \times i_2 -\omega _{2'} {\bar{s}}_{2'}{\bar{l}}_{2'}, i_1 -\omega _1 {\bar{s}}_1{\bar{l}}_1, i_2 -\omega _2 {\bar{s}}_2{\bar{l}}_2)^*. \end{aligned}$$
(42)

Apart from extra valley indices, this is exactly the same as in the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case.

4.4 Hermitian symmetry

Just as the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin operator, the spin–valley operator is Hermitian. The spin–valley Hamiltonian only consists of pairs of spin–valley operators and we have restricted ourselves to real couplings, making it Hermitian aswell. Complex transposition, therefore, leaves the Boltzman factor in the thermal expectation value invariant. Applying complex transposition on both sides of Eqs. (26, 27) and explicitly evaluating the RHS by “pulling” the complex transpose into the thermal expectation value, we obtain the constraint

$$\begin{aligned} G(1', 1)\delta _{i', i} = G(i -\omega s l, i - \omega ' s' l')^* \delta _{i', i} \end{aligned}$$
(43)

for the local one-particle correlator and

$$\begin{aligned} \begin{aligned} G(1', 2', 1, 2) \delta _{i_{1'}i_1} \delta _{i_{2'}i_2} = \delta _{i_{1'}i_1} \delta _{i_{2'}i_2} \\ \times G(i_{1} -\omega _1 s_1 l_2, i_2 -\omega _2 s_2 l_2,\\ i_1 - \omega _{1'} s_{1'} l_{1'}, i_2 -\omega _{2'} s_{2'} l_{2'})^* \end{aligned}\nonumber \\ \end{aligned}$$
(44)

for the two-particle correlator. These constraints are again of the same form as for the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case.

4.5 Lattice symmetries

The spin models we consider are all formulated on lattices that can be specified in terms of an underlying Bravais lattice and a possibly multi-atomic basis. Therefore, lattice symmetries exist necessarily for any spin–valley model and are very important to efficiently implement the pf-FRG. Their implementation is the same whether one considers \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models or spin–valley models. We can therefore use the same approach as for the conventional pf-FRG as, e.g., explained in Ref. [27]. There, all sites are assumed to be identical, in the sense that one can map any site to any other site via a lattice automorphism T that leaves the lattice itself invariant. On the fermionic operators, such a transformation acts as

$$\begin{aligned} g_{T}\begin{pmatrix} f_{i sl}^{\dagger } \\ f_{i sl}^{{\dagger }} \end{pmatrix} g_{T}^{-1} =\begin{pmatrix} f_{T(i) sl}^{\dagger } \\ f_{T(i) sl}^{{\dagger }} \end{pmatrix}. \end{aligned}$$
(45)

In the case of bond-directional couplings, the transformation would additionally have to be combined with transformations in spin and valley space. For the one-particle correlation function, this implies

$$\begin{aligned} G\left( 1' , 1\right) \delta _{i', i}=G\left( T\left( i\right) \omega ' s'l', T(i) \omega s l \right) \delta _{i', i}. \end{aligned}$$
(46)

The locality constraint in Eq. (28), resulting from the local U(1) symmetry, already reduces the spatial dependence of the self-energy to only one site index \(i_1\). Using lattice automorphisms, we can map all sites to an arbitrary reference site \(i_0\) and therefore completely remove the spatial dependence of the one-particle correlation function. Similarly, for the two-particle correlation function, it implies

$$\begin{aligned}&G\left( 1', 2' , 1,2\right) \delta _{i_{1'}i_1} \delta _{i_{2'}i_2} = \delta _{i_{1'}i_1} \delta _{i_{2'}i_2} \nonumber \\&\quad \times G\big (T\left( i_{1}\right) \omega _{1'} s_{1'} l_{1'}, T\left( i_{2}\right) \omega _{2'} s_{2'}l_{2'},\nonumber \\&\quad T\left( i_{1}\right) \omega _{1} s_1l_1, T\left( i_{2}\right) \omega _{2} s_2 l_2 \big ). \end{aligned}$$
(47)

Combining this with the bi-locality constraint in Eq. (29), and again mapping the first index \(i_1\) to an arbitrary reference site \(i_0\), the spatial dependence of the two-particle correlator can be reduced to just one lattice site.

4.6 Parametrization of correlation functions

To make use of the symmetry constraints on the correlation functions, it is advantageous to parametrize them, so that the symmetry constraints manifest in a more practical form. To this end, we can extent the parametrization for the correlation functions for generic \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) spin models introduced in [27] also to spin–valley models. This ultimately leads to the parametrization of the self-energy and two-particle vertex in Eqs. (15, 16) and the symmetry constraints in Eqs. (18, 19). Starting with the one-particle correlation function, we argued that due to the local U(1) symmetry and lattice symmetries, it is independent of the lattice site. Additionally, due to Matsubara frequency conservation, which is a consequence of translational invariance in imaginary time, it is diagonal in the frequency arguments. Expanding the spin and valley dependence in Pauli matrices \(\theta ^\mu \theta ^\kappa \) (\(\mu ,\kappa = 0, 1, 2, 3\)), the one-particle correlation function can be parametrized as

$$\begin{aligned} G(1', 1) = G^{\mu \kappa }(w) \theta _{s' s}^\mu \theta _{l' l}^\kappa \delta _{i'i} \delta _{\omega ' \omega }. \end{aligned}$$
(48)

Similarly, the two-particle correlation function depends only on two lattice sites and three frequencies, for which we choose the three transfer frequencies defined in Eq. (17). The parametrization then reads

$$\begin{aligned}&G(1', 2', 1, 2)\nonumber \\&= \Big (G^{\mu \nu \kappa \lambda }_{i_1 i_2}(s, t, u) \theta _{s_{1'}s_1}^\mu \theta _{s_{2'}s_2}^\nu \theta _{l_{1'}l_1}^\kappa \theta _{l_{2'}l_2}^\lambda \delta _{i_{1'}i_1} \delta _{i_{2'}i_2}\nonumber \\&\quad - (1' \leftrightarrow 2')\Big ) \delta _{\omega _{1'} + \omega _{2'}-\omega _1 -\omega _{2}}. \end{aligned}$$
(49)

Plugging this parametrization into the symmetry constraints derived in Sects. 4.14.5, we obtain the symmetry constraints for the basis functions of the parametrization listed in Table 1. In the derivation of these constraints, we make heavy use of Eq. (34) and the particle-exchange symmetry

$$\begin{aligned} G(1', 2', 1, 2) = G(2', 1', 2, 1), \end{aligned}$$
(50)

which is present in all purely fermionic models.

Table 1 Symmetry constraints for the basis functions of the parametrization of the correlation functions

The list of symmetry constraints is very similar to the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case derived in Ref.  [27], but has two significant differences. First, as already discussed in Sects. 3.2 and 4.2, the symmetry constraint relating s and u frequencies, or the particle-particle and crossed particle–hole channel, is missing because the spin–valley Hamiltonian is not invariant under a local particle–hole transformation but only under the global version. Second, the symmetry constraints do not imply that the one-particle correlation function is completely diagonal in all spin and valley indices. In the parametrization, this would manifest in \(G^{00}\) being the only non-vanishing basis function. Instead, for a general spin–valley Hamiltonian, also the terms \(G^{ab}\) with \(a, b > 0\), which come with the factor \(\sim \theta ^a_{ss'}\theta ^b_{ll'}\), are allowed. This would increase the number of flow equations and therefore also the numerical complexity significantly. Additionally, we could not use the conventional pf-FRG flow equations given in Eqs. (10, 11), where a diagonal one particle correlator (and self-energy) was assumed. Fortunately, in the context of moiré materials, many Hamiltonians of physical relevance posses additional symmetries in the spin and valley space [11, 35] that further constrain the spin and valley dependence of the self-energy and two-particle vertex. It turns out that the minimal symmetry needed in order for the one-particle correlator to be diagonal is a \({\mathbb {Z}}_2 \times {\mathbb {Z}}_2 \times {\mathbb {Z}}_2\) symmetry in either the spin or valley sector. On the level of spin–valley operators, this means that the Hamiltonian is invariant under the transformation (for the case of the spin sector)

$$\begin{aligned} g_\mu \ \sigma _i^\mu \otimes \tau _i^\kappa \ g_\mu ^{-1} = \xi (\mu ) \sigma _i^\mu \otimes \tau _i^\kappa , \end{aligned}$$
(51)

for each \(\mu \) individually. This simply reverses the signs of all \(\sigma ^\mu _i \otimes \tau ^\kappa _i\) with \(\mu > 0\). Assuming a completely diagonal spin exchange matrix as in Eq. (5), the spin–valley Hamiltonian is indeed invariant under this transformation. This directly implies that all terms proportional to a single \(\sim \theta ^\mu \) (with \(\mu >0\)) in the correlation functions have to vanish. More precisely, it imposes the constraint

$$\begin{aligned} G^{\mu \kappa }(\omega ) = \delta _{\mu 0} G^{0\kappa }(\omega ), \end{aligned}$$
(52)

which in combination with the first equation in Table 1 implies

$$\begin{aligned} G^{\mu \kappa }(\omega ) = \delta _{\mu 0}\delta _{\kappa 0} G^{00}(\omega ) \equiv \delta _{\mu 0}\delta _{\kappa 0} G(\omega ), \end{aligned}$$
(53)

resulting in a completely diagonal one-particle correlation function parametrized by a single basis function \(G(\omega )\). For the coupling matrices stated in Eq. (5), we can therefore use the standard pf-FRG approach also for spin–valley models. Assuming this additional symmetry, in the two-particle correlator, only diagonal components in the spin sector \(\sim \theta ^\mu \theta ^\mu \) (no sum over \(\mu \)) are allowed, resulting in the constraint

$$\begin{aligned} G^{\mu \nu \kappa \lambda }_{i_1 i_2}(s, t, u) = \delta _{\mu \nu } G^{\mu \mu \kappa \lambda }_{i_1 i_2}(s, t, u) \equiv \delta _{\mu \nu } G^{\mu \kappa \lambda }_{i_1 i_2}(s, t, u).\nonumber \\ \end{aligned}$$
(54)

Imposing these additional constraints, all factors of \(\xi (\mu )\xi (\nu )\) in Table 1 are equal to one and the relations reduce exactly to the constraints given in Eqs. (18, 19) with the self-energy and two-particle vertex replaced by the one- and two-particle correlation functions. We can therefore still consider a completely imaginary one-particle correlator that is odd in frequency space and completely diagonal. The two-particle correlator is either completely real or imaginary, depending on the sign of \(\xi (\kappa , \lambda )\), and all negative frequency components can be mapped to a positive counterpart.

The argument why these constraints on the disconnected correlation functions carry over to the one-particle irreducible correlation functions, i.e., the self-energy and the vertex, is the same as given for the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) case in [27]. For the self-energy, it simply follows from the Dyson equation [30]:

$$\begin{aligned} G(1', 1) = \frac{1}{i\omega - \varSigma (1', 1)}, \end{aligned}$$
(55)

from which it is easy to see that all constraints carry over to the self-energy. For the two-particle vertex, the tree expansion (neglecting the three-particle vertex) relates it to the connected two-particle correlation function \(G^{(c)}\) as [30]

$$\begin{aligned}&G^{(c)}\left( 1', 2' , 1,2\right) \nonumber \\&\quad = -\sum _{3,4,5,6} \varGamma (3,4 , 5,6) G\left( 1' , 3\right) G\left( 2' , 4\right) G(5 , 1) G(6 , 2).\nonumber \\ \end{aligned}$$
(56)

As the one-particle correlation function is diagonal in all indices, it is clear that all constraints carry over from the connected correlation function to the two-particle vertex. That the constraints from the disconnected two-particle correlation function carry over to the connected correlation function can be proven by their definition via generating functionals [30].

4.7 Symmetries of the flow equations

To verify that the parametrization and the symmetry constraints derived in the previous sections are indeed preserved also for the flowing self-energy and two-particle vertex for any value of \(\varLambda \), they can additionally be proven using the pf-FRG flow equations given in Eqs. (10, 11) . That the parametrization for the self-energy in Eq. (15) and for the two-particle vertex in Eq. (16) is indeed complete can be seen by inserting them into the RHS of the flow equations and confirming that no additional terms are generated.

For the additional symmetry constraints, the proof can be performed via induction, as already explained in Refs. [27, 39]. This essentially amounts to verifying the fulfillment of the constraints in the initial conditions and then showing that the derivatives \(\frac{d}{d\varLambda }\varSigma \) and \(\frac{d}{d\varLambda }\varGamma \) given by the RHS of the flow equations also fulfill them, assuming the self-energy and two-particle vertex themselves already do. The proof that the self-energy is odd, imaginary, and completely diagonal has to be repeated for spin–valley models due to slight differences in the flow equations. This is quite lengthy and, therefore, done in B. For the two-particle vertex, the proof of the symmetry constraints is much easier on the level of the unparametrized vertex, as there the flow equations still have a much simpler form. We therefore postulate the relations

$$\begin{aligned} \varGamma ^\varLambda (1', 2', 1, 2)&= \varGamma ^\varLambda (2', 1', 2, 1) \end{aligned}$$
(57)
$$\begin{aligned} \varGamma ^\varLambda (1', 2', 1, 2)&= \varGamma ^\varLambda (1, 2, 1', 2')^* \end{aligned}$$
(58)
$$\begin{aligned} \varGamma ^\varLambda (1', 2', 1, 2)&= \varGamma ^\varLambda (-2', -1', -2, -1) \end{aligned}$$
(59)
$$\begin{aligned} \varGamma ^\varLambda (1', 2', 1, 2)&= s_{1'}s_1l_{1'}l_1 s_{2'}s_2l_{2'}l_2\nonumber \\&\quad \times \varGamma ^\varLambda ({\bar{1}}, {\bar{2}}, {\bar{1}}', {\bar{2}}') , \end{aligned}$$
(60)

where we defined \(-1 = (i_1 -\omega _1 s_1 l_1)\) and \({\bar{1}} = (i_1 \omega _1 {\bar{s}}_1 {\bar{l}}_1)\). When translated to the parametrized two-particle vertex and then combined, these relations yield exactly the symmetry constraints given in Eq. (19). Proving the relations for the unparametrized vertex, therefore, directly proves the symmetry constraints of the parametrized vertex. As Eq. (57) simply amounts to a simple particle exchange, no further proof is required. Eq. (58) is proven in [27] and Eq. (59) in [39] using the general pf-FRG flow equations. The only remaining relation still left to prove is Eq. (60), which we also show in B. This proves that the parametrization and the symmetry constraints are indeed valid also for the flowing self-energy and vertex, at any value of the cut-off \(\varLambda \).

5 Results

To give an explicit example for the application of the pseudo-fermion functional renormalization group approach introduced in the manuscript and its efficient implementation in terms of the aforementioned symmetries, we apply it to elucidate the phase diagram of an SU(2)\(_\mathrm {spin}\) \(\otimes \) U(1)\(_\mathrm {valley}\) symmetric spin–valley Hamiltonian on the triangular lattice. The explicit Hamiltonian we consider is

$$\begin{aligned} {\mathcal {H}}= & {} \frac{J}{8} \sum _{\langle ij\rangle } (1 + \varvec{\sigma }_i\varvec{\sigma }_j)(1 + \varvec{\tau }_i \varvec{\tau }_j)\nonumber \\&\quad + \frac{J_{x}}{8} \sum _{\langle i j \rangle } (1 + \varvec{\sigma }_i\varvec{\sigma }_j)(\tau _i^x \tau _j^x + \tau _i^y \tau _j^y)\nonumber \\&\quad + \frac{J_{z}}{8} \sum _{\langle i j \rangle } (1 + \varvec{\sigma }_i\varvec{\sigma }_j)(\tau _i^z \tau _j^z), \end{aligned}$$
(61)

with a SU(4) symmetric term proportional to the coupling J and an in-plane \(J_x\) and out-of-plane \(J_z\) coupling that when non-zero break the SU(4) symmetry down to an SU(2) symmetry in the spin sector and a U(1) symmetry in the valley sector. We only include interactions between nearest neighbours \(\langle ij \rangle \).

Such a model can be motivated, e.g., from including the effect of Hund’s type couplings in a two-orbital extended Hubbard model and performing a strong coupling expansion [20]. It can therefore be regarded as a natural extension to previously studied models with either full SU(4) or reduced SU(2)\(_\mathrm {spin}\) \(\otimes \) SU(2)\(_\mathrm {valley}\) symmetry [18,19,20,21] by adding an XXZ type perturbation to the orbital sector and likewise provides an intermediate, but important step towards the more complicated spin–valley Hamiltonians proposed for various moiré systems [10, 11, 20].

5.1 Phase diagram

To obtain the quantum phase diagram, we fix the coupling J in front of the SU(4) symmetric part of the Hamiltonian in Eq. (61) to a positive value and then vary the values of the in-plane coupling \(J_x\) and out-of-plane coupling \(J_z\) which break the SU(4) symmetry. As described in Sect. 3, to determine the magnetic order for a particular pair of couplings \((J_x, J_z)\) we calculate the flow of the spin–spin and valley–valley correlations (and associated structure factors) as defined in Eqs. (13, 14), check whether or not a flow breakdown occurs and if so, which type of order is visible in the structure factor at the breakdown scale \(\varLambda _c\). Due to the SU(2)\(_\mathrm {spin}\) \(\otimes \) U(1)\(_\mathrm {valley}\) symmetry of the Hamiltonian all non-vanishing components of the spin–spin correlation are equivalent and we calculate only \(\chi ^{\varLambda s}_{ij} \equiv \chi ^{\varLambda s, xx}_{ij} = \chi ^{\varLambda s, yy}_{ij} = \chi ^{\varLambda s, zz}_{ij}\). For the valley–valley correlation, we can distinguish between in-plane and out-of-plane order by calculating the in-plane valley–valley correlation \(\chi ^{\varLambda v, x}_{ij} \equiv \chi ^{\varLambda v, xx}_{ij} = \chi ^{\varLambda v, yy}_{ij}\) and out-of-plane valley-valley correlation \(\chi ^{\varLambda v, z}_{ij} \equiv \chi ^{\varLambda v, zz}_{ij}\).

Starting at the SU(4) point with \(J_x/J = J_z/J = 0\), where all spin–spin and valley–valley correlations are equivalent, we observe no flow breakdown of the structure factors, as depicted by the grey line in Fig. 3. This indicates that no magnetic order is present in both the spin and the valley sector even for very-low-energy scales and indicates a putative quantum spin–valley liquid (QSVL) state [21]. Going away from the SU(4) point, however, we almost immediately observe a flow breakdown in either the spin or valley sector, indicating that the putative QSVL state is highly unstable in the presence of XXZ like perturbations. This is in line with results for the \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) XXZ model on the triangular lattice, where by varying the out-of-plane coupling a phase transition from in-plane \(120^\circ \) order to an “umbrella” order is observed at the SU(2) symmetric point [40]. Similarly, we observe a rich ensemble off different spin and valley ordered phases with both in- and out-of-plane ordering in the valley sector.

Fig. 3
figure 3

Flow of the structure factor at points of higher symmetry. All structure factors are shown at the momentum where they are maximal. The grey line shows the structure factor at the SU(4) point, where the considered spin–valley model corresponds to the SU(4) symmetric Heisenberg model. Here, all structure factor components are identical. The flow is smooth and convex down to the lowest numerically reliable cut-off and no flow breakdown occurs, indicating a putative quantum spin–valley liquid (QSVL) ground state. The purple and green lines show the spin and valley structure factor for \(J_x/J = J_z/J = -1\), where all terms containing valley operators cancel and the spin–valley model resembles an SU(2) symmetric Heisenberg model. In this case, the valley structure factors are strongly suppressed and the spin structure factor shows a sharp peak at the K and \(K'\) points, indicating \(120^\circ \) order in the spin sector

Before we present the full quantum phase diagram, however, let us first consider a classical mean-field approach to better understand the origin of the different phases. To this end, we note that the spin sector by itself will order either ferromagnetically (FM) or in a \(120^\circ \) order, depending on the sign of the exchange coupling. Assuming one of these states is realized, we decouple the spin and valley sector by approximating the pair of spin operators by its expectation value \(\varvec{\sigma }_{i}\varvec{\sigma }_{j} \approx \langle \varvec{\sigma }_{i}\varvec{\sigma }_{j}\rangle \), with \(\langle \varvec{\sigma }_{i}\varvec{\sigma }_{j}\rangle = 1\) for ferromagnetic (FM) and \(\langle \varvec{\sigma }_{i}\varvec{\sigma }_{j}\rangle = \cos (2\pi /3) = -0.5\) for \(120^\circ \) order. The resulting mean-field Hamiltonian is then given, up to a constant, by

$$\begin{aligned} {\mathcal {H}}_{\mathrm {MF}}= & {} E^s_0 \sum _{\langle ij \rangle } \bigg [\left( 1 +\frac{J_{x}}{J}\right) \left( \tau _i^x \tau _j^x + \tau _i^y \tau _j^y\right) \nonumber \\&+ \left( 1 + \frac{J_z}{J}\right) \tau _i^z \tau _j^z \bigg ], \end{aligned}$$
(62)

where the spin expectation value only appears in the positive factor \(E_0^s \equiv J(1+\langle \varvec{\sigma }_{i}\varvec{\sigma }_{j}\rangle )\) and, therefore, has no influence on the type of valley order. Approximating the valley operators by classical vectors with \({\left| \varvec{\tau }_i\right| }=1\), we perform a Luttinger–Tisza analysis [41, 42] on the mean-field Hamiltonian. This analysis predicts in-plane (out-of-plane) valley order for large values of \({\left| 1+J_x/J\right| }\) (\({\left| 1+J_z/J\right| }\)), which is either FM for positive, or \(120^\circ \) like for negative values. The precise phase boundaries along with the ground-state energies \(E_g\) are depicted in Fig. 4.

Fig. 4
figure 4

Classical phase diagram in valley space for fixed spin ordering obtained from a Luttinger–Tisza analysis. The grey lines depict the phase boundaries and the color illustrates the (normalized) ground-state energy, where blue denotes out-of-plane and orange denotes in-plane ordering. At \(J_x/J = J_z/J = - 1\), where the phase boundaries meat, the classical mean-field Hamiltonian vanishes. Away from this point the Luttinger–Tisza analysis predicts the following types of valley order: (II) in-plane ferromagnetic (FM), (III) out-of-plane FM, (IV) in-plane \(120^\circ \), and (VI) out-of-plane \(120^\circ \). The so-obtained valley order is independent from the fixed nearest-neighbour spin order

Fig. 5
figure 5

Phase diagram of the SU(2) \(_\mathrm {spin}\) \(\otimes \) U(1)\(_\mathrm {valley}\) symmetric spin–valley model. The color indicates which structure factor is dominant at the breakdown scale \(\varLambda _c\), where purple implies dominant spin order, colors between orange and yellow indicate dominant order in the valley sector, and the opacity determines the magnitude of the breakdown scale \(\varLambda _c/J\). In the case where we observe a flow breakdown in both the in-plane (\(\chi ^{\varLambda v, x}\)) and out-of-plane (\(\chi ^{\varLambda v, x}\)) valley structure factor the color determines the angle \(\phi \) illustrated in the cones on the right of the figure. (I–VI) show the structure factors at \(\varLambda _c\) for the different types of order we observe: (I) 120\(^\circ \) spin order, (II) out-of-plane FM valley order, (III) in-plane FM valley order, (IV-VI) 120\(^\circ \) valley order shifting from an out-of-plane (IV) to an in-plane (VI) orientation, with competing order (V) in between. For \(J_x/J = J_z/J = 0\), indicated by the star, the model is equivalent to the SU(4) symmetric Heisenberg model for which no flow breakdown is observed

Fig. 6
figure 6

Flow of the structure factors for different types of order as described under Fig. 5. The dashed lines show the breakdown scale \(\varLambda _c\). (I) shows dominant spin order (purple) with the valley structure factors strongly suppressed. (IV–VI) shows dominant valley order which shifts from an in-plane (orange) to an out-of-plane (blue) orientation

Special attention needs to be paid to the point at \(J_x/J = J_z/J = -1\) where the phase boundaries meet. Exactly at this point, the couplings in front of the valley operators are equal to zero and the mean-field Hamiltonian vanishes. Going back to the full quantum Hamiltonian, it reduces to only the term \(\sum _{ij} J (1 + \varvec{\sigma }_i \varvec{\sigma }_{j})\), which resembles an SU(2) symmetric Heisenberg model with antiferromagnetic coupling J. Here, the flow of the spin structure factor shows a sharp peak, while the valley structure factors are strongly suppressed, as is depicted in Fig. 3. The same behavior occurs in a larger region around \(J_x/J = J_z/J = -1\), which is shown in Fig. 5 along with the corresponding momentum resolved structure factor (annotated with the numeral I). The spin structure factor (in purple) shows strong peaks at the K and \(K'\) points, while the in-plane (orange) and out-of-plane (blue) valley structure factors show no distinct features when shown on the same color scale. This indicates \(120^\circ \) spin order, which again agrees with the results for the conventional \({{\mathfrak {s}}}{{\mathfrak {u}}}(2)\) Heisenberg model [40, 43].

In all other regions of the quantum phase diagram, the valley structure factors are clearly dominant and the spin structure factor shows only weak features. We enumerate the different types of valley order we find by the numerals II–VI, as shown in Fig. 5. The valley order at large negative couplings (II and III) agrees with the classically predicted results, as either the in- or out-of-plane structure factors show strong peaks at the \(\varGamma \) point, indicating FM order. At larger positive values for either the in-plane or out-of-plane coupling (VI–IV), the valley structure factors show peaks at the K and \(K'\) points indicating \(120^\circ \) like order. In contrast to the sharp phase boundary in the classical case, however, the valley order seems to gradually shift from mostly in-plane (IV), over competing in- and out-of-plane (V) to out-of-plane (VI) order. This is well visualized by the flow of the structure factors in Fig. 6. The valley structure factors both show flow breakdowns at approximately the same breakdown scale, but the magnitude at the breakdown scale shifts from a dominant \(\chi ^{\varLambda v, x}\) to a dominant \(\chi ^{\varLambda v, z}\) when going from IV to VI. To quantify this transition, we define the angle

$$\begin{aligned} \phi = \text {arctan}(\chi ^{\varLambda _c v, x}/ \chi ^{\varLambda _c v, z}), \end{aligned}$$
(63)

illustrated in Fig. 5 by the cones on the right and by the color scale ranging from from blue (in-plane) over green (competing in-plane and out-of-plane) to orange (out-of-plane)

Fig. 7
figure 7

Horizontal cuts through the phase diagram of Fig. 5 at \(J_z/J = 1\) (top), \(J_z/J = 0\) (middle) and \(J_z/J = -1\) (bottom). The color-coding and the labels (I-VI) denote different types of order and are explained in Fig. 5. The transition between these phases is always accompanied by a dip or kink in the breakdown scale. In the transitions between dominant spin and valley order, there are regions where both the spin and valley structure factor show flow breakdowns at a similar \(\varLambda _c\) and with similar magnitudes. These regions are shaded and colored both orange and purple, as, e.g., visible in the transition between II and I along the \(J_z/J =-1.0\) axis. At the SU(4) point (colored white), no flow breakdown occurs

Fig. 8
figure 8

Vertical cuts through the phase diagram of Fig. 5 at \(J_x/J = 1\) (top), \(J_x/J = 0\) (middle) and \(J_x/J = -1\) (bottom). The color-coding and the labels (I–VI) denote different types of order and are explained in Fig. 5. The transition between these phases is always accompanied by a dip or kink in the breakdown scale. In the transitions between dominant spin and valley order, there are regions where both the spin and valley structure factor show flow breakdowns at a similar \(\varLambda _c\) and with similar magnitudes. These regions are shaded and colored both orange and blue, as e.g. visible in the transition between III and I along the \(J_x/J =0\) axis. The white regime close to the SU(4) point marks points for which no flow breakdown is observed

To better illustrate the transitions between the different types of order, Figs. 7 and 8 show horizontal and vertical cuts through the phase diagram, respectively. The transitions between the phases are always accompanied by a dip or kink in the breakdown scale, indicating the positions of the phase boundaries. This is especially relevant for the transitions between mixed in- and out-of-plane valley order (V) to dominant in- or out-of-plane valley order (IV and VI), where the phase transition would not be as easily recognizable by just considering the evolution of the structure factors. The same is true for the transition from dominant valley to dominant spin order, as, e.g., depicted in the \(J_x/J = 0\) cut in Fig. 8. Here, the at first very dominant out-of-plane valley order (III) gradually transitions to dominant spin order (I), with a region in between where the spin and valley structure factors are of similar magnitude. The kink in the breakdown scale appears at the largest \(J_z/J\) where the valley structure factor still shows a clear flow breakdown (\(J_z/J \approx -1.6\)), even though the spin structure factor is already dominant for smaller \(J_z/J\). This is similar at all boundaries of phase I, which also becomes evident in the phase diagram of Fig. 5 by noting that the minima of the breakdown scale are positioned slightly inwards in the region of dominant spin order (colored in purple).

Of special interest are the cuts across the SU(4) point (\(J_x/J = 0\) and \(J_z/J = 0\)), which show that even for very small values of the in- and out-of-plane couplings, the flow develops a breakdown at a finite \(\varLambda _c\).

6 Summary

In this manuscript, we have presented a generalization of the established pf-FRG approach to generic spin–valley Hamiltonians in the self-conjugate representation of \({{\mathfrak {s}}}{{\mathfrak {u}}}\)(4), with either diagonal spin or valley interactions. We performed a careful symmetry analysis and derived a set of constraints on the vertex functions, which drastically lower the computational cost of tracking the flow of running couplings. Using a highly accurate solver for the functional flow equations, we subsequently applied this method to map out the quantum phase diagram of an SU(2)\(_{\text {spin}}\) \(\otimes \) U(1)\(_{\text {valley}}\) model on the triangular lattice, which presents a simplified variant of the more general Hamiltonian proposed for TLG/h-BN, but already hosts a rich variety of spin and valley ordered ground states. In addition, we were able to demonstrate, that, by promoting the spin symmetry group from SU(2) to SU(4), quantum fluctuations are boosted, ultimately resulting in a smooth RG flow down to the lowest energy scales, indicative of a spin–valley liquid state. However, this QSVL state appears to be very sensitive even to weak XXZ anisotropies in the valley sector, and we almost immediately detect the emergence of long-range order, when perturbing it.

While our focus in this manuscript has been on spin–valley Hamiltonians, we note that very similar models have been discussed for spin–orbit coupled systems that go beyond the celebrated Kugel–Khomskii model. The microscopic ingredients of such spin–orbital models are surprisingly similar to those of “Kitaev materials” [44]—a partially filled 4d or 5d orbital, the formation of a spin–orbital entangled local moment, and an edge-sharing octahedral crystalline environment. Specifically, a \(d^1\) configuration can lead to local \(j=3/2\) moments subject to bond-directional exchanges that break the original SU(4) symmetry of the \(j=3/2\) moments. As a concrete material candidate exhibiting this microscopic mechanism, \(\alpha \)-\(\hbox {ZrCl}_3\) – a 4d sister compound of the isostructural Kitaev material \(\hbox {RuCl}_3\)—has been put forward [45]. To study the phase diagram of spin–orbital ground states in such a setting with varying diagonal and off-diagonal couplings, one can again rely on the pseudo-fermion FRG approach put forward in this manuscript.