Twisted transition metal dichalcogenide (TMD) homobilayers have recently emerged as a powerful platform for studying correlated insulating states. In the strongly correlated limit, we construct an effective spin Hamiltonian on a honeycomb lattice that includes the Heisenberg interaction and nonsymmetric interactions such as a Dzyaloshinskii-Moriya interaction and a Kane-Mele coupling for the Mott-insulating phase at half-filling. For the twisted TMD homobilayers, the spin-orbit coupling in the Hubbard model, which is expected to induce the antisymmetric exchange couplings in the effective spin Hamiltonian, is a highly tunable and experimentally accessible quantity that can be tuned by an applied electric field. In this study, we investigate classical and quantum phase diagrams of the effective spin Hamiltonian using analytical and numerical methods. We show that the model exhibits a rich classical phase diagram including an antiferromagnetic (AFM) phase, a planar spiral ordered phase with high classical degeneracy, a $z$-AFM phase, a noncoplanar phase, a noncollinear phase, and a ${120}^{\circ }$-AFM phase. In the quantum treatment, we calculate low-energy magnon excitation spectrum, ground-state energy, and static spin structure factor using linear spin-wave theory and density matrix renormalization group methods to compose the quantum phase diagram of the effective spin Hamiltonian. Beyond the Heisenberg interaction, we find that the existence of these antisymmetric couplings is responsible for the quantum spin liquid, $z$-AFM, noncoplanar, and ${120}^{\circ }$ phases. Twisted TMD homobilayers, therefore, offer rich platforms for realizing rich phases of matter such as quantum spin liquid, noncoplanar, and ${120}^{\circ }$, resulting from the spin-orbit coupling.

• Received 19 April 2021
• Revised 13 September 2021
• Accepted 15 September 2021

DOI:https://doi.org/10.1103/PhysRevB.104.115154