Introduction

For the classification of states of matter, the role of topology as a principle beyond spontaneous symmetry breaking was first demonstrated in quantum Hall states1,2,3,4,5,6,7,8,9,10,11. However, the extensive current research interest in topological materials started with topological insulators (TIs), preluded by the discovery of quantum spin Hall (QSH) states in quantum wells4,5. QSH insulators, often synonymously referred to as two-dimensional (2D) TIs, are characterized by a finite bulk band gap and gapless helical edge states carrying opposite spins1,2,3,4,5,6,7,8,9,10,11. Time-reversal symmetry dictates the destructive interference between backscattering paths, resulting in dissipationless spin currents, bearing huge significance for electronic and spintronic applications12,13,14,15. Despite the intense pursuit in the past decade for QSH insulators beyond quantum-well systems, the only candidates found are monolayer WTe27,8,9,10 and WSe211.

Compared to the case of a time-reversal-invariant 2D TI classified by the \(\mathbb{Z}_2\) index, 3D TIs are classified by four \(\mathbb{Z}_2\) indices (ν0; ν) ≡ (ν0; ν1ν2ν3)16,17,18. A strong TI (STI) is indicated by a nonzero ν0, in which the bulk gap is closed by surface states featuring an odd number of Dirac cones on all surfaces of the sample1,2. In contrast, a zero ν0 and finite ν point to a weak TI (WTI), in which gapless surface states associated with an even number of Dirac cones appear only on the side surfaces. Compared to 2D TIs and 3D STIs, the territory of WTIs remains relatively less explored, with only a few candidates found to date, such as ZrTe5, HfTe5, Bi14Rh3I9, β-Bi4I4, CaSn, Bi2TeI and RhBi215,19,20,21,22,23,24. All these compounds consist of weakly coupled layers, a structure desirable for realizing WTI, as a WTI is topologically equivalent to a stack of 2D TI layers15,19,20,21,22,23,24. Considering how the sample can be readily cleaved to few layers or even monolayer, WTIs with a van der Waals-type structure provide a promising platform for realizing QSH states. Moreover, contrary to initial perceptions and what is implied in their name, WTIs have been demonstrated to be robust even under strong disorders, provided the time-reversal symmetry remains intact15,19,20,21,22,23,24.

Even more interesting than the identification of new WTIs is, perhaps, the realization of efficient tuning between an STI and WTI in a single material. A prominent example is ZrTe5, in which the traversal between STI and WTI can be readily realized by, e.g., applying strain15, or simply varying the temperature25. Finding more material platforms for such tuning is not only beneficial for the understanding of the respective topological phases, but also holds potential in an effective switching of topologically nontrivial edge/surface modes and the resultant transport properties.

Here, by performing density functional theory calculations, we found that the van der Waals-type compound AuTe2Br possesses a STI or WTI state depending on whether the raw experimental (Expt.) or fully optimized (Opt.) structural parameters are used as input for the calculations, respectively. This dichotomy indicates that AuTe2Br may locate at the boundary between an STI and a WTI state. Despite its sensitivity to structural variations, the critical topology is found to persist up to an applied pressure of ~15.4 GPa, where a pressure-induced structural phase transition from Cmc21 to Pmmm occurs. The low-pressure TI phase emerges from a topological nodal-line metal (TNLM) protected by mirror symmetry when the spin-orbit coupling (SOC) is ignored. In contrast, the high-pressure phase in the absence of SOC is a TNLM stabilized by a distinct mechanism due to the combination of the mirror-reflection and space-time-inversion symmetry. Furthermore, superconductivity is observed in a wide pressure range in the high-pressure phase. By a scrutinization on the structural parameters that dictate the topological transition between the strong and weak topological phases, our results add another candidate to the list of WTIs and hence QSH insulators via proper tuning. Our work also provides a fertile playground for the exploration of topological phase transitions and topological superconductivity.

Results

Critical topology at ambient pressure

AuTe2X (X = Cl, Br) crystallize in an orthorhombic structure, consisting of halogen atoms inserted between layers of AuTe2 networks stacking along the b axis (Fig. 1a), despite their space groups differing from each other—Cmcm (No. 63) and Cmc21 (No. 36) for Cl, and Br, respectively26,27,28,29. The topological properties of AuTe2X remained unexplored until recently29,30. Transport measurements revealed the compensated semimetal nature of AuTe2Br with ultrahigh carrier mobility of ~105 cm2 V−1 s−1 and a nonsaturated magnetoresistance reaching ~3 × 105 at 4.2 K and 58 T29. More interestingly, it was proposed that the QSH effect can be realized in monolayer AuTe2Cl through the confinement effect30. Our calculations on the band structure of bulk AuTe2Cl (Supplementary Fig. 1) give unanimously a topological nontrivial gap along the Γ–Y direction and a \(\mathbb{Z}_2\) of (1; 110), regardless of whether Expt. or Opt. structural parameters are used, confirming the STI nature of AuTe2Cl. Note that such an STI is topologically equivalent to the topological semimetal suggested by previous studies30, if a curved chemical potential between valence and conduction bands is defined31,32. Considering that some bands in AuTe2Br intersect the Fermi level, in the following we adopt the terms strong topological semimetal (STSM) and weak topological semimetal (WTSM) to avoid ambiguity. The topological properties are not affected by the semimetallic nature, and the approaches in the study of STI and WTI can be naturally utilized in our case.

Fig. 1: Band structure calculations for AuTe2Br without spin-orbit coupling (SOC).
figure 1

a Crystal structure of AuTe2Br with the Cmc21 space group. hlayer, dlayer and llayer (llayer = dlayer + hlayer) represent the height of an Au–Te layer (i.e., the vertical distance along b between the top and bottom Te atoms in the same Au–Te layer), the vertical distance between the top Te atom in one Au–Te layer and the bottom Te atom in the Au-Te layer above the previous one, and the vertical distance between two adjacent AuTe2Br layers, respectively. b The 3D bulk Brillouin zone (BZ). c–e are the calculated band structure, orbital-projected band structures along the Γ-Y-K directions, and the energy difference map between the Au dxz+yz and Te pz bands in the ΓYKZ plane of the BZ for AuTe2Br with experimental (Expt.) structural parameters, respectively. f–h are those results with fully optimized (Opt.) structural parameters. The contour of the nodal ring is shown by the red line in (e) and (h). The background color represents the magnitude of the gap between the Au dxz+yz and Te pz bands. With the gap increasing, the color changes from white to blue. Red ellipse highlights the zero gap.

Full size image

Our calculation results for AuTe2Br in the monolayer limit are presented in Supplementary Fig. 2. Fully optimized structure parameters are adopted for the monolayer, giving relaxed lattice constants close to the experimental values for the bulk counterpart. Similar to the case of monolayer AuTe2Cl, monolayer AuTe2Br is shown to be a 2D TI, rendering it a candidate platform for the study of the QSH effect.

Turning to the bulk AuTe2Br, the calculation results in the absence of SOC are shown in Fig. 1c–h for Expt. and Opt. structural parameters, respectively. The results are similar for the two sets of parameters. We find a metallic ground state with electron pockets at the X and M points and hole pockets at the Γ and Y points. The bands exhibit crossing points in the high symmetry directions Γ-Y and Y-K that are tied to the ΓYKZ plane of the bulk Brillouin zone. By projecting the states onto atomic orbitals, we find that the two bands forming the crossing points are contributed by the Au dxz+yz orbital and Te pz orbital, respectively, as shown in Fig. 1d, g. The band order is inverted around the Y point, indicating a nontrivial topology. Due to the presence of mirror symmetry associated with the ΓYKZ plane, the crossing points actually form a nodal line in this plane centering around Y, as depicted in Fig. 1e, h.

The calculation results for the band structure of bulk AuTe2Br with SOC included are shown in Fig. 2a, h for Expt. and Opt. structural parameters, respectively. The nodal line is gapped for both cases. Surprisingly, however, topologically distinct results are found for the two sets of parameters. Before structural optimization, there is a gapped Dirac cone arising from the band hybridizations between Au dxz+yz and Te pz orbitals along Γ–Z, resulting in inverted bands at the Y point (Fig. 2b). With structural optimization, the band order at Γ is inverted as well (Fig. 2f). To further characterize the band topology, the Wannier charge centers (WCC) evolution on the time-reversal invariant planes is calculated using the Wilson loop method, as shown in Fig. 2c, g. In a WCC plot, the \(\mathbb{Z}_2\) indices can be obtained by counting the number of times the WCC curves intersect with any arbitrary horizontal reference line33. For example, for the kx = 0 plane, no intersection is found so that the associated \(\mathbb{Z}_2\)= 0 [the top left panel in Fig. 2c], while for the kx = 0.5 plane, one intersection is found so that the associated \(\mathbb{Z}_2\) = 1 [the top right panel in Fig. 2c]. For a 3D TI, ν0 is determined by the parity of the sum of the \(\mathbb{Z}_2\) associated with all the ki = 0 (i = x, y, z) and 0.5 planes, whereas νi (i = 1, 2, 3) is determined by the \(\mathbb{Z}_2\) associated with the ki = 0.5 (i = x, y, z) planes. For the Expt. and Opt. case, respectively, we get (1; 110) and (0; 110) and, thus, an STI (STSM) state and a WTI (WTSM) state. This is also reflected in the calculated surface states on the (001) projected surface (note that this is a side surface): one surface Dirac point is pinned at \({{{\bar{\mathrm Y}}}}\) for the Expt. case (Fig. 2d), while two Dirac points are pinned at \({{{\bar{\mathrm {\Gamma}}}}}\) and \({{{\bar{\mathrm Y}}}}\) for the Opt. case (Fig. 2h).

Fig. 2: Band structure calculations for AuTe2Br with SOC included.
figure 2

a–d are the calculated band structure, orbital-projected band structures along the Γ-Y direction, the evolution of the Wannier charge centers [θ (2π)], and the calculated surface states on the (001) projected surface for AuTe2Br with Expt. structural parameters, respectively. e–h are those results with Opt. structural parameters.

Full size image

Tuning between the strong and weak topological phases

The dichotomy between the two cases indicates that AuTe2Br exhibits a critical topology, where the balance is tipped towards an STSM or a WTSM state by a small variation in the crystal structure. It is therefore interesting to study how the band topology evolves under hydrostatic pressure. To obtain the structural parameters under pressure, high-pressure x-ray diffraction (XRD) measurements were performed, and the results are displayed in Fig. 3a. With increasing pressure, the Cmc21 phase persists up to ~18.8 GPa. Upon further compression, new diffraction peaks emerge, indicative of a structural phase transition. Enthalpy calculations show that above 15 GPa, the Pmmm symmetry is favored (Fig. 3b, c). No imaginary modes are found in the phonon calculations above 20 GPa (Supplementary Fig. 4), confirming the robust dynamic stability of this Pmmm phase under high pressure. The lattice constants extracted from Rietveld refinements are displayed in Fig. 3d, showing a drastic drop of b across the structural transition.

Fig. 3: Pressure evolution of the crystal structure of AuTe2Br.
figure 3

a X-ray diffraction 2θ plot of AuTe2Br at room temperature up to 52.2 GPa. The ambient pressure Cmc21 structure persists to ~18.8 GPa, beyond which new diffraction peaks emerge (each marked with a dashed line and asterisk), indicating the emergence of pressure-induced structural phase transition. b Crystal structure of the high-pressure AuTe2Br with Pmmm space group. c Enthalpy calculations for several predicted structures. Above 15 GPa, the Pmmm symmetry has the lowest enthalpy. d Pressure dependence of the lattice constants derived from Rietveld refinement.

Full size image

The band structure under pressure before the structural transition is shown in Supplementary Fig. 5. With increasing pressure, the band dispersion along Γ–Y increases. However, for both Expt. and Opt. structural parameters, the band order at Γ and Y is maintained. In other words, the critical topology of AuTe2Br between an STSM and WTSM is robust against pressure, provided the space group stays the same.

At first glance, this seems to be inconsistent with the fact that the band topology is sensitive to variations in structural parameters. To solve this apparent discrepancy, we investigate the relation between the critical topology and the structural parameters. As discussed above and schematically shown in Fig. 2b, f, the key difference between an STI and WTI state is the band order at Γ. We plot for AuTe2Cl and AuTe2Br the dependence of the energy difference of the two bands at Γ, \(\left[ {{\Delta}E_{{{\mathrm{{\Gamma}}}}} \equiv E({{{\mathrm{Au}}}}\;d_{xz/yz}) - E({{{\mathrm{Te}}}}\;p_z)} \right]\), on representative structural parameters [Fig. 4, see Fig. 1a for the definition of these parameters]. AuTe2Cl being an STSM is robust against structural variations. For AuTe2Br, with Expt. structural parameters, the pressure-induced decreasing of hlayer pushes the data points under pressure to the left of the ambient pressure data point Br (Expt.) (Fig. 4a), meaning that AuTe2Br is always an STSM.

Fig. 4: The topology of AuTe2Cl and AuTe2Br as a function of representative structural parameters and pressure.
figure 4

a, b The topology—being a strong topological semimetal (STSM) or weak topological semimetal (WTSM)—of AuTe2Cl (labeled as Cl) and AuTe2Br (labeled as Br), as represented by the energy difference between Au-dxz/yz and Te-pz bands at the Γ point, as a function of hlayer and dlayer, respectively. Insets in (a) and (b) are pressure dependence of hlayer and dlayer, respectively. c llayer dependence of ΔEΓ. The orange star represents the experimental data, while the cyan squares stand for the simulation data obtained from allowing llayer to vary with the orange star as the starting point, while keeping other structural parameters unchanged. Through llayer tuning, the topological transition from STSM to WTSM may be realized.

Full size image

However, for the Opt. case, as shown in Fig. 4a, one would expect a transition from WTSM to STSM with the considerable pressure-induced decreasing of hlayer since the ambient pressure data point Br (Opt.) is already very close to the boundary. Such a transition is not observed in our calculations (Supplementary Fig. 5). This may be explained by the counteracting effect of the pressure evolution of another parameter, namely, dlayer: As shown in the inset to Fig. 4(b), ΔEΓ evolves with dlayer in a similar fashion to the case of hlayer, but dlayer itself increases with pressure.

To allow for a more direct comparison with future experiments that utilize the tuning of structural parameters to traverse the boundary between the strong and weak topological phases, we define a more experimentally-relevant parameter llayer = dlayer + hlayer. As shown in Fig. 4c, starting from Br (Expt.), a WTSM state may be achieved by increasing llayer, which can be realized by, e.g., tensile strain along b, or intercalation between the Au and Te layers. This adds AuTe2Br as another example of readily achievable topological transition between strong and weak phases to the existing case of ZrTe5.

Topology and superconductivity in the high-pressure phase

We now turn to the band topology of the high-pressure Pmmm phase. In the absence of SOC, the band structure exhibits a 3D metallic nature with some bands featuring large energy dispersion across the Fermi level (Fig. 5a). The 3D Brillouin zone is shown in Fig. 5b. There are two band crossings along two paths in the kx = 0 plane, Γ–Z and Γ–T (Fig. 5c). The crossings are attributed mainly to the Te-px and pz orbitals. At Γ, the pz band lies above px, which is a signature of band inversion, implying the nontrivial band topology of the Pmmm phase. Moreover, due to the recovery of the inversion symmetry in this phase, a mechanism different from the ambient pressure one dictates the presence of a nodal line in the kx = 0 plane and centering around Γ (Fig. 5d). Here, the nodal line is protected by the combination of the space-time-inversion symmetry and mirror-reflection symmetry34,35. With the inclusion of SOC, the nodal line is fully gapped with a gap size of ~0.02 eV, as shown in Fig. 5a. The (100) surface band structures are shown in Fig. 5e. Outside the projection of the nodal line crossings, the surface bands disperse upwards when SOC is ignored. Taking SOC into account, there is a metallic surface band connecting the projected bulk valence and conduction bands. This surface band exhibits spin-momentum-locked spin textures, as displayed in Fig. 5f. To further characterize the band topology, the WCC evolution on the time-reversal invariant planes is calculated (Supplementary Fig. 6), and the \(\mathbb{Z}_2\) index is determined to be (1;001), indicative of strong topological nature of high-pressure Pmmm phase of AuTe2Br.

Fig. 5: Band structure calculations for AuTe2Br at 50 GPa.
figure 5

a Calculated band structure of AuTe2Br at 50 GPa. The red and blue lines represent the results without and with SOC. b The 3D bulk Brillouin zone (BZ). c Orbital-projected band structure near the Γ point. The contour of the nodal ring in the absence of SOC is shown by the red line in (d). The background color represents the magnitude of the gap between the Te px and Te pz bands. With the gap increasing, the color changes from white to blue. Red ellipse highlights the zero gap. e Surface band structures without and with SOC for the (100) surface. f Fermi surface and corresponding spin texture at a fixed energy E – EF = 20 meV of the topological surface states in the presence of SOC.

Full size image

In addition to the nontrivial band topology, we observed superconductivity in a wide pressure range in the Pmmm phase. Temperature-dependent resistance data for AuTe2Br single crystals are shown in Fig. 6a. A superconducting transition emerges at ~19 GPa and persists to the highest pressure of ~51 GPa we measured. We then check the superconducting transition under magnetic field (Fig. 6b) and obtain the upper critical field Hc2 at various pressures (Fig. 6c). The pressure dependence of the superconducting transition temperature Tc is summarized in Fig. 6d. Nevertheless, the nontrivial band topology of AuTe2Br leaves the possibility of topological superconductivity an interesting topic for future studies.

Fig. 6: Pressure-induced superconductivity in AuTe2Br.
figure 6

a Temperature dependence of the resistance of AuTe2Br single crystals with silicone oil as the pressure transmitting medium. S1, S2 and S3 represent different runs. b Magnetic field dependence of the superconducting transition of AuTe2Br single crystal (S2) at 27.8 GPa. The superconducting transition temperatures Tc are defined as shown in the panel. c Temperature dependence of the upper critical field μ0Hc2. Solid lines are a guide to the eye. d Pressure dependence of Tc.

Full size image

Discussion

In summary, AuTe2Br is proposed as a new platform for the exploration of WTI states based on our first-principles calculations. Nontrivial topology was found both for the ambient pressure phase and the phase after a pressure-induced structural phase transition, the latter accompanied by superconductivity observed in our measurements. Importantly, our calculations also demonstrate the promising possibility of investigating QSH effect in monolayer AuTe2Br, which can be readily obtained due to the van der Waals-type structure.

The fact that AuTe2Br lies in close proximity to the topological transition between an STSM and WTSM state holds substantial potential in the continuous tuning of such a transition. Although we have shown that pressure may not be the ideal tuning knob, probably as a result of an opposite trend in the pressure evolution of structural parameters determining the electronic topology, we demonstrate that specifically designed variation of a key parameter, i.e., the distance between adjacent Au-Te layers, can serve this purpose. This may be achieved by intercalation, strain, etc. It would then be more interesting to see how superconductivity is affected when the topological transition is traversed.

Methods

Sample synthesis and magneto-resistivity measurements

AuTe2Br single crystals were synthesized, as described in ref. 30. The as-grown single crystals are soft, silver flakes with high quality, and the biggest natural plane is (0l0) plane [see Supplementary Fig. 3a]. Magneto-resistivity measurements under ambient pressure were performed in a physical property measurement system (PPMS; Quantum Design).

Resistance measurements under pressure

High-pressure resistance measurements were performed on AuTe2Br single crystals with silicone oil as the pressure transmitting medium (denoted as S1, S2, S3 for different runs) by using a diamond anvil cell (DAC). The experimental pressures were determined by the pressure-induced fluorescence shift of ruby at room temperature before and after each experiment. A direct current four-probe technique was adopted. Resistance measurements were performed with a PPMS.

XRD measurements under pressure

AuTe2Br single crystals were ground into powder by using a mortar for use in the high-pressure synchrotron angle-dispersive X-ray diffraction measurement. The high-pressure synchrotron XRD experiments were carried out using a symmetric diamond anvil cell (DAC) with a 300-micron culet diamond. A rhenium gasket was drilled by laser with a 90-micron diameter hole as the sample chamber. The sample chamber was filled with a mixture of the sample, a ruby ball, and silicone oil as the pressure transmitting medium. The experimental pressures were determined by the pressure-induced fluorescence shift of ruby. Synchrotron angle-dispersive XRD measurements were carried out at beamline BL15U1 of the Shanghai Synchrotron Radiation Facility (SSRF) using a monochromatic beam of 0.6199 Å. The diffraction patterns were integrated by using the Dioptas software, and Rietveld refinement was performed by using the GSAS software.

Electronic band structure of AuTe2 X (X = Cl, Br)

We carried out first-principles calculations within the framework of the projector augmented-wave (PAW) method36, as implemented in the Vienna Ab initio Simulation Package (VASP)37,38. A kinetic energy cutoff of 500 eV and a Γ-centered k mesh of 11 × 11 × 7 were selected in all calculations. The energy and force difference criterion were defined as 10−6 eV and 0.01 eV/Å for self-consistent convergence. The van der Waals (vdW) corrections39,40 and spin-orbit coupling (SOC) effect are considered for all calculations. The electronic band structure for AuTe2X (X = Cl, Br) were calculated with experimental structure and fully optimized lattice parameters (PBE + D2). PBE + D2 represents Perdew–Burke–Ernzerhof (PBE)39, generalized gradient approximation (GGA) functional with D2 empirical vdW corrections41. The WANNIER90 package42,43,44 was adopted to construct Wannier functions from the first-principles results without an iterative maximal-localization procedure. The WANNIERTOOLS code45 was used to investigate the topological features of surface state spectra. The high-pressure electronic band structure of AuTe2Br were also calculated with experimental structure (Expt.) and fully optimized lattice parameters (Opt.), respectively. The cell parameters of AuTe2Br under lower pressure for calculations are displayed in Supplementary Table 1.

High-pressure structure calculations for AuTe2Br

Beyond the pressure where a pressure-induced structural phase transition takes place, we performed a high-pressure crystal structure searching of AuTe2Br by using an in-house program called MAGUS (machine learning and graph theory assisted universal structure searcher), which is accelerated by the employment of Bayesian optimization and graph theory. This method has been successfully applied in many systems under high pressure, such as compounds inside plants and layered materials46,47,48,49. The structural optimization and electronic structure calculations were carried out within density functional theory using the projector augmented-wave method as implemented in the VASP. We chose s1d10, s2p4 and s2p5 as the valence electrons for Au, Te and Br respectively while using the GGA in the PBE exchange-correlation functional. The plane-wave cutoff was set as 340 eV and the Brillouin zone (BZ) was meshed choosing the gamma-centered Monkhorst–Pack approximately 2π × 0.025 Å−1.