Agricultural expansion has led to a large-scale loss of natural vegetation and landscape fragmentation, seriously threatening biodiversity. In addition to directly encroaching on natural habitats, agricultural expansion can indirectly affect surrounding natural habitats. However, the indirect effects of agricultural expansion on natural habitats and their responses to landscape fragmentation processes are poorly understood. Taking Lishui city, China, as an example, changes in land use/land cover and habitat quality during 2000–2020 were assessed. Then, an agricultural expansion scenario was designed to separate the direct and indirect effects of agricultural expansion on natural habitats. Regression analysis and geographical detector methods were used to explore the relationships among the four landscape fragmentation processes (area, edge, shape, and isolation) and the direct/indirect impacts. The results showed that agricultural expansion resulted in a forest loss area of 376.04 km². Agricultural expansion led to a 6.08 % decrease in natural habitats, of which 43.84 % were direct effects and 56.16 % were indirect effects. Agricultural expansion indirectly impacted 14,209.06 km² of natural habitats, and this value was approximately 32 times larger than the area that was directly impacted (449.51 km²). Different landscape fragmentation processes had linear/nonlinear effects on natural habitats. The interaction effects of landscape fragmentation showed duality and nonlinear enhancement. Changes in area and edge contributed the most to the direct effects, but indirect effects were also affected by the shape of agricultural land and the isolation of forest. The agricultural land system balance policy was the main driver of agricultural expansion and landscape fragmentation. Sustainable agricultural landscape management is necessary to mitigate the impact of agricultural expansion on biodiversity. This study provides new insights for agricultural landscape planning and biodiversity conservation.