Cover crops have been implemented for decades as sustainable agricultural soil management practice. However, knowledge about the temporal and spatial variation in soil nutrients and the succession of microorganisms during soil restoration under cover crops maintained for a different number of years are still poorly understood. Here, using the 16S and ITS rRNA gene sequencing techniques, we investigated the evolution of microbial community structure, and the pattern of nutrient accumulation was analyzed in a chronosequence of white clover used as a cover crop for 4, 8, and 13 years in a semiarid orchard on the Loess Plateau, China. Clean tillage was used as the control. We found that 4 years of cover crop had little effect on the soil nutrients, while 8- and 13-year-old cover crops contributed greatly to the soil nutrients. The use of cover crops for 13 years effectively increased the bacterial alpha diversity and the relative abundances of Actinobacteria, Firmicutes and Ascomycota in the 0−20 cm soil layer. These results indicated that long-term organic input stimulated transformation of the dominant population of microorganisms into a population conducive to the decomposition of plant residues. Furthermore, bacteria were more sensitive to the implementation of cover practices than fungi. Notably, on the genus level, the evolution of the bacterial community structure was more stable, while the evolution of the fungal community structure was more disordered with an increase in the years of cover crop. The nitrogen components were the best environmental predictors of the state of the microbial community. Overall, this study adds to our understanding of the patterns of nutrient accumulation and microbial succession under different years of cover crops and has implications for the formulation of a sustainable development strategy in semiarid agriculture systems.