The profound effects of land-use types on soil microbial communities have been uncovered extensively in areas where land-use conversions are unidirectional. However, little is known about the response of soil microbes to land-use types in areas undergoing frequent bidirectional land-use conversions. Here, 36 paired cropland and grassland soils across soil types were collected in the agro-pastoral ecotone of northern China (APEN), and soil microbial communities were examined by high-throughput sequencing. According to the land-use history interpreted from remote sensing images from 2008 to 2017, the average frequency of conversion between cropland and grassland was 3 to 4 at all sites, and the duration of current land uses ranged from 1 to 9 years. Paired t-tests revealed that soil microbial diversity, measured as observed OTUs and Faith’s PD, did not differ significantly between cropland and grassland. In contrast, soil microbial community composition remarkably varied with the land-use type based on the nonmetric multidimensional scaling (NMDS) and analysis of similarity (ANOSIM). The above phenomena were also observed for different soil types (Kastanozems and Cambisols). The similarity percentage (SIMPER) analysis showed that the microbial taxa contributing most to the variation in the community composition were Pseudarthrobacter, Azotobacter, Bacillus, and Blastococcus, which are closely associated with the vegetation type (maize vs. grass). Meanwhile, there were no significant differences in soil properties between cropland and grassland, suggesting that vegetation types alone might drive changes in the community composition. Moreover, soil microbial diversity decreased with the duration of the croplands but did not change with the duration of the grasslands. The community composition between cropland and grassland differed more over a long duration (5 to 9 years) than over a short duration (1–4 years). Therefore, the longer the land-use type lasted, the more pronounced the differences in soil microbes between the two land-use types became. In conclusion, compared to soil microbial diversity, the community composition was more susceptible to land-use types under frequent bidirectional conversions, but both responses were enhanced with the duration of current land uses.