Habitat loss and fragmentation are critical threats that cause biodiversity loss. However, the driving factors of species diversity in fragmented habitats are complex. This study was conducted in the agro-pastoral ecotone of northern China, a region that has suffered from severe habitat fragmentation due to agricultural activities. Based on the plant species richness, patch characteristics (patch area, fractal dimension, and isolation), landscape characteristics (grassland area ratio within buffer zones with different radii (50–500 m) around each patch), and environmental characteristics (mean annual precipitation, mean annual temperature, altitude, extremely high temperature, extremely low temperature, slope, and aspect) of 82 remnant grassland patches, we first explored the appropriate buffer zone with the most powerful impact on species richness of remnant grassland patches (SRP). Furthermore, we analyzed the relationships between a series of predictors and SRP. Finally, we confirmed the critical driving factors of SRP through a random forest analysis. The results showed that the landscape scale with the most powerful impact on SRP was 150 m. The patch area, grassland area ratio of a buffer zone with a 150 m radius, mean annual precipitation, and slope all had significantly positive effects on SRP. Among these factors, patch area and patch isolation were the top two driving factors of SRP. Our study suggests that large and highly connected patches should be protected for the construction of natural reserves. Meanwhile, the habitat around patches within a suitable range should also not be ignored. We highlight that the island biogeography theory could be used to predict plant species richness of remnant grassland patches in the agro-pastoral ecotone of northern China.