Soil microbes play essential roles in the biogeochemical processes of organic carbon and nutrient cycling. Many studies have reported various short-term effects of fertilization on soil microbes. However, less is known about the effects of longterm fertilization regimes on the rhizosphere. Therefore, the objective of this study was to explore how the soil microbial communities in the rhizosphere respond to different long-term fertilization strategies. Based on a 21-year field treatment experiment in Guizhou, China, we extracted phospholipid fatty acids (PLFAs) to determine the microbial community structure in both the non-rhizosphere (NR) and rhizosphere (R). Six treatments were included: no fertilizer (CK), mineral nitrogen fertilizer (N), N with potassium (NK), phosphorus with K (PK), NPK, and NPK combined with manure (MNPK). The results showed that total PLFAs under unbalanced mineral fertilization (N, NK and PK) were decreased by 45% on average in the NR compared with CK, whereas MNPK increased fungi and G^– bacteria abundance significantly in both the NR (by 33 and 23%) and R (by 15 and 20%), respectively. In addition, all microbial groups in the R under these treatments (N, NK and PK) were significantly increased relative to those in the NR, except for the ratio of F/B and G⁺/G^–, which might be due to the high nutrient availability in the R. Soil pH and SOC significantly regulated the soil microbial community and structure, explaining 51 and 20% of the variation in the NR, respectively. However, the rhizosphere microbial community structure was only significantly affected by soil pH (31%). We concluded that the soil microbial community in the NR was more strongly affected by long-term fertilization than that in the R due to the rhizosphere effect in the agricultural ecosystem. Rhizosphere nutrient conditions and buffering capacity could help microbial communities resist the change from the long-term fertilization.