Mining activities have caused heavy metal pollution in Xikuangshan (XKS), China. This work aims to assess heavy metal ecological risk in XKS using a potential ecological risk (PER) index and detect the responses of the soil bacterial community under heavy metal stress at different pH levels. The PER index indicated that antimony (Sb), cadmium (Cd), and arsenic (As) posed the most serious ecological risk at the sampling sites. We compared the effects of heavy metals on the structure of the microbial community under acidic and neutral conditions. Our results suggested significant differences between microbial community responses to heavy metals in acidic and neutral soils. Zinc (Zn) and lead (Pb) were the main heavy metal factors affecting the bacterial community under acidic conditions, whereas the effect of Sb, As, and chromium (Cr) exceeded them in neutral soils. This was possibly due to the different characteristics of heavy metals and their interactions with soil properties. The main genus were positively correlated with Sb, As, Zn, Pb, and Cd in both acidic and neutral soils, which may due to increasing resistance under conditions of long-time pollution. The results of our structural equation modeling indicated that the variations in the bacterial community structure were mainly explained by heavy metals in acidic and neutral soils. The soil nutrients and pH also had significant direct influences on the bacterial community under neutral conditions, as well as indirect effects due to their impact on heavy metals in neutral and acidic soils. Several heavy metal-resistant bacteria can be used for remediation under acid and neutral conditions.