To assess the effect of zinc smelting on environment and human health, the concentration, distribution, sources and health risk of eight heavy metals (Pb, Cd, Cu, Zn, Cr, Ni, Mn, and As) were investigated from agricultural soils in a long-term Zn smelting area in Guizhou, China. Different health risk assessment methods and models were used to access health risks. The results show that the concentrations of all the eight heavy metals were significantly higher than their corresponding background values (p < 0.05). Pb, Cd, and As were the most contaminated elements, with 93.6%, 90.3% and 48.4% of agricultural soil samples higher than the risk screening values, respectively. Statistical analysis indicated that Pb, Cd, Cu, Zn, Mn, and As could be mainly accounted for Zn smelting activities, while Cr and Ni may be generally more controlled by natural sources. The geo-accumulation index (I_geo) suggested that the most seriously contaminating heavy metals were Pb, Cd, and Zn, with 96.8%, 90.3%, and 96.8%, respectively, of the soil samples classified as moderately to extremely contaminated. The non-carcinogenic health risk associated with Pb, Cd and As were observed for children, meanwhile, the carcinogenic risk due to As was found for both adults and children. Regardless of cancer risk or non-cancer risk, local children are at greater risk than adults. Therefore, Pb, Cd and As play the leading role to cause potential health risks in the study area, which need to be paid more attention and also effective control measures should be taken.