The problem of heavy metal pollution of soils in China is severe. The traditional spectral methods for soil heavy metal monitoring and assessment cannot meet the needs for large-scale areas. Therefore, in this study, we used HyMap-C airborne hyperspectral imagery to explore the estimation of soil heavy metal concentration. Ninety five soil samples were collected synchronously with airborne image acquisition in the mining area of Yitong County, China. The pre-processed spectrum of airborne images at the sampling point was then selected by the competitive adaptive reweighted sampling (CARS) method. The selected spectral features and the heavy metal data of soil samples were inverted to establish the inversion model. An ensemble learning method based on a stacking strategy is proposed for the inversion modeling of soil samples and image data. The experimental results show that this CARS-Stacking method can better predict the four heavy metals in the study area than other methods. For arsenic (As), chromium (Cr), lead (Pb), and zinc (Zn), the determination coefficients of the test data set ($R_P^2$) are 0.73, 0.63, 0.60, and 0.71, respectively. It was found that the estimated results and the distribution trend of heavy metals are almost the same as in actual ground measurements.