Poverty affects many people worldwide and varies in space and time, although its determinants are geographical factors. This paper presents a case study from Hubei Province, Central China, investigating the spatial and temporal changes in poverty determinants at the county and village levels from 2013 to 2017. We investigated the variation in the spatial autocorrelation of poverty incidence at the two levels using global and local Moran’s I. We then explored the spatial and temporal variations of poverty determinants using the Lineman, Merenda, and Gold method. We found that the overall spatial autocorrelation gradually mitigated, whereas the local spatial pattern remained unchanged at both levels. Deeply poor areas were concentrated in the western part of Hubei Province and the southwestern part of Yunyang County. The effects of geographical conditions on poverty decreased across the study period, with the R² value decreasing from 85% to 73% at the county level and from 57% to 38% at the village level. Furthermore, the contribution of natural environmental factors to poverty slightly decreased at both scale levels, whereas the socioeconomic factors had a significantly increased effect on county-level poverty over time. By contrast, the factors that have a major effect on village-level poverty remained stable. The results might indicate that the implementation of various targeted poverty alleviation measures since 2013 have mitigated the restrictions of local geographical factors on poverty alleviation.