Technological progress in energy efficiency is generally considered as an effective measure to address climate change issues. However, the expected results of such progress might be impacted by the rebound effect. Employing the linear approximation of the Almost Ideal Demand System (LA-AIDS) model, this paper aims to measure the direct rebound effect of residential buildings, considering the differences in the rebound effect for residential buildings between urban and rural China over the period 1994–2016. The empirical results reveal the magnitudes of direct rebound effect for residential energy consumption. The anticipated energy savings in residential buildings cannot be fully achieved in either urban or rural areas, and even significant “back-fire” effect exists in rural China. More specifically, the magnitudes of the rebound effect for residential buildings in urban areas fluctuate between 79.43% and 110.00%, while that in rural areas increase from 115.28% to 120.40%. The main reason of rebound effect is that the energy demand for residential buildings would increase when energy efficiency improves. This study provides a significant reference for the formulation of more targeted policies to achieve the anticipated energy savings, such as development of energy efficiency education program for urban residents and clean energy project for rural residents.