This study addresses the question of why industrial restructuring towards light industries or services sometimes fails to achieve carbon emission mitigation goals. By employing a new perspective of dividing industry segments into emission-dominating and non-emission-dominating ones based on Logarithmic Mean Divisia Index (LMDI) decomposition method, this paper analyses city-level carbon dioxide emission reduction performance under three distinctive industrial restructuring directions. Results indicate that carbon dioxide emission dominating segments are relatively fixed across cities, regardless of the various city types in China. The key point to achieve emission reduction through industrial restructuring is to identify and control the emission-dominating segments, instead of economic-leading ones. Besides, emission reduction performance of industrial restructuring from emission-dominating industry segments to services is better than that to non-emission-dominating ones. More importantly, industrial restructuring not involving output scale controlling of emission-dominating segments, or that recklessly rushing towards services are likely to fail the emission mitigation goal. This paper presents a strong international reference that merits cities facing policy hesitation over industrial restructuring directions while in pursuit of emission mitigation. It suggests that cities first focus on identifying the carbon dioxide dominating segments, of which the output scale should be controlled. For cities whose emission-dominating segments are not economic-leading ones, it is necessary to carry out industrial restructuring towards services or non-emission-dominating segments; while for cities heavily dependent on emission-dominating segments, energy efficiency should also be improved.