Buildings are currently responsible for 39% of global energy related carbon emissions. Thus, carbon reduction in the construction industry is crucial to global climate issues. This study contributes a real-time operational carbon emission prediction method for the early design stage of residential units based on a convolutional neural network (CNN). The CNN model was trained based on a dataset with 2000 real residential units focusing on typical cases in northern China, particularly in Beijing, which is a significant energy consumption area. The R² values of the operational-stage heating and electricity carbon emissions in the validation dataset reached between 0.91 and 0.98. The trained model based on this method can provide the real-time prediction of the carbon emission potential of proposed residential units to buttress architects to make scientific design decisions. The model was then validated on new cases that varied in the total area, rooms, orientation, window locations, and window sizes. The performance of the CNN model is verified by comparing the results with the output of an EnergyPlus model. While the variation trends of the output results of the two methods are found to be consistent, the CNN model results still have certain deviations from those of the EnergyPlus model. Consequently, the proposed method can reflect the variation of operational-stage carbon emissions due to the change of the floor layouts of residential units, and has a faster speed and higher convenience than the simulation-based method.