Based on the mechanism of landslide dam failure caused by overtopping, a simplified mathematical model for simulating dam breach overtopping flow processes was proposed in our previous work. The model is composed of five main modules including hydrodynamic, sediment transport and erosion, headward erosion, breach lateral evolution, and breach side slope stability. However, the breach lateral evolution module is still based on the assumption existing in most available models that breach lateral spreading is linearly related to the undercutting, i.e., the lateral erosion rate is twice as much as the vertical undercutting rate in the process of dam breaking. This assumption is obviously lack of theoretical basis. Through a theoretical analysis of sediment initial motion on the river channel slope, a lateral erosion formula was deduced in this paper. Then the lateral enlargement model was developed to update the breach evolution module in the previously established simulation model. To verify the effectiveness of the modification, A case study was conducted in the same overtopping failure event of Baige landslide dam, which was located at the border of Tibet Autonomous Region and Sichuan province in China, took place on November 3, 2018. A good model performance was achieved with predicted flood volume, peak discharge and occurrence time within 10% margin of error compared with field monitoring data. Furthermore, a comparing with the previous model was carried out and the results obtained by the modified model applying the new lateral enlargement formula proposed in this study indicate that the predicting accuracy of the model can be improved significantly for the simulation of landslide dam overtopping failure.