The ‘state-of-the-art’ urban climate models have not been evaluated against dense meteorological networks under various weather conditions. In this study, we conducted high-resolution urban climate simulations in Beijing and investigated the relationship between their performances and urban canopy parameters (UCPs). The latest version of single-layer (UCM) and multi-layer (BEP) urban canopy models were tested separately. Results show that model performances are insensitive to rainfall events in summer, but change significantly with wind conditions in winter. Inclusion of UCPs has a limited impact on air temperature simulations. In terms of wind speed simulations, consistent overestimations by UCM and BEP are found in both summer and winter. The overestimation by BEP reduces significantly when UCPs are available, especially on windy days in winter. On the other hand, performance of UCM can degrade over urban areas with canopy parameters. Wind speed biases are found to correlate significantly with UCPs. The accuracy of wind speed simulations by UCM and BEP increases with urban fraction and building surface ratio. This indicates poor model performances over low-density urban areas, which requires enhanced aerodynamic parameterizations to better account for UCPs. This study reveals the limitation of current urban climate simulations and provides guidance for future model developments.