Building thermal load prediction is of great significance for energy conservation in HVAC systems. Due to the visible and complicated time delay between influencing factors and building thermal load, ensuring the consistency of time variation between feature variables and load in prediction is challenging. In this paper, the time-shifting correlation of feature variables to building thermal load was quantified, and the bidirectional network structure was introduced to develop prediction models to solve the forecasting delay issue of algorithms. The performance of four load prediction models was compared using measured data obtained from a railway station in Tibet to discuss the superiority of bidirectional network structure in building thermal load prediction based on deep learning method. The results show that long short-term memory (LSTM) and gated recurrent unit (GRU) models have significant prediction delays, while bidirectional long short-term memory (Bi-LSTM) and bidirectional gated recurrent unit (Bi-GRU) models do not, which indicates the bidirectional network structural characteristics can better capture the trend of thermal load variations in time. Moreover, Bi-GRU has the best performance, with all prediction relative errors being less than 1%, which demonstrates that Bi-GRU has a great advantage in building thermal load prediction. In addition, the performance of seven feature variable sets was compared to further demonstrate the convincingness of time-shifting correlation analysis between parameters. The results show that the proposed quantitative time-shifting correlation analysis method can evidently solve the time-dependent problem of feature variables in building thermal prediction. Finally, we also discussed the impact of prediction horizon on prediction accuracy, which illustrates the 15-min forecast interval is more suitable for ultra-short-term building load prediction.