This paper presents a carefully designed train delay prediction model, called FCLL-Net, which combines a fully-connected neural network (FCNN) and two long short-term memory (LSTM) components, to capture operational interactions. The performance of FCLL-Net is tested using data from two high speed railway lines in China. The results show that FCLL-Net has significantly improved prediction performance, over 9.4% on both lines, in terms of the selected absolute and relative metrics compared to the commonly used state-of-the-art models. Additionally, the sensitivity analysis demonstrates that interactions of train operations and weather-related features are of great significance to consider in delay prediction models.