Aiming at the problem that lung nodule image quality assessment (IQA) when lung is constructed by a matrix iterative model (IMR), we in this paper proposes a novel IQA model based on multi-modal visual features highly representative to lung nodules. In order to achieve a competitive IQA performance, our proposed model adopted a medical-record-guided deep architecture as a significant part to provide additional information for our lung nodule IQA architecture. In the first place, such significant deep model constructs feature vector for each lung nodule image input by the model based on Bi-LSTM and Seq2Seq. Subsequently, the deep features was fed into to a carefully-trained decoder to reconstruct the original input data. Next, the deep feature associated with the lung nodule images are inputted into an end-to-end IQA deep model based on the 3D CNN directly. Comprehensive experimental results on the medical image sets collected from two hospitals in mainland China have shown that the designed deep model can reflect the subjective quality of lung nodule image accurately, and achieves better overall medical IQA assessment performance than the other well-known and non-reference IQA methods.