Due to the complexity of the tumor and a large amount of patient information, an intelligent system is used to filter and extract hidden information, which will be beneficial to make accurate diagnostic decisions. The treatment and prognosis of breast cancer depend on the tumor stage. The PET-CT image can clearly show the lesion area and lesion range, especially for advanced-stage patients. Images and blood tests are crucial for accurate staging, tumor monitoring, and providing guided treatment plans. Multi-source data collaborative analysis can mine hidden attributes to provide a more intelligent treatment plan. This paper proposes a framework for predicting the diagnosis of the patient’s disease by combining images and labeling parameters. The blood test data and image data of patients are filtered based on the establishment of the medical decision-making module. The module selects detection indicators that correlate with tumor staging for analysis and trains a prediction model to assist doctors in providing a second diagnosis. The proposed framework for breast cancer diagnosis was tested on 5470 patient data from three well-known hospitals in China. The test results indicate that it performs well in diagnosing cancer staging with a prediction accuracy of 0.88.