Some thyroid tumors that are cytologically diagnosed as benign may be pathologically diagnosed as malignant. Here, we investigated the long-term outcomes of patients with thyroid tumors with benign cytology, and the factors for malignancy. We retrospectively reviewed the cases of 3,102 patients with thyroid tumors >1 cm cytologically diagnosed as benign at our hospital during a 1-year period from January 2007. The median follow-up duration for all patients was 68.7 (range 0.0–168.7) months. Immediate surgery and delayed surgery were performed in 393 and 148 patients, respectively. Eventually, 541 (17.4%) of the 3,102 patients underwent a thyroidectomy, and 2,561 (82.6%) were observed without surgery. Among the surgically treated patients, the tumors of 525 (97.0%) and 16 (3.0%) were pathologically diagnosed as benign and malignant, respectively. There was no significant difference in age, gender, tumor size, serum thyroglobulin level at surgery, or the tumor volume-doubling rate (TV-DR) between the benign and malignant cases. Only the ultrasonographic findings based on our hospital’s classification system were directly and significantly linked to pathological diagnosis (p < 0.01). Among the tumors of the 667 patients who were followed without surgery for >10 years, 89.9% remained unchanged and 7.2% were reduced in size. Ultrasonographic evaluation provides important information for therapeutic decision-making regarding surgery versus observation for cytologically benign tumors.