Long-term passenger subscription is vital to the survival and operation of the customized bus (CB) system, which is a demand-driven and user-oriented transit service. A better understanding of passenger loyalty toward the CB service will help provide better operation. The urgent and outstanding issue is how to incorporate the unobserved heterogeneity in loyalty—in other words, how to reflect the effects of the frailty to terminate subscription. This study fills the research gap through an empirical study in Dalian, China. Three different survival models are developed to investigate the mechanism of subscription behaviors, among which the shared frailty model considering the unobserved heterogeneity is demonstrated to be the most appropriate. The results indicate that the historical purchase characteristics are the most important to CB user loyalty modeling and forecasting. Males are more sensitive than females to the number of intermediate stations because of the potentially increased uncertainty in waiting time related to the intermediate stations. The heterogenous frailties resulting from the heterogeneity of the perceptible service quality in terms of convenience and efficiency in subscribing/returning tickets and information availability in the progress of the CB system significantly contribute to user loyalty deviations.