To examine the association of baseline body mass index (BMI) and BMI change with cognitive impairment among older adults in China. The study included data from the Chinese Longitudinal Healthy Longevity Study, a national community-based prospective cohort study from 2002 to 2018. Baseline BMI and BMI change were available for 12,027 adults aged older than 65 years. Cognitive impairment was defined as Chinese version of the Mini Mental State Examination score lower than 18. Multivariable Cox proportional hazard model was used. Among 12,027 participants (mean age was 81.23 years old and 47.48% were male), the proportion of underweight, normal, overweight and obese at baseline was 33.87%, 51.39%, 11.39% and 3.34%, respectively. During an average of 5.9 years’ follow-up, 3086 participants (4.35 per 100 person-years) with incident cognitive impairment were identified. Compared with normal weight group, adjusted hazard ratio (AHR) for cognitive impairment was 0.86 (95% CI 0.75–0.99) among overweight group, whereas corresponding AHR was 1.02 (95% CI 0.94–1.10) in underweight and 1.01 (95% CI 0.80–1.28) in obese participants. Large weight loss (< −10%) was significantly associated with an increased risk of cognitive impairment (AHR, 1.42, 95% CI 1.29–1.56), compared to stable weight status group (−5% ~ 5%). In the restricted cubic spline models, BMI change showed a reverse J-shaped association with cognitive impairment. BMI-defined overweight, but not obesity, was associated with a lower risk of cognitive impairment among elderly Chinese adults, while large weight loss was associated with an increased risk. These findings are consistent with weight loss in the prodromal phase of dementia.