The COVID-19 pandemic has had significant health, social, and economic consequences internationally. While the pandemic has direct implications on infected patients and families, there is a need to examine the pandemic's effect on patients with non-COVID-19-related diseases. This study examines the impact of the COVID-19 pandemic on 272 rare disease patients with 89 distinct rare diseases in Hong Kong using a cross-sectional online survey between April 10 and April 29, 2020 from the patient and caregiver perspective. The pandemic has impacted patient's health status in 46%, service use patterns in 71%, mental health in 79%, daily living in 82%, social life in 92%, and financial status in 81% of patients. Patient's health status, medical and rehabilitation, and mental health were more impacted by the COVID-19 pandemic in the group of patients with any level of dependency according to the Barthel Index for Activities of Daily Living compared with that in the group of patients who are fully independent (p < 0.0001; p < 0.0001; p = 0.0420). This study is the first study to examine the impact of COVID-19 pandemic on the rare disease population in Hong Kong, and demonstrates the pandemic's effect on service and resource utilization, and patient's physical and mental well-being.