Regular medical records are useful for medical practitioners to analyze and monitor patient's health status especially for those with chronic disease. However, such records are usually incomplete due to unpunctuality and absence of patients. In order to resolve the missing data problem over time, tensor-based models have been developed for missing data imputation in recent papers. This approach makes use of the low-rank tensor assumption for highly correlated data in a short-time interval. Nevertheless, when the time intervals are long, data correlation may not be high between consecutive time stamps so that such assumption is not valid. To address this problem, we propose to decompose matrices with missing data over time into their latent factors. Then, the locally linear constraint is imposed on the latent factors for temporal matrix completion. By using three publicly available medical datasets and two medical datasets collected from Prince of Wales Hospital in Hong Kong, experimental results show that the proposed algorithm achieves the best performance compared with state-of-the-art methods.