This study proposed an assistive system to recognize the special character CAPTCHAs for the visually impaired community in the Chinese region. To improve the recognition precision, a convolutional neural network (CNN), which is named Captchanet for recognition, was proposed. Firstly, a ten-layer network architecture was designed and three improved training strategies were proposed for the deep learning model. Secondly, a customized Chinese character training set was designed using a novel and fast method, with the view of overcoming the limitation in labeled data collection and uneven data distribution. Finally, the experiments were conducted on the test set gathered from public websites to test the effectiveness of the proposed Captchanet. The statistical results demonstrated that the Captchanet has better classification performance and has obtained higher success rates of recognition than the well-known machine learning approaches and CNN-based approaches.