The detection of the optic disk (OD) and fovea is crucial to the automatic diagnosis based on fundus images. This task is very challenging, especially when varieties of lesions exist. Traditional handcrafted feature-based methods are inaccurate, and deep learning based methods fail easily in abnormal cases. We propose a framework that simultaneously detects the OD and fovea based on deep convolutional neural networks. The original image is first preprocessed and then followed by pseudo label generation. These labels are then fed into a fully convolutional neural network with residual modules for localization of the OD and fovea. Polar transformation is then introduced to the segmentation of the OD. The proposed algorithm achieves a relatively high success rate for OD localization and a 100% success rate for fovea localization on several public datasets. For the segmentation of the OD, the proposed algorithm achieves a low overlapping error on several public datasets. Compared with previous work, the proposed method achieves promising accuracy and robustness, and it is useful for practical applications since it detects the OD and fovea simultaneously and completely.