Mainstream architectures of convolutional neural networks (CNNs) for image recognition follow the protocol of shrinking input data size to extract semantic information. However, poor overall accuracy (OA) may be achieved when dealing with small input size and low-resolution images with classic CNNs. This paper proposes a novel, deep CNN architecture called DRSNet for small patch size Landsat 8 remote sensing (RS) image recognition. A module called residual inception channel attention block is applied for feature extraction, which combines the advantages of Inception-ResNet and channel attention. Pooling layers are replaced with reduction modules to prevent representational bottleneck. Unlike existing CNN structures, DRSNet adopts upsampling steps before final pooling layers, retrieving lost information caused by previous downsampling steps. Compared with existing state-of-the-art CNNs, DRSNet achieves the highest classification accuracy in our RS data set. Experimental results show that our model improves OA by 2%–9%, accelerates convergence speed, and effectively reduces loss. DRSNet also exhibits impressive results and outperforms baseline CNNs in three public data sets, namely, EuroSAT, Brazilian Coffee Scenes, and UCMerced Land Use. The proposed network is practical for large-scale land surface classification using free, low-resolution RS data; hence, it could be useful for nongovernment organizations or governments of developing countries.