The development of remote sensing sensor techniques allows us to now readily capture many types of indoor and outdoor scene images, which often include many weak texture regions with notable geometric distortions. Obtaining qualified matches from these difficult stereo images using existing methods is challenging. The recent achievements of deep-learning models have shown that the convolutional neural network (CNN) is adept at the image matching task. However, in practical applications, the following challenges remain: first, it is difficult to detect features in the weak texture regions of an image, and existing CNNs fail to extract discriminative image information from the quantized features of weak texture; second, as a result of the complex distortion across wide-baseline stereo images, it is difficult to match feature primitives detected in the image pair. To solve these problems, we propose the perspective invariant local feature transformer (PILFT) algorithm. Our method includes four main steps. (1) The affine scale-invariant feature transform is proposed to automatically extract the corresponding features from images, and then the perspective of the matched image is corrected to eliminate as much geometric deformation as possible. (2) The residual network is used to extract potential features from stereo images to obtain coarse and fine feature maps at different scales. (3) Using an attention mechanism, location and context information are added to the coarse level features, which are predicted by a dual-softmax function layer. (4) The features are precisely predicted on the fine feature map using the coarse reference, and the final matching results are determined by calculating the matching probability. A large number of experiments on wide-baseline weak texture images demonstrate that the proposed method has advantages over the existing algorithms in the number of matches, correct match rate, and matching accuracy. The pseudocodes of PILFT are available at https://github.com/KiltAB/PILFT.