In the desert region of northwest China, the frequency of wind-sand disasters is high. All types of concrete buildings built in this area face severe wind erosion due to high wind speed, resulting in varying degrees of wind-erosion damage to concrete. To accomplish intelligent identification of concrete wind-erosion damage, a concrete wind erosion experiment was conducted in the laboratory, and a concrete wind-erosion damage dataset was generated under the interference of water stains, scratches, shooting distance, and background noise. This paper combined with transformer theory to improve YOLO-v4 and proposed an object detection algorithm called MHSA-YOLOv4 suitable for wind-erosion damage of concrete. The results demonstrate that MHSA-YOLOv4 exhibits improved object detection performance than YOLO-v3, improved YOLO-v3, and YOLO-v4. On the test set, ACC, Precision, Recall, and mAP of MHSA-YOLOv4 are 91.30%, 91.52%, 92.31%, and 0.89, respectively. MHSA-YOLOv4 can accurately identify wind-erosion damage of concrete images under different test conditions, which reflects strong robustness. The applicability of computer vision technology to the intelligent identification of wind-erosion damage on concrete has been verified.