Abstract
As high-speed railways continue to be constructed, more maintenance work is needed to ensure smooth operation. However, this leads to frequent accidents involving maintenance workers at the tracks. Although the number of such accidents is decreasing, there is an increase in the number of casualties. When a maintenance worker is hit by a train, it invariably results in a fatality; this is a serious social issue. To address this problem, this study utilized the tunnel monitoring system installed on trains to prevent railway accidents. This was achieved by using a system that uses image data from the tunnel monitoring system to recognize railway signs and railway tracks and detect maintenance workers on the tracks. Images of railway signs, tracks, and maintenance workers on the tracks were recorded through image data. The Computer Vision OpenCV library was utilized to extract the image data. A recognition and detection algorithm for railway signs, tracks, and maintenance workers was constructed to improve the accuracy of the developed prevention system.
1. Introduction
With increasing construction of new high-speed railways, more railway maintenance work is necessary to ensure seamless operation. This, however, is accompanied by the frequent occurrence of accidents involving maintenance workers on the tracks.
According to a press release [1] from the Ministry of Land, Infrastructure and Transport, from 2007 to 2017, the number of railway accidents decreased by 14.6%, although the overall length of railway tracks had increased. However, the number of casualties continues to increase every year. The victims of such casualties are predominantly maintenance workers who are hit by a running train.
According to the Korea Transport Safety Authority, the majority of domestic railway accidents are either railway casualty accidents or railway safety accidents. The casualty accidents involve railway track workers accounting for about 60% of all accidents in a year. This suggests that workers are not adequately protected by existing safety management systems. In addition, a report from the Korea Transport Safety Authority shows that the fatality rate for domestic railway workers is approximately three times that for the leading European countries. Accordingly, appropriate railway safety measures are urgently required. As new high-speed railways are constructed and existing railways are improved, more maintenance work is required; thus, maintenance workers are more frequently exposed to unsafe situations that often result in accidents. When a maintenance worker is hit by a train, it almost always results in a casualty, which is a serious concern. The main safety measures for maintenance work are installing signs—which are expected to be recognized by train drivers—to mark a work zone and allocate a safety guard who notifies train drivers of the work zone. In other words, the existing safety measures rely heavily on human capacity. Unfortunately, it is entirely possible for a train driver to miss a work zone sign, and safety guards at the track side can also be exposed to accidents, along with maintenance workers. Thus, these measures are essentially incapable of preventing railway accidents. With the objective of preventing railway accidents, this study attempted to develop an improved safety system for maintenance workers on the tracks by utilizing the tunnel monitoring system installed on trains and railway vehicles. Image data of the tunnel monitoring system are obtained from PTZ (Pan-Tilt-Zoom) cameras installed throughout a train. The tunnel monitoring system monitors detect electric car lines and railway tracks in real time, as the train is running. This study involved constructing an algorithm for extracting images of railway signs, tracks, and track workers from the image data of a tunnel monitoring system by means of Computer Vision OpenCV library and recognizing those images. As the proposed method can detect track workers, tracks, and work zone signs as objects, it is expected to provide train or railway vehicle drivers with track information in advance, thus alerting them to the possibility of an accident.
This study was conducted on trains running on the Shinbundang Line. The total length of this line is 31.1 km, extending from Gangnam station in Seoul to Gwanggyo station in Gyeonggi-do. This study utilized images stored by the tunnel monitoring systems in trains running on the Shinbundang Line. Image data obtained between April 15 and 21, 2019, were used in this study. Images were stored in the tunnel monitoring system for one week.
1.1. Literature Review
The existing management methods for railway maintenance work can be classified as follows: (i) GPS-based notification systems that inform maintenance workers of a train or railway vehicle approaching a work zone near the track. However, these systems are inefficient for underground or tunnel sections. There are also problems related to communication fees and security licenses. (ii) Wireless communication system that alerts workers to approaching trains or railway vehicles by means of infrared sensors. These systems require a separate detector for approaching trains or railway vehicles to be installed. Moreover, as the number of work zones increases, the cost of installing detectors also increases. (iii) Frequency-based methods to inform railway maintenance workers of the approach of a train or railway vehicle. As these methods require a separate frequency transmitter/receiver to be installed, they are expensive to construct and maintain. (iv) A safety fence is installed for works that are carried out near tracks on which trains and railway vehicles are running. The safety fence should be installed only in a work zone, and a safety manager and other staff should be available to warn maintenance workers of an approaching train or railway vehicle. This method becomes impractical when there are several work zones.
The four aforementioned railway safety measures and the studies that have been conducted on them have common limitations regarding human error. These existing systems are dependent on the working conditions of maintenance workers and, thus, have drawbacks related to human error and efficiency. As workers need to attach an alert device both on the safety helmet and on their body while they are working with their tools, the efficiency of maintenance work is severely affected. Among existing studies on this issue, in order to prevent casualty accidents caused by large construction machines, Nieto developed an alert system in which GPS receivers are attached to large construction machines, and an alarm is sent to all workers when such a machine is approaching [2]. As each large construction machine is equipped with a camera system and a GPS receiver, if there is a worker in a predetermined work zone, an alarm is sent to the driver so that an appropriate action can be taken. Teizer proposed a system (that uses radio frequency (RF)) that sends an alarm to the driver of a construction machine and to a construction worker when the machine is approaching the worker, thereby preventing an accident [3]. RTSA (2012) developed automatic track warning systems to enhance the safety of track workers. Saito’s system utilizes GPS to send an alarm to track workers through a mobile radio when a train approaches [4]. Hjort proposed an electronic data-transmission software program (ETW) using GPS in order to improve the safety of track workers [5]. D’Arco demonstrated that GNSS (Global Navigation Satellite System) produced fewer time and distance errors than GPS in railway maintenance sites. The GNSS improved the error of GPS in the measurement of distance. Besides, the GNSS achieved a higher estimation accuracy than GPS by combining relative distance estimates, and each track worker had to wear a receiver. Eirini Konstantinou (2019) eliminated noise with Kalman Filter, an algorithm for computer vision, and suggested using support vector machines (SVMs) to track, control, and monitor workers’ locations. Mingyuan Zhang (2020) proposed a method to assess the safety level of construction workers based on computer vision and fuzzy reference, noting that construction workers have accidents in the environment of the construction site.
Existing studies have focused mainly on alerting workers to approaching trains. However, the working conditions (such as noise) near tracks and unauthorized works continue to cause accidents during maintenance works. Maintenance workers at the track side and a train driver cannot be relaxed until the train completely passed through the work zone. All of them are under heavy pressure. This study aimed to improve the environment of maintenance work done either on or near railway tracks on which trains would be running. To achieve this, an algorithm was developed to detect objects on or near the tracks, thereby avoiding accidents and facilitating the safe passage of a train through the work zone.
This study presents a method that is different from those used in earlier studies. The target train of this study was equipped with PTZ (Pan-Tilt-Zoom) cameras both at the front and the back. The PTZ camera-based object image detection data were utilized to show objects such as maintenance workers, tracks, and work zone signs to the train driver. This study developed an evacuation system that gives such railway track information to the train driver and helps the workers and the train avoid any contact. This would help the driver take appropriate action to avoid an accident.
This study is distinguishable from earlier studies in the following three ways.
First, text information of maintenance (work) signs was detected from PTZ camera images and provided to the train driver in advance. This study focused on the fact that the maintenance work signs should be installed 500 m before the work zone.
Second, the railway track was recognized and maintenance workers on or near the tracks were detected as objects by using PTZ camera images. Information thus obtained was provided to the train driver.
Third, the region of interest (ROI) for the recognition of railway tracks was set in order to detect other objects, in addition to maintenance workers. Information was provided to the train driver in real time so that the train could pass safely through the work zone.
Intel’s OpenCV library and OpenCV_Python 3.6 were used for image processing and object detection.
2. Methods
This study constructed three algorithms by utilizing the OpenCV library. To construct these algorithms, a license-plate extraction algorithm, a lane-detection algorithm, and a vehicle and object-recognition algorithm were utilized. These algorithms are being actively studied in the field of intelligent transport system. The details of the method of this study can be summarized as follows. First, existing studies on text and license-plate extraction algorithms were reviewed. Considering that maintenance (work) signs are installed 200 and 500 m before the work zone, an algorithm was constructed, which extracts the text “500 m from work zone” and provides it to the driver of a train or railway vehicle before the train or vehicle enters the work zone. Second, the lane-detection algorithm, which had been extensively studied in the field of intelligent transport service, was utilized to construct a railway-track-detection algorithm. Third, the vehicle and object recognition algorithm of the intelligent transport service was utilized to construct an object-detection algorithm to detect maintenance workers near tracks.
2.1. Algorithm to Detect and Recognize Railway Signs
For sign recognition, there are two methods: shape recognition and color recognition. As the shape-recognition method is highly likely to capture similar shapes in the background, the recognition efficiency is poor and the recognition itself takes a long time. However, this method has a faster processing rate than the RGB color-recognition method and is less affected by the surroundings. Accordingly, a template-matching algorithm was adopted as the railway-sign recognition and detection algorithm, as it is less affected by the surroundings of railway tracks and can quickly detect and recognize specific signs (work sign).
Template matching is a technique for finding a specific image from an original image. In this study, the original image and a target image were processed by using gray scale. Subsequently, the target image was detected and recognized by using a specific red-color box. The template-matching function was used to recognize a work sign in a railway sign image. The process of template matching is presented in Figure 1.
The template-matching function was effective in detecting a work sign image using the original image. However, although this matching function could solve the problem of translation, recognition of rotated and scaled objects proved to be difficult, even with template rotation and scaling. Accordingly, this study utilized the RANSAC (Random Sample Consensus) algorithm, which accurately filters the matching result of key points between two images. The RANSAC algorithm assumes the existence of homography transformation between two images. This algorithm filters out incomplete matching results and retains only the results satisfying the motion model between two images [6]. The principle of the RANSAC algorithm is to create a model by random sampling from the data and determine how different the model is supported by, i.e., how many data have a distance from the model less than a constant value (T). The RANSAC algorithm is applied in the following order: Two points are selected randomly. F (x) of the straight line passing between the two points is obtained. The number of datasets is calculated, in which the distance of the above F (x) is T and less. In case is larger than the saved , a new is saved. After the above process is iterated N times, the optimal is returned to F (x). A result is derived by applying the least-squares method to datasets satisfying F (x). Here, N is to be selected so that at least one dataset among all available datasets can satisfy the probability Q consisting of a model and inlier (appropriate point). If the probability of all data being inlier is u, the probability of all data being outlier is . From this, N can be calculated as follows:
The above process is iterated N times to determine the ultimate model [7]. When images were rotated and then matched by means of the RANSAC algorithm, target images that had not been shown by template-matching could be detected. After rotating the image picture using the RANSAC algorithm, the image was not detected in the template matching, but the RANSAC algorithm was detected, as shown in Figure 2.
2.2. Algorithm to Detect and Recognize Railway Tracks
The majority of the existing studies on lane detection have used conventional cameras to acquire images and survey the road in front and set it as ROI, before applying the lane-detection algorithm [8]. The lanes are then detected using edge detection [9], Hough transform [10], template matching [11], and so on. Template matching is the most widely used technique. This method detects lanes by constructing a top-hat filter, which utilizes the brightness difference between lane and road and applies the corresponding template to the ROI. However, this method is not as effective in detecting curved lanes as it is in detecting straight lanes. A few attempts have been made to solve this problem by dividing an image into multiple sections and applying the Hough transform to each section [12]. Accordingly, this study constructed a Hough transform algorithm to detect railway tracks. Unlike road lanes, railway tracks cannot be distinguished by colors. For this reason, various methods, including the Gaussian filter, were used to construct an algorithm to detect and recognize the features of railway tracks.
There are various color models, such as RGB, YUV, and HSV. As road lanes are clearly distinguished by yellow and white, they are suitable for a color model. On the other hand, railway tracks are difficult to distinguish by colors. Accordingly, various color modes were applied, and the HSV model was selected to detect railway tracks. HSV is a color space that expresses images by hue (H), saturation (S), and value (V). The darkness and lightness of a color are expressed by the saturation channel, and the brightness is determined by the value. The HSV color space does not indicate a combination of colors; it indicates the color itself. It thereby achieves good intuitiveness. In case an object needs to be detected from an image by using colors, the HSV space seems to be more appropriate than the RGB space [13]. Railway track images were converted to the HSV color space using the OpenCV library, and the railway tracks were extracted from HSV. All objects other than the railway tracks were colored black; Figure 3 shows the result. There was a loss of railway track. To obtain a clearer image, an ROI (Region Of Interest) was set. By setting the ROI, the irrelevant objects were expressed in black, and, thus, the railway track images were detected and extracted. Straight railway tracks were clearly recognized in images. However, all the tracks were not straight. Thus, HSV alone was not sufficient. To solve this problem, a different HSV color model and the edge detection algorithm were used in conjunction.
Among various edge detection methods, the Canny edge operator is most widely used as it is the most clearly defined. This method is recognized as the best optimized edge detection method from the following aspects, which are the conventional criteria for evaluating the performance of edge detection operators: efficiency of edge detection (Good Detection), locality of edge detection (Good Localization), and single response to an edge [14]. This study utilized Canny edge detection to address the insufficiency of HSV-based detection. However, it was necessary to recognize curved tracks in a different way from straight tracks. Accordingly, a Hough transform algorithm was implemented to detect representative lines, which were then applied to curved tracks.
2.3. Algorithm to Recognize Maintenance Workers
In the field of intelligent traffic service, several algorithms have been developed to detect pedestrians, and several other methods are still being studied. This study used HOG [1] and SVM [2] algorithms, which are most widely used and verified in the field of intelligent traffic service, to recognize maintenance workers on tracks. The details of the process were as follows: first, an image was inputted for recognition, and a feature vector was extracted by using the HOG feature. After that, a pretrained SVM was used to distinguish maintenance workers on the tracks. In the next step, images for training were inputted. After a HOG feature vector was extracted from the inputted training images, the SVM was trained, and training data were extracted and then utilized to recognize workers. Generally, maintenance workers always wear a uniform during work. If this feature is extracted, classified, and detected, the processing rate may be accelerated according to image background, and the workers can be recognized more quickly.
The flow chart for recognizing maintenance workers at the tracks using HOG descriptor and SVM algorithms is presented in Figure 4.
When only the HOG descriptor algorithm was implemented, only one of two workers at the trackside was detected. To address this and accurately detect workers at the tracks, the features of workers were classified using the SVM classifier.
2.4. Experimental Application and Evaluation of Algorithms
After algorithms were constructed based on images of railway signs, tracks, and maintenance workers, they were applied to real images for verification. For images of actual railways, the image data stored by the railway image recording device of the Shinbundang Line were used. The algorithms were applied to the image data and were verified. This study attempted to recognize and detect railway signs, tracks, and maintenance workers while focusing on the safety of the workers. Template matching, RANSAC, and OCR were implemented for railway sign recognition. Color transform HSV, Canny edge, Gaussian blue, ROI, and Hough transform algorithms were applied to railway track images. An HOG descriptor and SVM model were implemented for detection of maintenance workers at the tracks by using the OpenCV library. Color images having a resolution of 1920 × 1080 pixels were converted to 800 × 600-pixel images to evaluate the constructed algorithms. The test images were trackside images captured either during daytime (in the open) or in a tunnel.
2.5. Application of Algorithm to Image Data
The recognition and detection of railway signs were evaluated using images stored by the image-recording device of a train running through tunnels on the Shinbundang Line. Figure 5 represents the application of algorithm to image data. To detect maintenance workers at the tracks, a morphological operation was used to remove noise, detect outlines, and apply the Gaussian blur. Moving objects in binary images were detected and recognized and were marked with red squares. Initially, there were several errors in detection. However, with continued use, the maintenance workers, who were classified by using the SVM classifier, could be recognized and detected.
(a)
(b)
(c)
3. Results of Algorithm Verification
To numerically evaluate the algorithms, this study adopted the concepts of precision and recall, which have been used for performance verification and evaluation of object-recognition and detection algorithms in several studies.
Precision is a measure of accuracy. Precision is the ratio of true detections to all detection results. It can be expressed by the following equation:
Here, TP stands for true positive, which indicates accurate detection, while FP stands for false positive, which indicates incorrect detection. Hence, precision is the percentage of the accurate detections among all the detections made by an algorithm. If an object detection algorithm detects five objects, of which four are TPs, the precision is 4/5 = 0.8.
Recall denotes a detection rate or a recall rate. In other words, it is the ratio of true detections to all the targets. Recall can be expressed by
Here, FN stands for false negative, which indicates objects that are to be detected but have not been detected yet. TP (true positive) refers to a case where a target object is accurately detected. FP (false positive) refers to a case where a nontarget object is wrongly recognized or detected. FN (false negative) refers to a case where a target object has not been recognized and detected. TN (true negative) refers to a case where a wrong thing that is not a target is accurately sensed and has not been recognized and detected. The above classification can be summarized in Table 1.
It is not sufficient to use only precision or only recall for performance evaluation of an object-detection algorithm. Let us assume that there are 10 objects and four out of 5 objects are correctly detected. Then, precision = 4/5 = 0.8 and recall = 4/10 = 0.4. Precision indicates good performance but recall does not. It is noteworthy that the values of precision and recall are always between 0 and 1, and, when the precision is high, the recall tends to be low and vice versa. Accordingly, it is almost the same to evaluate the performance of an algorithm using either of these parameters. It is necessary to apply both of them for evaluating an algorithm accurately. In this regard, the precision-recall curve and AP are needed. Furthermore, there is a criterion to judge whether an object has been detected correctly: this criterion is the intersection over union (IoU). This study utilized the performance verification index for object-detection algorithms, which was proposed by Everingham [15]. This method is IoU (Intersection over Union). As shown in Figure 6, let us assume that there is an image labelled with a ground truth boundary box. The ground truth boundary box wraps the object that is to be recognized and detected. When the ground truth box of the image was not given, the boundary box was detected by an object-recognition and detection algorithm, as follows.
(a)
(b)
IoU measures the area of overlap between a recognized and detected boundary box and the ground truth boundary box and then divides the overlapped area by the union area. The equation is presented below. If the IoU value is 0.5 and above, the result is judged to be true. Otherwise, if the IoU value is less than 0.5, the result is judged to be false.Here, R is the boundary box detected by algorithm and G is the ground truth.
The performance of the algorithm for recognizing and detecting railway signs was verified by analyzing image data in real time. The data were acquired by the railway track and tunnel-monitoring system. The focus of the performance verification was whether the algorithm could recognize and detect railway signs while the train was running at 50 km/h, 70 km/h, and 90 km/h in a tunnel.
The number of false negatives is that of railway signs that were not recognized and detected. The number of false positives is that of fluorescent lights or trains on the opposite side, which were recognized and detected. In other words, objects other than railway signs were also recognized and detected. The detection rate is the number of detected railway signs among all the railway signs. When a train was running on Shinbundang Line, the algorithm was evaluated using the image data of the real-time tunnel monitoring system.
When the recognition and detection algorithm for railway signs was applied for 0–50 km/h image data, there were no false negatives. However, there was one false positive where a fluorescent light in the tunnel was detected instead of a railway sign. When the train was moving at 51–70 km/h, there was one false negative, and there were three false positives. The traffic control system at the trackside and fluorescent lights in the tunnel were wrongly detected. When the train was moving at 71–90 km/h, there were two false negatives and four false positives. Table 2 presents the results of false negatives and false positives.
The algorithm for recognizing and detecting maintenance workers cannot be verified according to the velocity of train. For this reason, this object-detection algorithm was verified by classifying cases as follows: first, the workers were scattered or grouped (gathered). Second, they were gathered in the longitudinal direction. And third, they were gathered within a facility on the ground as shown in Table 3.
Although a worker was recognized and detected, when the object detection IoU was <0.5, the result was false. When the workers were scattered (i.e., they were a certain distance away from each other), the algorithm showed neither any false negatives nor any false positives. However, when the workers were arranged in the longitudinal direction, false negatives were obtained. In addition, when the workers at the trackside were grouped together, their faces, arms, legs, and bodies were concealed such that the number of the workers could not be accurately detected in some cases. Although false positives and false negatives were obtained, the algorithm showed performance indices of 0.5 and above. Thus, it can be concluded that the algorithm for recognizing and detecting workers at the trackside performs sufficiently well.
4. Conclusions and Further Scope
Appropriate signs should be installed 200 and 500 m before a work zone, in order to alert train drivers in advance. Drivers are expected to pay careful attention while operating their trains or railway vehicles; a guard should also be available to send a signal to all drivers. However, not only maintenance workers at the trackside but also the guards are often hit by trains and become victims of casualty accidents. Accordingly, to prevent such accidents, this study utilized the tunnel monitoring system installed on trains to recognize and detect maintenance workers at the tracks. This study attempted to develop an algorithm to recognizing a work zone and warn or even stop an approaching train, in addition to the existing alert system that lets maintenance workers know of any approaching trains.
However, this study cannot perfectly ensure the safety of maintenance workers at the tracks. It has limitations that need to be addressed in the future. An image-processing library was used to recognize and detect workers at the trackside. However, not all trains are equipped with the tunnel-monitoring system. In addition, urban railways undergo maintenance work only at night, which imposes a time limitation. The functions, models, and library, which were used to recognize and detect railway signs, tracks, and workers, are not final solutions. As there are several ongoing research and development projects, various methods need to be considered. Nevertheless, this study is significant as it developed a new approach. Existing systems only alert maintenance workers to the approach of a train or a railway vehicle. The proposed system enables train drivers to recognize and detect a work zone in advance and to be prepared for an emergency. Therefore, this study contributes to enhancing the safety of workers at railway tracks.
This study has the following limitations that need to be addressed. First, the proposed algorithm of this study is difficult to generalize. In order to address this, various methods need to be applied and analyzed. Second, this system recognized and detected objects at the trackside (railway signs, railway tracks, and maintenance workers) by using only the image data of the tunnel-monitoring system. However, it is necessary to collect diverse data and extend the spatial scope of research. Third, this study collected and utilized only the data of Shinbundang Line as the image data of the tunnel-monitoring system. As urban railways are equipped with various tunnel-monitoring systems, the scope of research needs to be extended by using data of the tunnel-monitoring systems of other railway lines. In spite of these limitations, this study will contribute to preventing accidents caused by the mistakes of train or railway vehicle drivers. Moreover, this study provides a basis for future studies aimed at preventing maintenance workers from being hit by trains or railway vehicles.
Data Availability
Image data of the tunnel monitoring system are obtained from PTZ (Pan-Tilt-Zoom) cameras installed throughout a train.
Conflicts of Interest
The authors declare that they have no conflicts of interest.