1 Introduction

In the modern age of science and technology, the use of operation supporting systems is indispensable regardless of the type of operation. However, as the systems develop, it becomes important to improve humans’ abilities for efficient and accurate operation with them.

In 2018, vessel traffic services (VTSs) in Tokyo Bay, Japan, were integrated from five places (four port traffic control offices in Tokyo, Chiba, Kawasaki, and Yokohama and one Vessel Traffic Service Center in Kannonzaki) into one place (Tokyo Wan Vessel Traffic Service Center in Yokohama) based on the Third Transport Vision in Japan (JCG 2013, 2018a) (Fig. 1). In addition to the expansion of the VTS area and increasing work volume, technical assistance to improve the abilities of VTS operators is required. For example, technical assistance to improve the efficiency of individual tasks and speed up of skills learning appears to be important.

Fig. 1
figure 1

Integration of vessel traffic services in Tokyo Bay, Japan (From Shimbun 2017)

Full size image

Among VTS operations, vessel behavior monitoring starts with the detection of conditions that require attention (CRAs). Fast and accurate detection of CRAs from VTS monitoring screen can prevent the occurrence of the risk. To acquire this detection ability, it is important to first fully understand the task of detecting CRAs. Therefore, in our previous study, we conducted a traffic management simulation experiment for operators with different levels of ability; hereafter, this experiment is referred to as the “previous experiment”. The management and cognitive processes of the operators in the task of detecting CRAs were analyzed (see Song et al. 2018, 2019 for details). As a result, we found that inexperienced operators did not detect many CRAs because their abilities to predict the situation changes of CRAs were low.

Traditionally, the knowledge of the experienced operators was orally transmitted to the inexperienced operators. However, because the experienced operators have been learned the knowledge by themselves from the experience, it is quite difficult to actually convey them to others. Therefore, to improve the abilities of inexperienced VTS operators as quickly as possible, it is necessary to develop an advanced training method, by which the inexperienced operators can understand the prediction methods of experienced operators more easily and quickly.

In this study, to quickly improve the prediction abilities of inexperienced operators, we developed an education tool called vessel traffic routine (VTR) based on the cognitive processes of an experienced operator.

In respect of our study method, human cognitive processes are redefined into three stages of “situational awareness”, “situation judgment” and “decision making” based on the recognition primed decision model (RPDM) (Klein 1998). In addition, VTS operations are analyzed and evaluated based on this redefined cognitive process. We should notice that the primary purpose of our study is to establish a method for analyzing and evaluating VTS operators’ cognitive processes and to develop a cognitive training method for VTS operators.

The remainder of this paper is organized as follows. In Sect. 2, we describe the positioning and novelty of our study. In Sect. 3, an outline of the vessel behavior monitoring service is explained. In Sect. 4, we propose a method of the cognitive task analysis of VTS operators from the results of an experiment in the previous study (Song et al. 2018, 2019). In Sect. 5, we describe the developed VTR. In Sect. 6, we outline our traffic management simulation experiment for inexperienced operators to verify the effectiveness of VTR. Finally, in Sect. 7, we evaluate the verification results from the cognitive processes of inexperienced operators. Finally, discussions and conclusions of this study are shown in Sects. 8 and 9.

2 Literature review and positioning of our study

2.1 Existing studies on enhancing human ability to use the system and positioning of our study

Since recent years, in various traffic fields, there have been many studies on improving the humans’ abilities with the operation-supporting systems. The contents of these studies can be broadly divided into three categories: (1) proposition of humans’ abilities training or learning methods, (2) proposition of humans’ abilities supporting methods and (3) proposition of humans’ workload reduction method. In this study, we develop a training method for improving VTS operators’ abilities, thus our study belongs to the first category.

In rail traffic flow control, as a literature review of the first category, Vanderhaegen (2019) proposed learning tools based on a new concept of human and system inclusion and applied them to human driving tasks with systems. In air traffic control, as a literature review of the second category, Karikawa et al. (2011) emphasized that the capability of air traffic controller to flexibly respond to situation changes is a very important factor in the skills of air traffic controller and proposed a method for analyzing responding ability using cognitive system simulation (COMPASi) (Karikawa et al. 2008).

In VTS, literature reviews of the first category are shown as follows. Chen et al. (2013) demonstrated the usefulness of VTS operator behavioral assessment for establishing effective training methods. Mansson et al. (2016) investigated the perception of ship masters, maritime pilots, tug masters and VTS operators operating in Australia through qualitative research interviews for proposing measures to improve teamwork in the maritime traffic system.

Literature reviews of the second category are shown as follows. Swierczynski and Czaplewski (2015) proposed an estimation tool of navigational parameters to help VTS operators possible to obtain more accurate information. As mentioned in the next section, VTS operators use visual inspection, automatic identification system (AIS) and radar to grasp the marine traffic flow. Vessels are divided into with and without AIS. Through radar, it is possible to identify ships without AIS. In addition, if a target ship is specified using Automatic Radar Plotting Aids (ARPA) function attached to the radar, the track of the target ship can be accurately grasped by automatically measuring its direction and distance etc. (Kudou et al. 2013). Lin and Huang (2006) compared the characteristics of ARPA and AIS using an experimental method to suggest to VTS operators how to use the system efficiently. By proposing a new method for collision avoidance called the fuzzy logic method, it is possible to calculate the risk of collision between many ships that cannot be achieved by ARPA and to accurately predict the time and position of collision (Kao et al., 2007). In addition, it is possible to propose rudder steering procedures for collision avoidance and send appropriate alerts by using the fuzzy logic method. By building a fuzzy monitoring system with these functions, the decision-making ability of VTS operator for collision avoidance can be strengthened (Su et al. 2012; Kao et al. 2017). Li et al. (2016) used a hierarchical task analysis method to investigate the required information for VTS operators to provide services efficiently and appropriately. Park et al. (2020) proposed a reduction method of communication time by quantitatively analyzing VTS communication using queuing theory. Inter VTS Data Exchange Format (IVEF) is a standardization for the exchange of information between VTS and was defined by the International Association of Marine Aids to Navigation and Lighthouse Authorities (IALA). Park and Bang (2016) developed a system that enables simulations for IVEF to facilitate the exchange of information on maritime traffic. Park et al. (2012) proposed an authentication system that enhances the security of VTS systems using IVEF. Praetorius and Lutzhoft (2012) investigated user needs for risk management through observations and interviews at VTS Center to propose improvements to VTS operator's decision support system.

Literature reviews of the third category are shown as follows. To reduce alarm fatigue, Li et al. (2017) considered the human limitations that may be the root cause of alarm fatigue and proposed multiple new modules of VTS alarm system by analyzing user requirements using the qualification function deployment (QFD) method. Xu et al. (2020) proposed a work shift planning solution that would not tire VTS operator by analyzing the data of automatic identification system (AIS).

As mentioned above, we have been investigating existing studies on improving humans’ abilities with systems in various traffic fields in recent years. By classifying these existing studies, we can see that the studies of the second category are most occupied, but the studies of the first and third categories are relatively few in VTS field. It follows from this result of the literature search that it is worth doing our study, which belongs to the first category.

2.2 Existing studies on cognitive technology in VTS field and novelty of our study

To improve humans’ abilities with the systems, appropriate training methods are required. To develop an appropriate training method, it is necessary to analyze the task as a preparation (Tannenbaum and Yukl 1992). As a traditional task analysis method, we could collect samples to construct a training method based on human behavior modeling (e.g., Latham and Saari 1979).

Alternatively, Howell and Cooke (1989) suggested that more effective human ability training methods can be created by investigating the human cognitive processing procedure. LaFrance (1987) suggested that systematic acquisition of the expertise of experienced operators can be useful in determining training content. Kraiger (2002) noted that it is worth observing the changes in learners’ cognitive behaviors to evaluate the effectiveness of a training method. Therefore, we believe it is effective to learn the cognitive processing procedures of experienced operators and evaluate the learners’ cognitive behaviors for developing a training method of humans’ abilities.

In recent years, there have been few studies on cognitive task analysis and evaluation of VTS operation. Only studies on the classification of recognized information, VTS operation measuring methods based on recognized information (Wiersma and Mastenbroek 1998), and on the information display method for situation prediction (Kim et al. 2013) have been conducted. These studies focused on the information to be recognized but did not make it clear regarding how the information was recognized.

In this study, we not only focused on the recognized information during VTS operation but also on the processes to recognize the information of VTS operators, and developed a cognitive training method for VTS operators.

3 Outline of vessel behavior monitoring service

VTS operations are indispensable for smoothing the flow of maritime traffic and preventing the occurrence of accidents. In the case of aviation, the pilot cannot operate the airplane without the instructions of the air traffic controller, while in the case of the voyage, ships are slower than airplanes and the operators of ships can see the surrounding conditions. Thus, in principle, the operation of ships is entrusted to the ship side. However, in congested sea areas, there is a possibility that the operator of ships cannot confirm the surrounding conditions sufficiently and accidents are increasing due to the carelessness of the operators. In addition, due to the complexity of maritime traffic, it is possible that many accidents are caused by the operator not being able to see through the surrounding conditions. Against this background, it is important for VTS operators to detect CRAs faster and to provide information to vessels. Note that CRAs are situations to pay attention that include a potential collision risk in maritime traffic situations and consist of several vessels.

An overview of the types of VTS operations is given in the VTS manual of the International Association of Marine Aids to Navigation and Lighthouse Authorities (IALA) (IALA 2021). Each country formulates its own VTS operations based on the VTS manual. VTS operation in Japan is mainly composed of information provision, advice and instruction, and navigation control (JCG 2018b). These are conducted based on the vessel behavior monitoring service, which is the core of VTS operation, where VTS operators survey the maritime traffic situation by monitoring vessel behavior and confirming ship information from monitoring screens.

The vessel behavior monitoring service conducted by the Tokyo Wan Vessel Traffic Service Center covers the areas inside and around Tokyo Bay. Vessel behavior inside Tokyo Bay can be tracked by radar, automatic identification system (AIS), and TV camera, whereas vessel behavior around Tokyo Bay can only be tracked by AIS. AIS is a system that automatically sends and receives ship-specific data such as ship name, position, course, ship speed, and destination using very high frequency (VHF) radio. On the other hand, radar can uniformly grasp objects that reflect radio waves emitted from its own ship, thus it is possible to grasp ships that do not have AIS (Kudou et al. 2013). A VTS operator can provide information to vessels in both these areas by international VHF radiotelephone (Fig. 2) and ship phone (JCG 2018b).

Fig. 2
figure 2

VTS operator monitors the vessel behavior through the monitoring screens

Full size image

Figure 2 shows that a VTS operator monitors vessel behaviors through two monitoring screens. VTS operator checks the behavior of ships in the VTS area and tries to find CRAs mainly through the radar processing screen. VTS operator maintains the safety and efficiency of maritime traffic by transmitting four types of communication messages related to the observed CRAs, “information”, “warning”, “advice” and “instruction”. If the VTS operator finds a serious risk, a warning, advice, or instruction is sent to the concerned ships, whereas only information is sent to the concerned ships if the level of risk is lower. Note that before the operator sends messages of advice or instruction, it is necessary to obtain permission from a VTS supervisor who manages the entire area (IALA 2021). In this paper, the above task is called the CRA detection task.

4 Method of cognitive task analysis for VTS operators at different levels of ability

In the cognitive task analysis of our study, it is first necessary to identify the difference in cognitive processing procedures between inexperienced and experienced operators. An outline of this analysis method is given in this section.

It is possible to say that the objective of our research is to find the ethnomethodology of the worksite. Ethnomethodology means people's methodologies (Garfinkel 2012), which are made up and maintained by the members of society in their everyday life. The authors think that finding the ethnomethodology of the work is very important to explore what kind of knowledge and experience the operators are using and how the operators work, that is, operators’ work-specific abilities. In addition, a typical approach to the cognitive engineering analysis of operation is to conduct an experiment. Concretely, a simulation experiment is effective to observe the processes of operation and analyzing the features of the processing procedures of an operator. Therefore, we conducted the previous experiment, which was a traffic management simulation for operators with different levels of ability. Note that in this study, we conducted a traffic management simulation experiment for inexperienced operators (see Sect. 6 for details).

In air traffic control, Inoue et al. (2012) proved that methods such as recording operations and conversations and interviewing air traffic controllers are useful in order to find out the existence of some rules (ethnomethodology) based on experimental methods. In addition, if the number of subjects is very small, ethnomethodology approach also can be used, for example, as shown in the ethnomethodology approach for analyzing interactions in a math classroom (Ingram 2018). In this study, we decided to analyze the detail of the process for each subject's task from the start to the end.

In visual search, Williams (1966) suggested that the probability of finding a target depends on two factors as follows: the visibility of the target and the speed of information input. That is, the easier the target is and the slower the target changes, the faster the target will be found. Duncan and Humphreys (1989) also suggested that the greater the similarity between the target and the non-target, the more difficult the visual search will be. As shown in Fig. 3, on the radar processing screen of VTS, all the vessels appear to be one thin line. In addition, the positional relationship of the ship changes realistically. Therefore, we can see that it is might be difficult for VTS operators to detect CRAs on the radar processing screen from the viewpoint of visual search.

Fig. 3
figure 3

Targeting sea area in this experiment (display of the occurrence position of CRAs 1–9)

Full size image

To observe the CRA detection task of the VTS operators, it also might be a good idea to observe visual searching of the operators. Recently, observation of eye movements during visual search is carried out to investigate cognitive processes (Rayner 2009). In the field of aviation and voyage, there are some examples in which eye movement was used as a usability evaluation tool to evaluate the usefulness of electronic maps and charts (Ottati et al. 1999; Arslan et al. 2020). Watson et al. (2010) examined the relevance between strategy instructions for visual search and the eye movements made during search and recommended adopting a passive cognitive strategy to make visual search more efficient. This is because the advantage of using a passive strategy is that the information in the staring can be used to detect the target more efficiently than the information from the broader attention. Therefore, in this study, we analyze the cognitive process of VTS operators using eye movements in order to extract cognitive strategies for detecting CRA.

In summary, the authors record the eye movements, operations, and conversation contents of VTS operators based on a simulation experiment at the site when the CRA detection task is performed. In addition, after the experiment, by conducting an interview with the operators, the work method (ethnomethodology) are analyzed, and in particular, the cognitive processes of the operators are analyzed and extracted.

4.1 Outline of previous experiment

The purpose of the previous experiment was to collect data on the CRA detection task performed by VTS operators and to clarify the management and cognitive processes of the operators based on the collected data (Song et al. 2018, 2019).

In this experiment, three operators (inexperienced, mid-level, and experienced) were asked to perform the CRA detection task. By comparing the management and cognitive processes of operators at different levels, the differences between the operators were analyzed.

The management process of a VTS operator was extracted based on which related ships of the traffic situation they check on the radar processing screen and the information provided to the target ship when they perform the CRA detection task. Furthermore, the cognitive process of a VTS operator was extracted by analyzing how and which situations are recognized from the extracted management process. Therefore, three kinds of human data for each subject were recorded: (1) eye movement data of the subjects on the monitoring screen, (2) operation data of the subjects on the monitoring screen, and (3) contents communicated with concerned ships by the subjects (see Figs. 4 and 6).

Fig. 4
figure 4

Structure of the experimental system

Full size image

To extract the cognitive processes of a VTS operator, eye movement data were applied. By looking the sea area watched by the operator, it can be inferred if they identified the situation occurring in that area. In addition, it is possible to estimate whether or not the type of occurrence situation has been identified by looking at the method of watching at the sea area. For example, if the target sea area with the screen size is watched, we can estimate that the operators only noticed the occurrence situation, but if the target sea area while zooming the screen is watched, we can estimate that the operators have been identified the type of occurrence situation. From the length of time, the operator watched the target area, the ease of dealing with the situation occurring in the target area can be inferred. For example, when the target area was watched by the operator for a long time, we can infer that it was difficult for them to determine what kind of situation occurred. Therefore, in the previous study, eye movement data were analyzed based on three factors: (1) gazing area, (2) distance between gaze areas and (3) gazing time in the target area. In other studies, to extract the cognitive processes of chess players and readers, the same factors of eye movement data were considered. As a result, the cognitive processes were successfully extracted by enabling the distinction between cognitive states depending on the working states of the subjects (Chase and Simon 1973; Simola et al. 2008).

Another method for extracting cognitive processes is to have the subject speak their thoughts (e.g., Green and Gilhooly 1990; Soraji et al. 2012). This involves presenting a subject with a task and having them talk about what they were thinking while completing the task. One of the difficulties in applying this method to VTS operation is that it is difficult for the VTS operator to communicate their thoughts continuously because they must provide verbal information to concerned ships during operation. It is therefore reasonable to use eye movement data to elucidate the cognitive processes of VTS operators. In addition, in the data collection method of the experiment, as shown in Fig. 4, the cognitive processes are analyzed by collecting all cognitive data including not only the eye movement data but also VTS operators’ operation data and conversation data. Therefore, we can consider that the analysis results in this study are objective and valid.

4.1.1 Experimental scenario

Figure 3 shows the target sea area of the experiment surrounded by a black dashed line. As shown by this figure, the target sea area of the experiment is centered at the Uraga Suido Traffic Route and Nakanose Traffic Route, where are surrounded by pink dashed lines. This sea area is where a large number of container ships and ships carrying dangerous goods pass through to enter Tokyo Bay.

Because the highest traffic volume occurs in the morning, the time frame of the experiment was set to 20 min between 7:00 and 8:00 am. In addition, as Fig. 3 shows, a total of nine CRAs were included in this scenario. Among these CRAs, there are five of high importance (CRAs 1–5 in descending order of frequency of occurrence). In addition, this figure shows that in the experiment, a triangle or circle denotes a ship, and the direction and length of the thin line connecting the triangle or circle denote the course and speed of the ship, respectively.

As described above, in this experiment, one scenario included multiple CRAs. For this reason, we believe it is possible to observe the differences in cognitive processes between the subjects using only one scenario.

4.1.2 Experimental system

Figure 4 shows the configuration of the experimental system. To record the management actions executed and information handled by the operators as much as possible, three video cameras were used to observe the overall operation, including the contents of radio communication and conversation between the operators, radar processing screen, and ship entry schedule, at the same time. In addition, to grasp the contents watched on the radar processing screen, eye movement data were acquired and recorded using a glass-type eye tracker (Takei Scientific Instruments Co., Ltd., 2020), which tracks the line of sight.

Figure 5 shows an example of the content tracked by the eye tracker. The movement of the line of sight is displayed by connecting the line of sight (blue line) from one gazing point (pink point) to the next. Moreover, the size of the gazing point (red circle) indicates the length of the gazing time. In this experiment, when the gazing point stays in an area where a CRA occurs, it is assumed that the subject is checking that CRA.

Fig. 5
figure 5

Line of sight display on the eye tracker screen

Full size image

4.2 Analysis of management processes of VTS operators

To elucidate the cognitive processes of VTS operators, it is first necessary to understand the contents recognized by the operators.

In the previous study, we observed and analyzed the management processes of the subject as they performed the CRA detection task during VTS operation based on the data of the previous experiment.

4.2.1 Relationship between management actions and recognized contents of VTS operators

The management process is composed of management actions. As Fig. 6 shows, the management action log table is extracted in time series to get management actions. Based on the previous experiment’s data, we observed that there are three types of management action taken by a VTS operator: “watching”, “clicking”, and “calling” (Fig. 6). “Watching” is when the operator watches the ships and checks their behavior on the radar processing screen. “Clicking” is when the operator clicks the concerning ship and checks the ship’s information, including its destination, speed, course and distance to other ships on the radar processing screen. “Calling” is when the operator calls the concerned ship to provide information on the other ships. The time series connecting these management actions is the management process (Song et al. 2019).

Fig. 6
figure 6

Components of management action log table in time-series during CRA detection task

Full size image

In addition, it is possible to infer the recognized contents from management actions. The reason for this is after recognizing some information and determining whether the concerned ships cause CRAs, VTS operators determine and execute management actions from a cognitive viewpoint (Song et al. 2019).

Figure 7 summarizes the relationship among management actions, recognized contents and cognitive stages of VTS operators when they perform the CRA detection task. This section describes the management actions and recognized content. In the next section, the cognitive stages are explained.

Fig. 7
figure 7

Relationship diagram among management actions, recognized contents and cognitive stages of VTS operators during CRA detection task

Full size image

As Fig. 7 shows, the operators perform the CRA detection task by repeating the cycle of (1) “watching”, (2) “clicking”, and (3) “calling”. We can observe the flow of management actions in this figure as follows. First, to confirm the behaviors and information of ships, the operator watches or clicks on the ships displayed on the radar processing screen. Then, they call the concerned ships and provide confirmed information on other ships.

In this figure, we can also observe the flow of recognized contents as follows. First, all the concerned ships that cause CRAs are identified by checking their positional relationships. When the target ships are approaching each other, they are considered to cause CRAs because it is highly likely that a collision will occur. Secondly, the ships to be called and provided information are correctly selected. Thirdly, the call timing is determined appropriately.

4.3 Analysis of cognitive processes of VTS operators

Hollnagel (1993) noted that human behavior is largely determined by the situation in which it is executed. Klein (1998) defined the human cognitive process as conducting an appropriate and effective behavior from experience and making decisions in complicated and ambiguous situations. Klein also built a recognition primed decision model (RPDM) at the same time. The RPDM consists of four parts: (1) confirmation of situation change, (2) inquiry of knowledge, (3) prediction of situation progress, and (4) execution of behavior to solve the problem. The authors can therefore consider that cognitive ability in situations is important for humans to decide their behavior.

Based on the RPDM, we conclude that the cognitive processing procedure for humans to solve problems is as follows. The first stage is to recognize a situation change, the second is to predict its progress, and the third is to decide an action and execute it. However, knowledge is required in every stage.

Therefore, as Fig. 7 shows, in this study, it was assumed that the human cognitive process consists of three stages: (1) “situational awareness”, (2) “situation judgment”, and (3) “decision making”, which are colored in yellow, pink, and green, respectively.

Here, in this study, the cognitive processes of VTS operators as subjects in the previous experiment were created and analyzed.

First, based on the recognized contents obtained (Sect. 4.2.1), the cognitive actions of each stage taken by the operators during the CRA detection task were defined as follows. Situational awareness is the recognition of CRAs. Situation judgment is the prediction of the situation changes of CRAs. Decision-making is making a call to the concerned ships.

Second, based on the definitions in the previous paragraph, all cognitive actions of the three stages were extracted from the management actions. Then, cognitive process flowcharts (e.g., see Fig. 9) for the operators were created along with their time series. Comparing these flowcharts, the commonality and differences in the cognitive processes between inexperienced, mid-level, and experienced operators were obtained as follows.

In brief, we observed a commonality between the experienced and inexperienced operators: both predict the situation changes of CRAs before detecting CRAs. We found a difference in that it takes more time for an inexperienced operator to predict the situation changes of each CRA. In other words, the number of CRAs detected within a given time is smaller than that for mid-level and experienced operators.

Accordingly, we believe that a knowledge package that can predict the situation changes of CRAs in a shorter time is needed as an education tool for inexperienced operators.

5 Development of vessel traffic routine

5.1 What is vessel traffic routine?

In this paper, we propose VTR, which is a package of knowledge required for VTS operators to predict the situation changes of CRAs. VTR shows the situation changes of marine traffic to be predicted and describes information such as the timing of processing. We used the processing procedures of an experienced operator as material to develop VTR. Moreover, we used an explanation method based on a marine chart depicting the CRA situation in a time series, called the “situation chart”, for developing VTR.

5.2 Processing procedures of an experienced VTS operator

The processing procedures of a VTS operator are to understand a problem in the situation and then take action to solve the problem. Therefore, we consider the processing procedures to consist of “cognition processing” and “content processing”. The former includes the recognized timing and content, and the latter indicates the information provided to the concerned ships. The characteristics of the processing procedures of experienced operators quickly predict the situation changes of CRAs and accurately process the predicted CRAs with appropriate timing.

In this study, we extracted the processing procedures based on the cognitive processes of an experienced operator under the scenario of the previous experiment and developed VTR for the five CRAs of high importance. It is noted that these five CRAs (CRAs 1–5) were detected by both mid-level and experienced operators in the previous experiment (Song et al. 2018, 2019), so that these CRAs can be regarded as important events.

The processing procedures for the target CRAs were extracted as follows. Some risks were included in the target CRAs. When to recognize these risks and how to deal with them were analyzed. Here, let us consider an example to illustrate the processing procedure of an experienced operator for CRA 1.

Figure 8 shows the tracks of ships and situation changes of CRA 1. CRA 1 occurs near buoy No. 2 on the Uraga Suido North Traffic Route, and it has three ships involved. These ships are called the “main ship”, “surrounding ship, forward”, and “surrounding ship, rearward”. As a supplemental explanation, the main ship is the ship that is likely to cause risks, and the surrounding ships are likely to be affected by the main ship.

Fig. 8
figure 8

Tracks of ships and situation changes of CRA 1

Full size image

In CRA 1, there is a possibility that the main ship will deviate from the route and overtake the surrounding ship, forward. Therefore, the three risks that are likely to occur with the situation changes of CRA 1 are shown in Fig. 8. In this figure, Risk A is the main ship approaching the surrounding ship, forward, Risk B is running aground in the shallow water area, and Risk C is approaching the surrounding ship, rearward, when the main ship returns to the route.

Figure 9 summarizes the cognitive process flowchart in the processing of an experienced operator along with the corresponding time series. The left and middle columns show the recognized time (timing) and management actions in each cognitive stage (cognitive actions), respectively. The three stages of the cognitive process of the VTS operator are indicated by yellow, pink, and green colors in the middle column, which mean “situational awareness”, “situation judgment”, and “decision making”, respectively. See Sect. 4.3 for the explanation of these three stages. The right column shows the recognized contents and information provided to the concerned ships.

Fig. 9
figure 9

Cognitive process flowchart of experienced operator for CRA 1

Full size image

From Fig. 9, it can be observed that the processing method of the experienced operator for CRA 1 is as follows: First, the existence of CRA 1 is recognized from the large difference in speed between the main ship and the surrounding ship, forward, and it is judged that the main ship will overtake the surrounding ship, forward, outside the route. Then, the situation changes of CRA 1 are predicted such that Risks A, B, and C will occur. Finally, decisions are made to provide information to the concerned ships to prevent the risks.

5.3 Developing VTR

VTR describes the positional relationship, processing timing, and processing content for two or more ships that require situational awareness and situation judgment. When developing VTR, several situation charts were created to show the situation changes, which means that one CRA is composed of several charts. These situation charts were devised by confirming the situation corresponding to the recognized timing of the experienced operator.

Figure 10 shows an example situation chart for CRA 1. This chart shows the situation 5 min after VTS operator starts recognizing CRA1. In this figure, we represent Risk A and B are predicted in the cognitive stage of situation judgment through the concerned ships’ states. In addition, we represent how Risk A and B are prevented through the provision of information in the cognitive stage of decision making. Also. in this CRA, “the risk avoidance area” is shown by the light green half-moon-shaped area as shown in Fig. 10. If the vector state of the main ship intersects the risk avoidance area, the VTS operator should recognize the possibility of the occurrence of related risks.

Fig. 10
figure 10

An example situation chart for CRA 1: 5 min after recognition starts

Full size image

It is thus possible for inexperienced operators to predict the situation changes quickly by understanding and remembering VTR. In the next section, we report a traffic management simulation experiment with inexperienced operators as subjects to verify the effectiveness of VTR.

6 Traffic management simulation verification experiment

6.1 Outline and purpose of the experiment

We conducted a traffic management simulation verification experiment using the training simulator owned by Japan Coast Guard on October 30th and 31st, 2018. We prepared one scenario in advance.

The purposes of the verification experiment are as follows:

  1. (i)

    Collect the management actions and information described in Sect. 4.2.1, that is, (1) the ships that were watched by inexperienced operators on the radar processing screen, (2) the ships that were clicked, and (3) the information provided to the concerned ships during the CRA detection task.

  2. (ii)

    Evaluate the cognitive processes of inexperienced operators based on the data collected.

  3. (iii)

    Verify the effectiveness of VTR based on the results.

Note that the experimental scenario and facilities were identical to those of the previous experiment described in Sect. 4.

6.2 Experimental method

With the cooperation of the 3rd Regional Coast Guard Headquarters, we recruited four inexperienced operators who had just been assigned to the Tokyo Wan Vessel Traffic Service Center in April 2018. In addition, we explained the flow of the experiment to the operators recruited in advance and obtained their consent to participate in the experiment. These four inexperienced operators participated in the verification experiment, and they were divided into two groups: (1) with prior explanation of VTR and, (2) without prior explanation of VTR. To verify the effectiveness of VTR, the differences between the two groups were observed. We could make the subjects in one group instead, and let them attend the first experiment without explanation of VTR, and then let them attend the second experiment after the explanation of VTR. However, in this case, it is easy for the same subject to recognize the CRAs twice on the same scenario. Thus, in this experiment with one scenario, we divided four subjects into two groups for verifying the effectiveness of VTR. It is noted that the VTR including 5 CRAs presented in Sect. 5 was used in this experiment.

We explained VTR to group (1) as follows. First, we explained the processing procedure for each situation chart of VTR. Then, we told them to manage other CRAs during the situation changing time of the target CRA. The reason for this is it is important to pay attention to all of the CRAs that occur in the sea area evenly. One CRA in the VTR was explained for approximately 3 min, so the total time spent explaining the whole VTR was approximately 15 min. There might be a concern regarding the fact that the situation charts on the radar processing screen in the experiment is similar to the VTR charts. This is because the scenario of the previous experiment was used for the verification experiment and to develop VTR. However, not only the ships included in the CRAs but also all the moving ships in the sea area are displayed on the screen. Moreover, because the ships are displayed with thin lines (see Fig. 3), we consider it to be difficult to recognize CRAs only by memory unless the processing procedure for the CRAs is easily understood.

Immediately before the experiment, a collaborator, who is an experienced operator, explained the target sea area to the subjects. The subjects were requested to follow the VTS operation as usual and were not informed that CRAs were included in the scenario.

After that, we set the eye tracker and had each subject wear the eye tracker for approximately 3 min. In addition, for approximately 15 min after the experiment, there was an interview in which each subject was enquired about the intentions of his operation, such as providing information to the concerned ships.

Table 1 shows information about the four subjects. In principle, VTS operators who have been working for less than three years are regarded as inexperienced. However, in this study, the operator who had been working for more than three years with no training was also regarded as an inexperienced operator. In other words, the decision of a VTS operator’s ability level is based on their education and experience.

Table 1 Information on subjects
Full size table

7 Evaluation of verification experiment results

From the perspective of the human factor, the evaluation methods of task execution results can be divided into two types: (1) Evaluation of human mental workload and (2) Evaluation of human performance. In addition, there are subjective evaluation and objective evaluation respectively. Vanderhaegen et al. (2020) reported that unconscious errors were generated in humans when heart rate and dynamic events were synchronized using a subjective workload evaluation method. In air traffic control, Manning et al. (2000) focused on the performance of air traffic controller and evaluated them subjectively by dividing them into evaluation stages. Borghini et al. (2017) investigated the use of neurophysiological signals to objectively assess cognitive behavior. There was also a study using optical brain imaging to estimate and evaluate the cognitive workload of air traffic controller (Ayaz et al. 2011; Harrison et al. 2014). In VTS, Malagoli et al. (2017) measured psychophysiological signals and objectively evaluated the cognitive workload of voyage pilots. On the other hand, in recent years, there has been little research on the evaluation of human performance in VTS. Therefore, this study proposes an objective evaluation method for human performance. To be concrete, cognitive behavior is evaluated quantitatively and objectively mainly based on eye movement data.

The purpose of the evaluation is to determine whether VTR improved the speed and accuracy of CRA prediction for inexperienced operators.

For the evaluation of the verification experiment results, we proposed two methods: (1) a balance sheet of confirmation of CRAs on the screen (balance sheet) and (2) an evaluation table of the processing procedure (evaluation table). By creating the balance sheet based on the eye movement data, it was analyzed whether inexperienced operators can confirm the sea area such that all the CRAs are checked in a well-balanced manner on the radar processing screen. Note that we regard the checking process of a VTS operator as “well-balanced” if he/she checks all the display information on the screen evenly. Also. by creating the evaluation table, the processing procedures of each subject were quantified based on all of the verification experiment data.

7.1 Creating a balance sheet for confirmation of CRAs on the screen

We believe that checking all the CRAs in a well-balanced manner helps VTS operator to quickly improve their prediction ability for situation changes. Thus, as an evaluation method of VTR, by aggregating the number of times each CRA was gazed, we created a balance sheet. By checking whether the number of times each CRA gazed was evenly arranged, we analyzed whether the subject was watching multiple CRAs in a well-balanced manner.

7.2 Creating an evaluation table for the processing procedure

7.2.1 Concept

As the processing procedure for detecting CRAs, VTS operators aim to recognize the target CRA, predict the risks contained in the CRA, and provide information to the concerned ships quickly and accurately. Therefore, to evaluate the inexperienced operators, we considered the earliness of the recognized timing and accuracy of the content processing for each recognized content are two important points.

Furthermore, as the characteristic of the human cognitive process, whether the appropriate action can be done is affected by the gained experience (Klein 1998). Because experienced operators have much more experience than inexperienced ones, it is possible for experienced operators to take more rational action from a cognitive perspective. Note that “more rational action” means more efficient performance. Therefore, the processing procedure performed by experienced operators is a good example for inexperienced operators. For this reason, by comparing the processing procedure of each subject with that of an experienced operator in the previous experiment, each inexperienced operator was scored. The higher the score, the better the processing procedure of an inexperienced operator and the higher their level of ability to predict the situation changes of CRAs.

7.2.2 Structure of evaluation table

Figure 11 shows the configuration of the evaluation table for the processing procedure for each CRA. Regarding the evaluation index, we targeted each cognitive actions of the three stages extracted from the management actions of VTS operators (Fig. 7). To be concrete, the cognitive actions of the corresponding CRA are identified from the management action log table in time-series (Fig. 6), and the start time and contents are evaluated.

Fig. 11
figure 11

Structure of evaluation table for processing procedure of each CRA

Full size image

As Fig. 11a shows, the Recognized timing and Content processing for the recognized content are used as evaluation indices. The Start times of the three stages of the cognitive process are considered as the Recognized timing. To which ship the operator called (Ship of calling) and what kind of information was provided to the ship in the calling (Information provided) are important, so that they are evaluated as the Content processing.

The Entire CRA and all risks related to the CRA (Risk A, B, C in Fig. 11a) are considered as Recognized content for each CRA. Note that recognizing an Entire CRA needs only to be evaluated by the start time of “situational awareness”, which is the initial time of noticing the CRA. Table 2 shows an example of an evaluation table of the processing procedure of Subject 1 for CRA 1. We can observe from this table that the recognized content of CRA 1 is the Entire CRA 1, Risk A, B, and C, and evaluations were conducted for each recognized content.

Table 2 Processing procedure evaluation table for Subject 1 on CRA 1
Full size table

As Fig. 11b shows, the verification experiment results of each subject are compared with the standard value as an evaluation method. The scores are given by calculating the difference between the two data (see Table 2 for an example). In addition, in this study, the standard values are model data of the processing procedure that can be handled by inexperienced operators. Thus, the standard values were set based on the processing procedure of the experienced operator in the previous experiment and VTR described in Sect. 5.

As shown in Fig. 11c, the Total recognized timing, Total content processing, and Total, which is the sum of the former two, were calculated as Evaluation result of the target CRA. The Total recognized timing for the target CRA is calculated as the sum of the Recognized timing for all recognized content. The Total content processing for the target CRA is calculated as the sum of Processing content for all recognized content. Then the Total can be regarded as the evaluation result of the target subject for the target CRA. For example, we can see from Table 2 that the evaluation results of Subject 1 for CRA 1 are 105.5 and 81%. Note that the percentage is the ratio of the score to a perfect score. The high score and the high percentage indicate that the evaluation results of the subjects are good. Finally, the sum of Totals for each CRA is the evaluation result of the target subject considering all of the CRAs.

7.2.3 Assigning evaluation scores

It was observed in the previous experiment that once VTS operator has predicted the situation changes of a CRA, they call the concerned ships (Song et al. 2018). In other words, the content processing can be determined at the stage when some prediction has been made. Therefore, in the components of the CRA processing procedure, we consider that it is more important for VTS operator to recognize and predict CRAs quickly. Accordingly, in the evaluation indices, the recognized timing is assigned a higher score than the content processing. As mentioned in Sect. 5, depending on the distance between two ships, there is a possibility that a collision risk will occur. When such a situation change of a CRA is recognized, one of the ships is called and the information is provided. That is, the situation concerning two ships is predicted and then one ship is called. Therefore, we believe it is appropriate that the score of the content processing is half that of the recognition timing. The score range of each recognized timing was set from 0 to 10, so the score range of each content processing was set from 0 to 5.

The scoring system was added to increase the score as it approaches the standard value. Figure 12 shows the scoring function of recognized timing based on the time difference. When the verification experiment results are later than the standard value, that is, the time difference between the two is negative, the higher the score, the closer the time difference is to 0 s. If the verification experiment results are earlier than the standard value, that is, the time difference is positive, a score of 10 is given (see Table 2 for an example). Thus, by setting the maximum and minimum values of the time difference based on the verification experiment results, the two linear functions created in Fig. 12 are suitable for scoring the recognized timing.

Fig. 12
figure 12

Scoring function of recognized timing based on the time difference

Full size image

In terms of the method of scoring the content processing, the score is set higher as the verification experiment results match the standard value. Since only three types of matching results were confirmed in the verification experiment results, “match”, “half match” and “not match”, 5, 2.5, and 0 scores are given (see Table 2 for an example).

If no target CRAs are recognized or no processes are performed, no scores are given in the corresponding columns.

7.3 VTR effects

To confirm the effects of VTR, we compared the differences in the evaluation results between the two groups of subjects. In this study, the following two effects were confirmed using the balance sheet and evaluation table.

Note that the verification experiment results showed that no more than five CRAs used in VTR were detected by the subjects regardless of whether they were given a prior explanation of VTR. Thus, the data of the verification experiment results were analyzed for CRAs 1–5.

7.3.1 Balance of confirmation on the radar processing screen

Table 3 shows the balance sheet, which demonstrates the number of times each CRA was gazed on the radar processing screen by each subject. We can observe from this table that Subjects 1 and 2 gazed the radar processing screen more evenly and in a well-balanced manner compared to Subjects 3 and 4. The reason for this is Subjects 1 and 2 realized that multiple CRAs occurred simultaneously with VTR and began to pay attention to all the CRAs that occurred in the target sea area.

Table 3 Balance sheet for confirming screen: number of times each CRA was gazed by each subject
Full size table

7.3.2 Improvement of speed and accuracy for prediction

Table 4 shows the evaluation results of each subject for CRAs 1–5 and all CRAs together. As can be seen, CRAs 1 and 2 were detected by all four subjects, while CRAs 3 and 4 were detected by Subjects 1 and 2. No subjects detected CRA 5. It is noted that, according to the VTS supervisor who supported this experiment, these results are consistent with the difficulty of the CRAs.

Table 4 Processing procedure evaluation results of each subject for CRAs 1–5 and all CRAs together
Full size table

In this table, we can observe the differences in the evaluation results between Subjects 1 and 2 and Subjects 3 and 4. For easy CRA 1, Subject 1 had a perfect score for total content processing. Subject 2 had a perfect score for total recognized time, and Subjects 1 and 2 had higher scores than Subjects 3 and 4 for the average total score. Regarding the relatively difficult CRAs 3 and 4, Subjects 1 and 2 detected them much better than Subjects 3 and 4. CRA 3 was detected by only Subjects 1 and 2. CRA 4 was detected by Subjects 1, 2, and 3.

Regarding all CRAs, in terms of total recognized timing, total content processing, and total, Subjects 1 and 2 had higher scores than Subjects 3 and 4.

Therefore, as a result of the explanation of VTR in advance, the timing for noticing risks became earlier and the content predicting and processing became more accurate.

8 Discussions

In this study, we assumed the human cognitive process in three stages: “situational awareness”, “situation judgment” and “decision making”, analyzed and evaluated the cognitive processes of VTS operators based on the results of simulation experiments. Since the primary purpose is to establish the analysis and evaluation method, we decided to reduce the amount of analysis as much as possible by minimizing the number of subjects. Certainly, the more subjects there are, the more data can be compared and the more reliable the experimental results will be. However, due to the small number of VTS operators and the huge amount of time it takes to analyze the data, we decided that the experiment could only be conducted a small number of times. In addition, instead of a comprehensive comparison between subjects, as Fig. 6 shows, by creating a management action log table for each subject in time-series, the details of the process for each subject’s task from the start to the end were analyzed. Therefore, it is difficult to draw an accurate conclusion due to the small number of subjects, but the authors think that it is possible to get results that represent the actual tendency within a realistic range.

Since the scenario used to create VTR and the scenario for the verification experiment are the same, it seems that the evaluation result of VTR effects depends on the scenario. However, as Sect. 4.1.1 mentions, when creating the scenario, nine CRAs were incorporated after reproducing a 20-min maritime traffic flow for a specific sea area. In other words, since multiple CRAs are included in the scenario, results that do not depend on the scenario to some extent are expected.

In addition, the existing training methods for VTS operators in Japan are mainly oral transmission of CRA detection methods and CRA detection simulation using a training simulator. In other words, at present, the CRA detection method has not been used as educational material, it is, therefore, a novel proposal to use VTR as an educational material.

9 Conclusions

In this study, by analyzing CRA detection task in VTS operations and developing a new training method for VTS operators from the perspective of the cognitive process, we obtained the following findings.

  1. (i)

    Under the condition that the number of subjects and the number of scenarios was limited, we proposed an experimental analysis method using eye movements for VTS operations. To be concrete, we observed actual VTS operations based on ethnomethodology at the worksite and proposed an experimental method. We constructed an experimental system including measurement of eye movement and conducted a simulation experiment using the training simulator.

  2. (ii)

    In examining the content of the training method for inexperienced operators, we analyzed and understood how the experienced operator actually predicts situation changes and how to deal with the situation from situational awareness to decision making based on the experimental results.

  3. (iii)

    To improve inexperienced operators’ abilities to predict the situation changes of CRAs, we developed VTR, which is a knowledge package developed using the processing procedure of an experienced operator extracted based on cognitive process analysis.

  4. (iv)

    In addition to understanding the cognitive process of inexperienced operators, we proposed a balance sheet of eye movement and an evaluation table that objectively scores the inexperienced operators' processing procedure based on human behavior. In addition, although the importance and role of VTS operation have been recognized previously, the evaluation criteria for the operation were not clear. In this study, by evaluating the cognitive processes of VTS operators, we believe that the number of CRAs detected within a given period of time and the accuracy of processing the CRAs at an appropriate recognized timing might be suitable evaluation criteria for VTS operation.

  5. (v)

    To verify the effectiveness of VTR, we conducted a simulation verification experiment using two groups of inexperienced operators with and without prior explanation of VTR. Based on the experimental results, a balance sheet and evaluation table were created to confirm the effects of VTR, and the analysis results were compared. Consequently, the effects of VTR were confirmed in terms of cognitive action, recognized timing, and content processing.

  6. (vi)

    As the future work, by applying the proposed VTR extraction method to the analysis of other sea areas, it is also possible to extract a specific VTR for each sea area. Therefore, we expect that the VTR can be used as educational material in the training course for VTS operators.

  7. (vii)

    It is noted that we have observed individual differences in the CRA detection method even for VTS operators of the same level. Thus in the future study, we should consider conducting experiments with an increased number of subjects. Moreover, the selected evaluation index in this study is selected experimentally based on the observation of the operators. Thus in the future study, it is desirable that we also consider other indices for the experimental protocol by analyzing human error for VTS operators.