A user task design notation for improved software design

Need for the representation of system tasks in the conceptual model

The main assumption of this study is that, by explicitly designing system requirements in the conceptual models, their levels of understandability and the error detection processes could be improved. For instance, for some specific software like serious games, there is a need for representing the system tasks and user tasks separately and explicitly. These so-called serious games are designed not only for pure entertainment but also for serious purposes such as training, scientific research, or advertising (Michael & Chen, 2005). In contrast to the studies that show advantages of serious games, others report several challenges of them (Bellotti, Berta & De Gloria, 2010; Fernández-Manjón et al., 2015). As there are limited implementations of serious games in the educational environments, their socio-cultural, educational, and technological challenges need to be analyzed deeply (Fernández-Manjón et al., 2015). To face these challenges, the earlier studies also suggest new design approaches and tools that considered a variety of user types for more effective serious-game design (Bellotti, Berta & De Gloria, 2010). As earlier studies report, serious games should provide testing and progress tracking to evaluate how much a user has learned from playing and architectures that reflect elements of personalization and experience specific to the user (Michael & Chen, 2005; Bellotti, Berta & De Gloria, 2010). It is also reported that participants, rules and procedures, and success and failure are key components of any purposeful human activity (Abt, 1987). Additionally, besides the main components of a game like story and art, a serious game also involves pedagogy that supports elements of fun (Zyda, 2005). Hence, a serious game can be considered a computer-based challenge that aims to improve training through entertainment (Zyda, 2005). All these constraints address the identification of player tasks explicitly, which may have an impact on the assessment of the user performance, and on the feedback mechanism by the software, which in turn affects the user experience and user interfaces of the software product. Serious games are non-simple systems and hence their design requires special attention (Barral et al., 2020).

UML-AD as a conceptual design tool for serious games

In order to address these challenges of the serious games, several studies explored using Unified Modeling Language (UML). For instance, for generally modeling the expert reasoning in serious games, (Barral et al., 2020) have proposed an UML profile library. As UML is a very well-known and highly popular modeling language, several researches explored its extensions to adopt it into different contexts or purposes. Many UML-based representations have been studied which address specific requirements of various domains. AODML (Vyas, Vishwakarma & Jha, 2017) is proposed as a modeling language for aspect-oriented programming domain that provides capability of specification of aspects in the design and implementation phases. Secure-UML (Lodderstedt, Basin & Doser, 2002) incorporates the access control infrastructures in the design. For medical image processing, StarUML is proposed that captures annotations of medical images Ayadi, Bouslimi & Akaichi, 2013). IoTsec is another example that includes a notation for model security in IoT systems (Robles-Ramirez, Escamilla-Ambrosio & Tryfonas, 2017). Similarly, Agent UML enables description of agent behavior (Bauer, Müller & Odell, 2001). Additionally, UML is extended to include some other artifacts such as model checking (Shu, Wang & Wang, 2017), model merging (Farias et al., 2019). Currently, UML is the most popular conceptual modeling language (Mohagheghi, Dehlen & Neple, 2009; Torchiano et al., 2013) and offers a communication platform for software engineers to define, build, visualize a software system (Booch, Rumbaugh & Jacobson, 2005). It provides various diagrams for static and dynamic behaviors of a software system. As a standardized modeling language used for software design tasks, UML is a powerful tool that benefits not only software developers but also other stakeholders (Cooper, Nasr & Longstreet, 2014). In order to better design the games in general and game theory modeling (Noreen & Saxena, 2017) in particular, and in order to better validate and review system requirements inspection, many researchers reported benefits of the Unified Modelling Language Activity Diagrams (UML-AD) (Dobing & Parsons, 2006; Muñoz, Mazón & Trujillo, 2010; Bolloju & Sun, 2012). UML-AD describes the dynamic characteristics of a system. It represents system behavior through the flow between system activities. In addition, UML-AD is a popular and powerful design tool to represent task flow design in serious game environments (Ismailović et al., 2012; Buendía-García et al., 2013; De Lope & Medina-Medina, 2016; Callaghan et al., 2018).

Representation of system tasks in UML-AD

To identify and to separate the system tasks from the user tasks in UML-AD representations, swimlanes can be used. In the UML-AD, these tasks are all represented by the same action notation. As reported in earlier studies, some effort is still required to better implement this notation into the software development life cycle (France et al., 1998). Accordingly, in this study, an extension to the UML-AD representation (named as UML-ADE notation) has been proposed and its impact on the understandability of the software design is evaluated experimentally for a serious game case.

Research procedure

This study was organized as a between-subjects empirical design. As explained in the Fig. 2, one week prior to the study, the participants were provided the user requirements documents prepared for each scenario. In this way, they had the opportunity to examine the contents and understand the main system requirements.

For the experimental study, the participants were randomly divided into two groups. The first group received the UML-AD representations for scenario-1 (See Fig. 3) and scenario-2 (See Fig. 4) which included five defects seeded-in, and the explanation document for UML-AD notation.

Similarly, the second group was provided the UML-ADE representations for scenario-1 (See Fig. 5) and scenario-2 (See Fig. 6) with the same five defects embedded in them and the explanation document for UML-ADE notation. During the experimental study with randomly divided groups, participants were asked to examine the UML diagrams according to the requirements document and to find the defects. They were informed about the number of defects and were given 50 min. to find them. Once the defects were found, the subject recorded them on the Web-based Defect Report Form. In order to better measure and compare each participant’s performance at defect detection, PP_i (Formula 2) and the difficulty levels of the defects DF_i (Formula 1) values were also calculated [30]. For the purpose of answering the research questions proposed in this paper, the data collected through the experimental study was analyzed both descriptively and statistically.

The research questions proposed in this study are:

RQ1: Can participants detect more defects in UML-ADE than in UML-AD?

RQ2: Are the defects seeded in UML-ADE easier to detect than in UML-AD?

RQ3: Do participants perform better on defect detection performance with UML-ADE than with UML-AD?

RQ4: Are the participants’ performance sensitivities higher with UML-ADE than with UML-AD?

Instruments

The study is conducted in the participants’ native language, Turkish. Several instruments were prepared. These include the user requirements document of the scenarios, conceptual data model diagrams represented in the UML-AD and UML-ADE models for each scenario, Web-based Defects Report Form, questionnaires, notation explanation documents, and experimental study diagrams.

For this study, two scenarios were used which were developed for endoneurosurgery training programs (Cagiltay et al., 2019). Each scenario includes some user tasks intended to improve certain endoscopic surgical skills like depth perception, left–right hand coordination and eye-hand coordination in a 3D simulated environment. The users were asked to perform each task in a given time period. Besides the classical user interface, a gamification interface was applied to these scenarios. In this concern, the user score was shown in the interface with the parameters that were used for the calculation of the score for each task such as the duration of time spent on performing the task, the accuracy of the task (if it was performed successfully in a given time period or not), and the error rates indicating the number of wrong movements or unnecessary touches to the environment. Additionally, a sound feedback from the system was given for each successfully performed task. For each scenario, a user requirement document was prepared. The design diagrams of the scenarios were prepared with the notations UML-AD (Figs. 3 and 5, respectively) and UML-ADE (Figs. 4 and 6, respectively) representations and several defects as listed in Table 1 were seeded into these diagrams. In order to collect the necessary data to better understand the participants’ performance during the experimental study, a custom ‘Defect Report System’ software was developed. This web-based application recorded detailed information about the participant’s demographics, the scenario being studied, and the participants’ defect detection process. Each time the participant detects a defect, they were asked to record it in the Defect Report Form. In this way, measures such as the order in which the participant spotted a defect and the time it took to find defects were recorded (for each detected defect of two scenarios).

Finally, a questionnaire was also prepared and applied to better understand the participants’ feedback and opinions on the diagrams. As the participants were not expected to be familiar with the UML-AD notation, the explanations for both versions of these notations were also provided to the participants. Therefore, two notation explanation documents were also prepared. The material used for the experiment can be accessed from Supplemental Information.

Participants

Seventy-two volunteers participated in the experimental study. These participants were senior-year students at the Departments of Computer Engineering, Software Engineering and Information Systems Engineering. Information regarding gender and the number of participants is given in Table 2. The participants were briefed about the procedure and their consent was obtained before the experiment.

Since the scenarios were distributed randomly to the participants, there is an uneven number of participants in Scenario-1 and Scenario-2 for both males and females. Accordingly, for Scenario-1 there were 35 participants, and for Scenario-2 there were 37 participants.

Experimental study diagrams

For the experimental study, five defects for each scenario were seeded in both UML-AD and UML-ADE representations. The originals of these diagrams are in Turkish; for clarity, the English versions are provided below. The lists of the defects were seeded in each scenario as given in the Table 1.

For the experimental study, these defects were seeded in the UML-AD and UML-ADE versions of each scenario (See Figs. 3–6 respectively). As shown in the figures, Scenario-2 is more complex compared to Scenario-1.

Measures

To analyze the defect detection experiment results, two measures proposed by Cagiltay et al. (2013) were used: Defect Detection Difficulty Level (DF) and Defect Detection Performance of a participant (PP). These measures are summarized below.

(1) DEFECT DIFFICULTY LEVEL (DF) MEASURE

In this measure, the DF value is calculated based on the average duration spent by all participants to detect the specified defect (D_j), the score gained from the defect detection order (O_j), Success rate of detecting defect j (S_j- Number of people who detected defect j/Total number of participants) and the number of people who detected the specified defect (D_j) (Cagiltay et al., 2013). (1)${\displaystyle D{F}_j=\frac{D_j•O_j}{S_j}.}$ (2) PARTICIPANT PERFORMANCE (PP) MEASURE

In this measure, the defect detection performance of each participant i (PP_i) is calculated. To do so, the defect detection difficulty level value of each defect (DF_j) value is used where n is the total number of defects detected by participant i, and s is the total number of defects seeded in the UML-AD &UML-ADE (Cagiltay et al., 2013). (2)${\displaystyle P{P}_i=\frac{\sum _{j=1}^{n}D{F}_j}{\sum _{k=1}^{s}D{F}_k}.}$ This study is conducted as a part of Endoscopic Surgery Education (ECE) Project which is supported by The Scientific and Technological Research Council of Turkey (TUBITAK). The project number is 112K287 and Ethics Committee Report number is B.30.2.ATL.00.04.14/12-016. As the ethical committee approval is mandatory for TUBITAK, Atilim University Human Research Ethical Committee approved documents were submitted to TUBITAK. The ethical committee report covers informed consent form, and the samples of the data collection tools.

Results on number of detected defects (RQ1)

In order to answer RQ1, the number of defects detected by each participant was analyzed. Hence, an independent sample t-test was conducted to evaluate the hypothesis. It was found that those who worked on UML-ADE representation of Scenario-1 detected more defects than those working on UML-AD representation of the same scenario. The test was significant, t(70) = 2.63 and p = 0.011. The group working on UML-ADE detected more defects (M = 3.19, SD = 1.49) compared to the other group working with UML-AD ( M = 2.26, SD = 1.52). Figure 7A shows the distribution of both groups.

Similarly, an independent sample t-test was conducted to evaluate the hypothesis that the participants who worked on the UML-ADE representation of Scenario-2 detected more defects than those who worked on the UML-AD representation of the same scenario. The test was significant, t(70) = 2.35 and p = 0.022. Participants working on UML-ADE (M = 2.77, SD = 1.49) detected more defects than those working on UML-AD (M = 2.00, SD = 1.21). Figure 7B shows the distribution of both groups.

Results for recognized difficulty levels of the defects (RQ2)

For the second research question, the difficulty levels of each defect was calculated by DFj [30]. Table 3 illustrates the values of these calculated defect difficulty levels for Scenarios 1 and 2. It should be noted that all recognized difficulty levels for both scenarios are lower in the UML-ADE version than in the UML-AD except for the Scenario-2 in defect 4 (D4). Even these values are close (see Table 3, 1,374 and 1,202 respectively) compared to the differences in other defects on both scenarios; further analysis is required to determine the cause of this situation. However, as seen in Table 3, the average recognized difficulty level value for the defects seeded in the UML-AD of Scenario-1 (1,839) is higher than that of UML-ADE (1,017). Similar averages are achieved for the Scenario-2 on UML-AD (2,740) and on UML-ADE (1,501). These results indicate that it was more difficult to find the same seeded defects in the UML-AD version than in the UML-ADE version. In the same way, the defects seeded in UML-ADE (average 1,501) were identified more easily when compared to the ones seeded in UML-AD version (average 2,740). Because of the limited number of defects, a statistical analysis was not conducted for this data. The results of the questionnaire data also supported this result. The averages of participants’ responses for the three items of the questionnaire are summarized in Table 4, according to which the participants found the UML-ADE representations easier to understand (3.79) than UML-AD (3.07).

Similarly, their estimates on the level of understanding of the game that is represented with the UML-ADE notations is higher (3.79) than that of UML-AD (3.37). Their responses likewise confirmed the complexity of the representations in UML-AD to be higher (2.77) than that of UML-ADE (2.47) (Table 4).

Results about participants’ defect detection performance (RQ3)

Each participant’s defect detection performance (PP_j) was calculated [30] to analyze the performance of participants of two groups. An independent sample t-test was conducted to evaluate the hypothesis that the participants’ performance in detecting defects on UML-ADE version is better than that on the UML-AD version for Scenario-1. The test was significant, t(70) = 2.77, p = 0.007. Participants working on UML-ADE performed better (M = .62, SD = .30) than the participants working on UML-AD (M = .42, SD = .30). Figure 8A shows the distribution of both groups.

Similarly, an independent sample t-test was conducted to evaluate the hypothesis that the participants’ performance in defect detection on UML-ADE is higher than it is on UML-AD for Scenario-2. The test was significant, t(70) = 2.76, p = 0.007. It was found that the participants working on UML-ADE performed better (M = .49, SD = .32) than those working on UML-AD (M = .29, SD = .27). Figure 8B shows the distribution of both groups.

Results about Reviewers’ performance sensitivity (RQ4)

The True Positive (TP) and False Negative (FN) values for the detected defects were also calculated to better understand the performance of the participants. TP indicates the correctly-identified defects by each participant while the FN value indicates the incorrectly-rejected defects by each participant. Accordingly, the true positive rate, which is also called the ‘Sensitivity Measure’, which indicates how well a participant can detect a defect, is also calculated using the following formula (Peng, Wang & Wang, 2012). (3)${\displaystyle Sensitivity=\frac{TP}{TP+FN}}$

An independent sample t-test was conducted to evaluate the hypothesis that the sensitivity value for the participants working on the UML-ADE version of the system design is higher than for the participants who worked on the UML-AD version. The test was significant, t(42) = 2.26, p = 0.26. The sensitivity value for the participants who worked on the UML-ADE version (M = 0.69, SD = 0.30), on average, is higher than that of the participants who worked on the UML-AD version (M = 0.57, SD = 0.33). In other words, the probability of the defect detection rate of the participants working on the UML-ADE version is significantly higher than that of the ones working on UML-AD.