Automatic Judgement of Online Video Watching: I Know Whether or Not You Watched

Abstract

Videos have long been viewed through the free choice of customers, but in some cases currently, watching them is absolutely required, for example, in institutions, companies, and education, even if the viewers prefer otherwise. In such cases, the video provider wants to determine whether the viewer has honestly been watching, but the current video viewing judging system has many loopholes; thus, it is hard to distinguish between honest viewers and false viewers. Time interval different answer popup quiz (TIDAPQ) was developed to judge honest watching. In this study, TIDAPQ randomly inserts specially developed popup quizzes in the video. Viewers must solve time interval pass (RESULT 1) and individually different correct answers (RESULT 2) while they watch. Then, using these two factors, TIDAPQ immediately performs a comprehensive judgement on whether the viewer honestly watched the video. To measure the performance of TIDAPQ, 100 experimental subjects were recruited to participate in the model verification experiment. The judgement performance on normal watching was 93.31%, and the judgement performance on unusual watching was 85.71%. We hope this study will be useful in many areas where watching judgements are needed.

1. Introduction

Modern society has continued technological advances in telecommunications and equipment, such as artificial intelligence (AI), ubiquitous computing, the Internet of Things (IoT), smart cities, machine learning, big data and satellites, street advertisements, smart devices, and the surrounding environment, so it can provide personalized videos. These videos may or may not be desired by the individual. Videos began a long time ago as movies or TV programs but now include all videos produced using computer technology as a generic term for moving pictures [1]. Videos are actively watched for one’s own interest and satisfaction, such as dramas, movies, news, information, privacy of others, and YouTube. However, even though you may not want to view online learning or advertising videos, occasionally you may have to watch them for your own goals or for your company’s purposes [2,3]; furthermore, there are even cases where watching is essential for safety, such as school violence, corporate violence, fire, industrial safety, and disasters. However, unwanted viewing places pressure on people and causes them to play videos without watching.

Especially in education areas such as e-learning, smart learning, flipped learning, and blended learning, videos are being used very actively [3,4]. Due to COVID-19 in 2020, most schools around the world have been ordered to close and classes are being conducted through online remote learning [5]. However, the form of online learning, which requires self-directed learning, makes it difficult to identify false viewers [6,7,8]. Currently, used online remote learning is marked as “Viewing Complete” in the system, but it is difficult to determine whether it was actually watched. In actual online training experiments, a lack of consistency in learning effectiveness is largely related to video learning [9,10]. However, teachers cannot know exactly whether video learning has faithfully been done, and there is little research related to video watching judgement.

Common online video watching judgement techniques, which have been used for a long time and are now commonly used, are examined below. First, if the video is played from beginning to end, the system recognizes it as watching completion. Second, if in the middle of the video, the viewer solves the quizzes related to the content he or she watched, the system leads to the next video, and if that plays to the end, the system recognizes it as watching completion. Third, if another window on the computer is over the video while the video is played, it stops playing. However, even if you just play it until the end, the system recognizes it as watching completion. Fourth, when the play stops in the middle of the video, if the viewer clicks, it moves to the next screen, and even if the viewer just repeats these tasks to the end, the system recognizes it as watching completion. These methods can easily incorporate simple obstacles, but even if the video stops, it then plays continuously as long as the obstacle is cleared, so it is eventually misjudged as watching completion. These instances are difficult to see as honest watching completion because even if students do other activities without watching videos, the system misjudges watching completion. Therefore, it is necessary to accurately determine whether the online video was watched normally.

This paper proposes the time interval different answer popup quiz (TIDAPQ) model to judge video watching. TIDAPQ is a model that presents two interval popup quizzes in a video of approximately 10 min. This model calculates the time difference of answer submissions (RESULT 1) and the individual/different correct answers (RESULT 2). Then, if both of them are TRUE, the TIDAPQ will judge completion as normal watching; otherwise, it will judge completion as abnormal watching. After developing the TIDAPQ, 100 students at engineering universities were recruited as participants for the experiments, and the model was verified. In this paper’s experiment, the video was content of a learning character, and two popup quizzes were used in the videos of approximately 10 min long. However, TIDAPQ was not developed to judge only learning video watching. Depending on the purpose of watching a video, the length of the video and the number of popup quizzes may vary; thus, this paper describes the most basic of the TIDAPQ’s various forms.

We hope that the results of this study will be used in various fields, including online learning, reward advertising, and announcements from institutions or companies that require judging the watching of videos.

The composition of this paper is as follows. Section 2 (Background) explains the existing video watching judgement methods and describes their weaknesses. Section 3 describes the TIDAPQ model; of the three watching judgement approaches, time difference pass (RESULT 1) is explained in Section 3.3, individual/different correct answers (RESULT 2) in Section 3.4, and comprehensive judgement, which makes a final judgement using these two, is explained in Section 3.5. Section 4 shows the verification experiments of the TIDAPQ model, and Section 5 presents the conclusions and suggestions.

2. Background

The first area to try video watching judgement was in education due to the expansion of the online education market with the development of telecommunications and technology. In particular, video watching judgement systems emerged in 2015. Video learning methods were not absent previously in education; however, while previous video learning was watched based on the learners’ needs in an auxiliary learning mode, current video learning consists of diversified forms of learning using video, such as e-learning and smart learning. This diversification has occurred because one can take online video classes such as edX, Coursera, and Udacity and obtain a certificate, and one can obtain a degree with only online classes such as the Academic Credit Bank System [1]. In particular, education has also appeared in which classroom classes can be conducted only when you do the prior video learning, such as flipped learning [11,12]. Currently, flipped learning using online video is a great means of educating students in schools where talented people are gathered, such as Harvard, MIT, Seoul National University (SNU), and Korea Advanced Institute of Science and Technology (KAIST) [13,14]. However, flipped learning is difficult to expand into groups that lack self-directed learning skills, such as elementary, middle, and high schools, because it is difficult to determine whether they have normally done prior learning via online video watching [9,10]. In addition, due to the influence of COVID-19, online video learning has already entered the realm of elementary, middle, and high school public education around the world [5]; however, educational materials are only being distributed online and it is difficult to know whether young students who lack self-directed learning have honestly watched the online learning videos [6,15]. The form of learning that was expected in the future has been moved forward to the present. Accordingly, ancillary technological developments will have to be achieved.

In addition to the education sector, there are records that have been studied in the reward advertisement field for the purpose of judging the viewing of advertising videos [16], but they have not been used. Since the use of smartphones has become more active, advertising apps have appeared where you can obtain points when you look at advertising videos for approximately 2~3 min. During these advertisements, it was hard to identify false viewers who engaged in different activities as soon as the video started playing. Because the advertiser had to pay points for customers who did not watch the advertisements, currently, the method of paying points for watching advertising videos has almost disappeared. Instead of video advertising, advertising apps now offer points for photography, writing, web pages, trying, touching, and signing up.

The following Section 2.1, Section 2.2 and Section 2.3 describe the viewing judgement method attempted in the fields of education and advertising that generated the need for video viewing judgement.

2.1. The Appearance of the Video Watching Judgement System

Education is the first area to feel the need for video viewing judgement, and Zaption, which provides learner analysis, appeared in 2015 [17,18], followed by Educannon and Workday [19,20], but the service was terminated without a long duration. They carefully analyzed learners to help online video learning judgements, but the function was too complicated and the teacher had to make the final decision on watching completion, so it was ineffective. The URLs for Zaption and Educannon remain, but the pages have been deleted; the URLs for Workday, which acquired Zaption, have been changed from a learner analysis system to a system that helps the business.

Zaption of Figure 1, a representative company in learning video viewing judgement, shows viewers by date, average viewing time, question completion, stars, average skip forward, and average skip backward with graphs. Zaption can also check each student’s submitted responses, last submission date, last viewed date, total viewing time, and total views and check the answers to the video quizzes completed by the student [18]. Very detailed analysis was possible, but it was difficult to judge watching completion by compiling this information.

2.2. Current Video Watching Judgement System

There are currently no systems to make a video watching judgement after this video watching learning system was discontinued between 2015 and 2020, but similar systems include Playposit and Office Mix [7,8].

Playposit chose “Inducing View through Interactive Video” as its signature feature, but just when the quizzes, related to the content of the video, are solved in the middle of a video, the video can simply play into the next screen [7]. This technique is difficult to call a special interaction and it is difficult to determine whether the video was watched.

Office Mix is a function supported in MS Office 2013 and above and can create lecture videos by placing the teacher’s voice and face on PowerPoint slides. The provided analysis menus show the quiz answer rate, slide view frequency, and slide view time for each slide, but not enough information to analyze each user; it only shows a rough analysis [8]. The learner watching analysis through Office Mix expired in 2018 [8]; existing MS Office buyers can continue to use it, but new buyers cannot use it for free. Microsoft is currently operating it for a fee by converting Office Mix’s extended capabilities to Microsoft Stream. Microsoft Stream said, “We have online intelligent video, so it induces learners to watch” [21]. However, this approach is only different in that the video includes lecture content and it is released to a designated group, but there is no special difference from YouTube. While watching the video, a user can ask questions through chatting and have opportunities to interact with the teacher, but the user cannot see the watching analysis data, such as quiz answer rate, slide view frequency, or slide watch time provided by Office Mix.

2.3. Khan Academy

Khan Academy, famous for free online lecture services, has been growing steadily since 2008. Khan Academy is not a service run for learner analysis, but it shows “How much did they invest in studying per day?”, “What video did they see?”, “When did they stop the video and what did they look at?”, “What exercise did they use?”, and “Where did they focus?” [22]. It also shows the exercises and videos that many students focused on [23]. It is not a detailed analysis, such as Zaption and Educannon, but it is enough to grasp the student’s learning status. However, Khan Academy also has difficulties for teachers to judge whether video watching has been completed by analyzing the data provided by Khan Academy. There are Edmodo [24], Moodle [25], Blackboard [26], Schoology [27], Brightspace [28], Litmos [29], and TalentLMS [30] as learning management systems (LMS), which give or manage points for watching videos and prior learning, but like Khan Academy, they also have difficulties judging video watching.

2.4. Advertising Video Viewing Surveillance System

In addition to education, a field that perceives the necessity of watching judgements is advertising. The content viewing monitoring system of mobile reward advertising was developed to determine whether customers watched a provided commercial video [31]. For video advertisements that pay money or point rewards for watching, it is important to determine whether sincere viewing is occurring. The content viewing monitoring system of Figure 2 can detect facial areas from images acquired by cameras in Android smartphone environments; it then can monitor the location of eyes and the opening of eyes, so it can determine whether clients are looking at the screen. The eye detection method of the system uses block contrast between the right central block and the surrounding block to detect the eyebrow and then, looks for the eye location using geometric properties between the eyebrows and eyes to determine whether the eyes are open or closed [31]. In the integrated image of the eyebrow area, the eyebrows are extracted using the characteristic that the area corresponding to the eyebrows is relatively dark compared to the surrounding blocks. At the same time, from the integral image of each eye seeking area, the eye candidate areas are extracted using the characteristics that the pupil blocks are relatively dark and symmetrical compared to the rest of the surrounding blocks. Then, the central pixel of the block with the maximum block contrast is taken as the pupil candidate point.

The content viewing monitoring system was developed to determine the viewing of reward advertisements, but viewing judgement, which uses eye color contrast, determines watching completion regardless of whether a face photo or a teddy bear is placed in front of the camera, so it is difficult to apply to actual reward advertisements. This technology needs to be supplemented slightly more.

However, this technology has the great advantage of being simple to use to judge viewing. The viewing learning judgement systems, described in Section 2.1, Section 2.2, and Section 2.3, were ambiguous in the criteria for viewing judgement and difficult to use to judge because they provided incidental data collected after viewing rather than judging while viewing. On the other hand, the content viewing monitoring system can perform the viewing judgement immediately after viewing using only videos, without incidental data that occur after viewing. It has great advantages of being simple to use and immediate judgement. It is expected that if this technology develops, it will be a means of effective viewing judgement.

The abovementioned methods of video watching judgement have a common weakness—false viewers may occur when playing videos and performing other activities. When you find a video that has been stopped while doing other activities, you can just answer the quizzes. Even if it is a difficult quiz, it is possible to ask your colleague for answers through a chat, allowing you to write the answers any time you want. The video, which has been displayed on the monitor for a long time, becomes a video that the student has focused on and points will pile up even if there is a face photo in front of the camera. Learner analysis has been well done, but teachers have difficulty determining whether they watched the video through extensive data analysis. To achieve smooth learning progression, it is necessary to accurately determine whether videos are being watched. This paper uses only video and immediately calculates the watching judgement based on the viewer’s events occurring while watching the video, clarifies the criteria for watching judgement, and presents a simple method of watching judgement.

3. Time Interval Different Answer Popup Quiz (TIDAPQ) Model

3.1. Definition of Watching Completion

Watching completion means staring at the screen without doing anything else within a set period. We do not call it watching completion if the viewer has been watching for a long time or if the viewer skips the screen and watches it quickly because of knowing it already. A viewer being on STOP for too long means that they have done something else without watching the video. Normal watching completion means that the viewers stared at the screen honestly from the beginning to the end of the video. In this paper, to explain accurately, an answer and the correct answer were used separately. An answer means any record submitted by the viewers, whether it is a wrong answer or a right answer, and the correct answer means the right answer. We also used the timepoint and the time separately. The timepoint means one moment in time and the time means general time.

3.2. TIDAPQ Model

Figure 3 shows systems with the time interval different answer popup quiz (TIDAPQ) model applied. TIDAPQ is a model that inserts popup quizzes in the video and mixes two methods—Time Interval Pass (RESULT 1) and Interval Different Correct Answer (RESULT 2)—to perform Watching Completion Comprehensive Judgement. The system with this paper’s TIDAPQ model was used for Section 4′s model verification, which obtained the character of learning because the participants of the experiment were students in the computer department of an engineering university. However, the TIDAPQ model is not only developed for judging learning video watching; the system may vary depending on the purpose of watching the video. In addition, this paper uses two popup quizzes in a 10 min long video to describe the most basic form of TIDAPQ, but the length of the video and the number of popup quizzes may vary depending on the purpose of watching the video. The system’s process, applied to the TIDAPQ model, is as follows.

Professors sign up for a membership and create classes. Students sign up for a membership and request admission to classes created by the professor. The professor only accepts requests from the students in the corresponding class. Professors can manage videos, learners, announcements, questions, and answers. When a professor uploads a video to the TIDAPQ system, the system takes two random popup quizzes from the Quiz Pool Database and inserts them into the video. Students can check their learning progress, ask questions, and be given their own unique numbers. Students should solve two popup quizzes while watching the video for approximately 10 min. The length of the video may vary depending on the purpose, but the study was conducted with a video of approximately 10 min. The number of popup quizzes may also increase if the video is longer, but this paper describes the most basic form among the various forms of TIDAPQ. Student events are used to judge watching.

Time Interval Pass (RESULT 1) calculates the allowed time zone using the viewer’s event time while watching the video, and if the viewer completes the work within the calculated time range, it is judged as TRUE or FALSE. The event times used for calculation are as follows: starting timepoint of watching, popping up timepoint of 1st quiz, disappeared timepoint of 1st quiz, popping up timepoint of 2nd quiz, disappeared timepoint of 2nd quiz, and ending timepoint of watching. Since the individual popping up timepoints are different, the individual allowed time zones are also different. Time Interval Pass (RESULT 1) will be described in more detail in Section 3.3.

The quizzes are taken randomly from the database and are inserted into the video for a certain period. Since the quiz inserted into one video is the same problem for a certain period, a group of viewers who accessed the quiz during the same period see the same problem. However, to prevent any exchange of information with others, the individual correct answers are different. The unique numbers given to viewers were used as primary keys in the database, and since the problems stored in the database were created using individual primary keys, the correct answer also depends on the individual primary key. The viewers who answered the popup quizzes correctly have a value of TRUE for individual/different correct answers (RESULT 2). Individual/different correct answers (RESULT 2) will be described in more detail in Section 3.4.

Comprehensive Judgement uses the result values of RESULT 1 and RESULT 2 to become TRUE when both are TRUE. When the result of Compressive Judgement is TRUE, the TIDAPQ system determines that the viewer has achieved Normal Watching Completion, and the viewer can automatically check the result immediately after watching the video. The Normal Watching Completion Comprehensive Judgement will be described in more detail in Section 3.5.

After being judged as normal watching completion, the video can be watched repeatedly without interference from the popup quizzes. The judgement results of the TIDAPQ system can be reflected in the grade or be earned as points and can be reflected in a performance assessment. Therefore, viewers who need the normal watching completion result will stare at the screen without any other activity while the TIDAPQ video is playing.

3.3. Time Interval Pass

3.3.1. Time Interval Pass Process

Figure 4 shows the process of time interval pass.

• The moment the video begins, the Watching Start Timepoint is recorded.
• While watching the video, the 1st quiz appears at any timepoint and the video does not stop. The moment the 1st quiz appears, the Popping Up Timepoint of the 1st quiz is recorded. Popup quizzes must be answered within 15 s. The content of the quiz is very easy, so students can answer it right away. The moment the answer to the 1st popup quiz is written, the Answer Timepoint of the 1st quiz is recorded. After 15 s of the Popping Up Timepoint of the 1st quiz, the 1st popup quiz disappears and the Disappeared Timepoint of the 1st quiz is recorded.
• Viewers should keep watching the video. Otherwise, they may miss the popup quiz.
• While watching the video, the 2nd quiz appears at any timepoint and the video does not stop. The moment the 2nd quiz appears, the Popping Up Timepoint of the 2nd quiz is recorded. Popup quizzes must be answered within 15 s. The content of the quiz is very easy, so students can answer it right away. The moment the answer to the 2nd popup quiz is written, the Answer Timepoint of the 2nd quiz is recorded. After 15 s of the Popping Up Timepoint of the 2nd quiz, the 2nd popup quiz disappears and the Disappeared Timepoint of the 2nd quiz is recorded.
• The moment the video ends, the Watching End Timepoint is recorded.
• TIDAPQ calculates the time interval pass (RESULT 1) using all timepoints.

3.3.2. Quiz Appearance Time Range

Figure 5 shows the time range in which the quizzes appear. The 1st popup quiz appears in front of the video and the 2nd popup quiz appears behind the video. The values used to calculate the appearance time are the start times of each popup quiz. Based on “Half” in Figure 5, the front is the start time possible range of the 1st popup quiz $\left(ST{R}_{1stQ}\right)$ and the back is the start time possible range of the 2nd popup quiz $\left(ST{R}_{2edQ}\right)$. The quiz pops up at any time within a set time range, and each individual has a different popup time. After normal watching completion is judged in the comprehensive judgement, the viewers can watch a part of the video repeatedly without interference from popup quizzes.

$$0< ST{R}_{1stQ} < Half-x$$

(1)

$$Half < ST{R}_{2edQ} < VideoEnd-x$$

(2)

Equation (1) is the calculation of the start time possible range of the 1st popup quiz ($ST{R}_{1stQ}$) and Equation (2) shows the calculation of the start time possible range of the 2nd popup quiz $(ST{R}_{2edQ})$. To prevent the 2nd quiz from appearing while the 1st quiz is shown, 60 s of free time x is placed between the two popup quizzes. Since the 1st quiz is shown for 15 s and time is required to prepare the next task, free time x was set at 60 s. Free time is adjustable.

3.3.3. Quiz Answer Time Range

Figure 6 and Equations (3) and (4) show the normal quiz answer submission time range. The answer time range of the 1st popup quiz $\left(AT{R}_{1stQ}\right)$ of Equation (3) cannot exceed Half, and the answer time range of the 2nd popup quiz $\left(AT{R}_{2ndQ}\right)$ of Equation (4) should be between Half and VideoEnd. Popup quizzes disappear after 15 s, and viewers should answer them immediately as soon as they appear. The time duration the quiz is displayed for is adjustable, but too long a time may interfere with watching and too short a time may be insufficient to write answers.

$$0< AT{R}_{1stQ} < Half$$

(3)

$$Half < AT{R}_{2ndQ}<VideoEnd$$

(4)

3.3.4. Judgement of Time Interval Pass (RESULT 1)

This section describes the time interval pass judgement (RESULT 1), one of the two methods used to calculate the watching judgement of TIDAPQ. Existing online lectures are recognized as watching completion, even if viewers watch longer than the length of video, but TIDAPQ requires video length and watching time to match, and the answer submission time difference of the two quizzes should be within the calculated range. After the entire watching judgement is over, it becomes possible to look at an incomprehensible part repeatedly. RESULT 1, the result of time interval pass, has to meet the following four conditions:

Condition 1.

Was the video length and play time the same? (An error time of 5 seconds is added).

Condition 2.

Was the 1st answer $\left(A{T}_{1stQ}\right)$ written between the Start Timepoint of the 1st Popup Quiz $\left(S{T}_{1stQ}\right)$ and the End Timepoint of the 1st Popup Quiz $\left(E{T}_{1stQ}\right)$?

Condition 3.

Was the 2nd answer $\left(A{T}_{2ndQ}\right)$ written between the Start Timepoint of the 2nd Popup Quiz $\left(S{T}_{2ndQ}\right)$ and the End Timepoint of the 2nd Popup Quiz $\left(E{T}_{2ndQ}\right)$?

Condition 4.

Was the time interval between two answers $\left(ATI\right)$ between the calculated maximum $\left(AT{I}_{max}\right)$ and minimum $\left(AT{I}_{min}\right)$ values?

The above four conditions are expressed in the following Equations. Please see Figure 7 and Condition 4 further described below.

$$\mathrm{Condition} 1: VideoLength \le PlayTime \le VideoLength+5 seconds$$

(5)

$$\mathrm{Condition} 2: S{T}_{1stQ} \le A{T}_{1stQ} \le E{T}_{1stQ}$$

(6)

$$\mathrm{Condition} 3: S{T}_{2ndQ} \le A{T}_{2ndQ} \le E{T}_{2ndQ}$$

(7)

$$\mathrm{Condition} 4: AT{I}_{min}\le ATI \le AT{I}_{max}$$

(8)

Figure 7 shows the time flow of TIDAPQ videos and the time interval pass process of viewer A. Viewer A cases submitted normal answers in the time interval pass judgement. The dotted line connects the time used to calculate TIDAPQ’s Time Interval Pass and viewer A’s time. Viewer A cannot know when the popup quiz will appear and the popup quiz disappears after 15 s, so if students turn on the video and do other activities at the same time, they may miss the popup quiz. Even if the quiz is exposed on the screen, the video does not stop automatically, so the viewers who want normal watching completion cannot perform other activities.

The $\mathrm{VideoLength} \le \mathrm{PlayTime} \le \mathrm{VideoLength}+5 \mathrm{s}$ of Condition 1 prevents playing longer than the length of video. If the play time is longer than the video length of Condition 1, even if Condition 2, Condition 3, and Condition 4 have been met, then it is likely that after solving the 2nd popup quiz, he or she did not keep in his or her own seat.

Condition 2 and Condition 3 are designed to recognize the quiz popping up at any timepoint and answer it immediately within 15 s, and if the answers have not been submitted within the time of Condition 2 and Condition 3, TIDAPQ shall deem them not to have been staring at the screen.

Condition 4 relates to how the video is played. Most of the existing video watching judgement systems prevent play bars from being manipulated. They also block the play speed from being adjusted. This approach is not a bad approach to judge a first watching. However, in the case of online video lectures, you may want to focus on parts you do not know and watch only one part over and over again, but if you are not allowed play bars and have to watch it again from the beginning, or if you are not allowed to control the play speed, this may cause you to avoid watching it again for review. The same goes for commercial and public announcement videos. After receiving the results of normal watching completion, even if you want to focus on a part of the advertisement video that describes the special features of the equipment or even if you want to focus on a part of an important announcement video, if the system limits the skip button, speed, and play bar, the viewers will give up repeat watching and intensive watching. Therefore, not only the judgement of the video’s watching completion but also free operation of the video should be allowed. Considering the inconvenience for these users, TIDAPQ allowed free play, and after watching completion was judged, the viewers were allowed to focus freely on the parts they wanted. In addition, TIDAPQ will be used for groups that need the result of normal watching completion, but if you are a viewer who does not need the result of normal watching completion, such as parents and simply people interested in information, you can freely manipulate even the first video without the stress of watching it from the beginning. However, the problem that can occur at this time is that viewers can freely move the play bar, find only the popup quizzes, and submit answers, or viewers can change the play speed to quick and achieve abnormal watching. Condition 4 has been added to solve this problem. With the free operation of play buttons such as skip, speed, and play bars, Condition 2 and Condition 3 can be resolved, but it is difficult to resolve Condition 4, as shown below.

Condition 4′s $AT{I}_{min}\le ATI \le AT{I}_{max}$ is that for the time interval between two answers $\left(ATI\right)$ to be within the calculated time range, it must satisfy the following—Equations (9)–(11). In Figure 7, the first 15 s popup quiz appears at $S{T}_{1stQ}$, and the second 15 s popup quiz appears at $S{T}_{2ndQ}$. $ATI$ is time interval between two answers, $AT{I}_{min}$ is minimum time interval between two answers, and $AT{I}_{max}$ is maximum time interval between two answers.

$$ATI=A{T}_{2ndQ}- A{T}_{1stQ1}$$

(9)

$$AT{I}_{min}=S{T}_{2ndQ}- E{T}_{1stQ}$$

(10)

$$AT{I}_{max}=E{T}_{2ndQ}- S{T}_{1stQ}$$

(11)

Equation (9) is the time interval between two answers ($ATI$), which is the time difference between the answer timepoint of the 1st popup quiz $\left({AT}_{1stQ}\right)$ and the answer timepoint of the 2nd popup quiz $\left(A{T}_{2ndQ}\right)$. Equation (10) is the minimum time interval between two answers ($AT{I}_{min}$), which is the time difference between the end timepoint of the 1st popup quiz $\left(E{T}_{1stQ}\right)$ and the start timepoint of the 2nd popup quiz $\left(S{T}_{2ndQ}\right)$. Equation (11) is the maximum time interval between two answers ($AT{I}_{max}$), which is the time difference between the start timepoint of the 1st popup quiz $\left(S{T}_{1stQ}\right)$ and the end timepoint of the 2nd popup quiz ($E{T}_{2ndQ}$). Therefore, to satisfy Condition 4 of Equation (8), the time interval between two answers $\left(ATI\right)$ must be between the maximum and minimum values. Using this approach, there can be false viewers who only solved both Condition 2 and Condition 3 with free play manipulation and who watched abnormally. In addition, viewers can concentrate and watch the video repeatedly, beginning with the second watching through free play manipulation.

By combining the four conditions above, RESULT 1, the result of TIDAPQ’s time interval pass judgement, can be expressed as follows:

$$RESULT\_1= Conditions\_1 \wedge Conditions\_2 \wedge Conditions\_3 \wedge Conditions\_4$$

(12)

RESULT_1: Result of Time Interval Pass

Thus, TIDAPQ’s time interval pass prevents other activities and makes it possible to stare at the screen while the video is playing. If any one of the four conditions is not satisfied, RESULT 1, the result of time interval pass, is FALSE, so viewers who need the normal watching completion result realize that “It is wise to keep watching while the video is playing”. In addition, the content of the popup quizzes should be easy to avoid excuses that “The problem is difficult and time is delayed”. The details of the problem are described in the sections below.

3.4. Individual/different Correct Answers

3.4.1. Different Correct Answers to the Same Quiz

The quizzes presented for the watching judgement in the existing system are related to the video, and all viewers have the same answer. Because there is only one answer, viewers can share the answer with their colleagues or friends and can present the answer without watching even difficult questions. In addition, problems related to the video may delay time in the process of solving the problem, and even though viewers have watched normally, if they write wrong answers that are judged as abnormal watching, such viewers may be disadvantaged. To prevent this situation, individual/different correct answers were developed.

The content of TIDAPQ’s popup quiz is presented with very simple questions that are not related to the video content. Questions related to the video content might occur due to the difficulty in creating several correct answers for the same question. The same problem ensures equity within the watching group, and the individual/different correct answers prevent the sharing of correct answers. In this study, considering that the time available to submit answers for time interval pass (RESULT 1) was 15 s, we created easy questions that can be answered by all viewers in less than three seconds. Additionally, even if the problem is easy, there may be viewers who submit only the correct answers shared without watching the video, so we have created more than 10 correct answers.

Table 1. Popup quiz content.

No.	Content
Quiz 1.	Add the front second digit and the back second digit of the unique number. (_ ☐ _ _ _ _ ☐ _)
Quiz 2.	Write the flower name of the back third digit of the unique number. (_ _ _ _ _ ☐ _ _) 1. Rose 2. Lilies 3. Hydrangea 4. Dandelion 5. MorningGlory 6. Azalea 7. Forsythia 8. Tulip 9. Daffodil 0. Gypsophila
Quiz 3.	Subtract the back second digit from the front double digits of the unique number. (☐ ☐ _ _ _ _ ☐ _)

shows the content of a popup quiz, and

Table 2. Individual, different correct answers to popup quiz.

	Correct Answer
Unique Number	Quiz 1	Quiz 2	Quiz 3
26053072	13	Gypsophila	19
37621116	8	Rose	36
72541207	2	Lilies	72

shows the individual/different correct answers for three viewers with the unique numbers 26053072, 37621116, and 72541207. Correct answers for Quiz 1 can be made up to 19, correct answers for Quiz 2 can be made up to 10, and correct answers for Quiz 3 can be made for more than 100. Viewers were given their own unique numbers to write the different correct answers to the same question. In this paper, the unique number was based on their birth date and class number. The unique number is used with the primary key of the database. Since the problem was created using the primary key, the correct answer should also be solved using the primary key.

3.4.2. Judgement of Individual/Different Correct Answer (RESULT 2)

This section describes individual/different correct answer judgement (RESULT 2) among the two methods that are used for the watching judgement of TIDAPQ. Equation (13) shows how to judge the individual/different correct answer. The video contains two popup quizzes. RESULT 2, the judgement result of the individual/different correct answer, becomes TRUE when both the correct answer to the 1st popup quiz $\left(C{A}_{1stQ}\right)$ and the correct answer to the 2nd popup quiz $\left(C{A}_{2ndQ}\right)$ are TRUE.

$$RESULT 2= C{A}_{1stQ} \wedge C{A}_{2ndQ}$$

(13)

$$\begin{array}{c}RESULT 2—\mathrm{Result} \mathrm{of} \mathrm{Individual}/\mathrm{different} \mathrm{Correct} \mathrm{Answer}\hfill \\ C{A}_{1stQ}—\mathrm{Correct} \mathrm{Answer} \mathrm{to} 1\mathrm{st} \mathrm{Popup} \mathrm{Quiz}\hfill \\ C{A}_{2ndQ}—\mathrm{Correct} \mathrm{Answer} \mathrm{to} 2\mathrm{nd} \mathrm{Popup} \mathrm{Quiz}\hfill \end{array}$$

3.5. Comprehensive Judgement of TIDAPQ

Equation (14) shows the comprehensive judgement to use two methods, which are calculated for the watching judgement of TIDAPQ. Comprehensive judgement becomes TRUE when both RESULT 1 of time interval pass judgement and RESULT 2 of individual/different correct answer judgement are TRUE. In comprehensive judgement, if TRUE results, it is recognized as normal watching, and if FALSE results, it is recognized as abnormal watching. As soon as the video is finished, viewers can automatically check the results.

$$COMPREHENSIVE JUDGMENT= RESULT\_1 \wedge RESULT\_2$$

(14)

$$\begin{array}{c}COMPREHENSIVE JUDGMENT—\mathrm{Comprehensive} \mathrm{Judgement} \mathrm{of} \mathrm{TIDAPQ}\hfill \\ RESULT\_1—\mathrm{Judgement} \mathrm{of} \mathrm{Time} \mathrm{Interval} \mathrm{Pass}\hfill \\ RESULT\_2—\mathrm{Judgement} \mathrm{of} \mathrm{Individual}/\mathrm{different} \mathrm{Correct} \mathrm{Answer}\hfill \end{array}$$

4. TIDAPQ Model Verification

4.1. Participants

To measure the accuracy of the TIDAPQ system, 100 engineering university students participated in a TIDAPQ model verification. We recruited students to participate in model verification experiments among freshmen who had just entered engineering universities. Two groups were recruited for a certain period of time: the participants to watch normally and the participants to watch abnormally. Of the applicants, students who lacked prior knowledge about the video were selected for the study and participated in the experiment. Among normal watching applicants, 67 lacked prior knowledge, and among abnormal watching applicants, 33 lacked prior knowledge. The study participants gathered in the same classroom and 67 people watched the video honestly and 33 people watched the video while doing other activities according to the director’s instructions. The group watching honestly all produced TIDAPQ results of TRUE, but the group watching the video while doing other activities all produced TIDAPQ results of FALSE.

Table 3. Participants of TIDAPQ Model Verification.

Experimental Participants (100)		TIDAPQ Results
Participants who watched the video honestly	67	ALL TRUE (67)
Participants who watched videos while doing other activities	33	ALL FALSE (33)

shows 100 model validation participants and their TIDAPQ results.

4.2. Research Procedure

4.2.1. Experimental Design

Figure 8 shows the model verification experimental procedure of the TIDAPQ system. From among first-year freshmen who had just entered the engineering university, we recruited students to participate in the model verification experiment. Since participants should solve 10 problems related to the content after watching the video, the questionnaire used in the selection consisted of questions asking whether they knew the terms and answers used in the 10 questions. In the questionnaire checking for prior knowledge, students who indicated no prior knowledge participated in the experiment. The participants gathered in the same classroom; one group continuously watched TIDAPQ videos, and another group watched TIDAPQ videos while performing other activities by the request of the director. After watching the video, they had to solve 10 questions immediately. The 10 questions earned one point for each question and are core problems related to the video.

4.2.2. Video Learning Design

Table 4. Video content design used for model validation.

Video Content for Model Verification
Artificial Intelligence (AI)
· Artificial narrow Intelligence (ANI)
· Artificial general Intelligence (AGI)
Machine Learning Techniques
· Unsupervised Learning
· Supervised Learning
Deep Learning
· Neural Network
TensorFlow, Brain.js

shows the video content used to verify the model. The video is approximately 10 min long and consists mainly of conceptual explanations. After participants watched a 10 min video with the following content, they immediately solved 10 problems related to the video.

4.3. Research Instrument

Table 5. Research Instrument for Selecting Research Participants.

Questionnaire Content for Research Participants Selection

· Do you know the difference between Artificial Intelligence (AI) and Machine Learning Techniques and Deep Learning? · Do you know the concept and type of Artificial Narrow Intelligence (ANI)? · Do you know the concept of Artificial General Intelligence (AGI)? · Do you know the concept and type of Machine Learning Techniques? · Do you know Group and Interpret Data Based on Input Data? · Do you know Develop Predictive Model Based on Both Input and Output Data? · Do you know Unsupervised Learning? · Do you know Supervised Learning? · Do you know the relationship between Deep Learning and Neural Network? · Do you know TensorFlow and Brain.js?

shows the content of the questionnaire for selecting the research participants. Among first-year freshmen who had just entered engineering universities, students without prior knowledge should participate in the model verification experiment, so we conducted a survey to check the students’ prior knowledge. The survey is related to the video and asks whether they understand professional terms and concepts such as artificial intelligence (AI), machine learning, and deep learning.

Table 6. Research instrument for model verification.

Test Content for Model Verification

What is the difference between Artificial Intelligence (AI) and Machine Learning Techniques and Deep Learning? What is Artificial Narrow Intelligence (ANI)? What is Artificial General Intelligence (AGI)? What are the concepts and types of Machine Learning Techniques? What is Group and Interpret Data Based on Input Data? What is Develop Predictive Model Based on Both Input and Output Data? What is Unsupervised Learning? What is Supervised Learning? What is the relationship between Deep Learning and Neural Network? What is TensorFlow, Brain.js used for?

shows the content of the test paper that the research participants took immediately after watching the video. One hundred participants, who did not know technical terms and concepts such as artificial intelligence, machine learning, and deep learning, had watched videos on TIDAPQ for model validation. Then, the following test was immediately performed.

4.4. Analysis and Results

To check the performance of TIDAPQ, model verification was performed;

Table 7. Precision and recall ratio.

Experimental Participants (100)		TIDAPQ Results	Inter Ratio		Precision	Recall Ratio	F1
			Pass	Fail
Participants who watched the video honestly	67	ALL TRUE (67)	63	4	92.64	0.94	93.31
Participants who watched videos while doing other activities	33	ALL FALSE (33)	5	28	87.5	0.84	85.71

F1 Score: Harmonized Average Value of Precision and Recall.

shows the precision and recall ratio of TIDAPQ. The participants who answered “I have no knowledge of the video” gathered in the same classroom; 67 of them had watched TIDAPQ videos honestly, and 33 of them had watched TIDAPQ videos while performing other activities, according to the director’s instructions. The sixty-seven people who watched the video honestly achieved Normal Watching Completion (TRUE) on the TIDAPQ system, and the thirty-three people who did not watch TIDAPQ properly, while engaging in other activities, achieved Abnormal Watching (FALSE) on the TIDAPQ system.

After watching the video, they had to solve 10 questions immediately. The 10 questions earned one point for each question, and those questions addressed core problems related to the video. On the written test, the criterion score of the video watching was set at 8 (Inter Ratio). Sixty-three people of the 67 who TIDAPQ judged with a normal watching scored more than 8 points on the written test. Twenty-eight people of the 33 who TIDAPQ judged with abnormal watching scored less than 8 points on the written test. TIDAPQ watching completion judgement determined that the participants who actually watched video had 92.64% Precision and 94% Recall Ratio, and the participants who did not watch video had 87.5% Precision and 84% Recall Ratio.

Finally, TIDAPQ watching completion judgement showed that TRUE Judgement performance is 93.31% and FALSE Judgement performance is 85.71%. Therefore, TIDAPQ has valid performance in determining whether the video has been watched honestly.

5. Conclusions

This study presents the time interval different answer popup quiz (TIDAPQ) model to determine whether viewers watched an online video honestly. One-hundred students at engineering universities were recruited as research subjects and participated in the model verification.

Modern society has drawn attention to the technological development of telecommunications and equipment, and the fourth industrial revolution, street advertising, and the surrounding environment provide individual videos through smart devices. The form of videos has also diversified. Videos have long provided us with much information through TV and video players; however, viewer ratings have only been investigated through surveys that showed no interest in individual watching judgements. Since then, videos uploaded online, including news, movies, dramas, YouTube, and even personal privacy have become virtually infinite, but this situation also did not require individual watching judgement. However, since 2015, several companies and studies have attempted to judge video watching [18,20,31]. These attempts appeared for the purpose of judging honest watching when viewers had to make unwanted watching mandatory. Education was the first area to attempt video watching judgement. The reasons are as follows: because of the development of technology, the field of online education has also developed, so an education program has been developed that can bestow a degree just through online classes such as the Academic Credit Banking System [1]. There have also appeared education programs in which classroom classes are possible only when video learning has taken precedence, such as flipped learning [11,12]. Especially in groups in which self-directed learning abilities are excellent, such as Harvard and MIT, flipped learning using online video watching is going well [13,14], but in groups in which self-directed learning ability has not been verified, especially in younger grades, it is difficult to proceed smoothly due to false watching [6,15]. Online video watching judgements which are currently being used have difficulty in determining honest watching [7,8,21]. In addition, due to the influence of COVID-19 in 2020, online video learning has reached the realm of elementary, middle, and high school public education around the world [5]; however, educational materials are only distributed online and it is difficult to know whether students have watched online learning videos [6]. Teachers want to know accurately whether video learning has faithfully been done to design quality online classes, and in the current global situation, there is an urgent need to judge online video watching. In the area of reward advertising, which paid points for watching an advertisement, viewer judgement was necessary [31]. However, there was no particularly successful method used and it was difficult to identify false viewers, so the point payment method through advertising videos has almost disappeared; instead, point payment is currently made based on photos, web pages, trying to use, or signing up, etc. If it is possible to judge video watching, it is expected that the reward advertising using video clips will also be expanded. Similarly, judgement of video watching is a necessary technology, but it is difficult to find research related to the judgement of video watching.

The online video watching judgement method, which is currently widely used, needs to solve obstacles even if the video stops. Additionally, even if viewers have been doing other activities, if the video has just been played from start to finish, the system misjudges watching completion. Similarly, because these methods misjudge abnormal watching as normal watching, it is difficult to identify false viewers. Therefore, a technology that clearly determines whether the video has been viewed honestly is needed.

TIDAPQ, developed in this study, is a model that inserts two popup quizzes in videos, makes watching judgements with time interval pass (RESULT 1) and individual/different correct answers (RESULT 2), and then, makes comprehensive judgements on whether viewers were normally watching or abnormally watching using these two. First, TIDAPQ calculates the allowed time range using various timepoints in the video, including popping up timepoint of the quiz and disappeared timepoint of the quiz, and if viewers’ event time comes within the calculated time range, it judges time interval pass (RESULT 1) as TRUE. Second, the quizzes are randomly taken from the database and shown for a certain period on the video screen. It is difficult to share the correct answer with colleagues because the quiz has been created using the unique number given to each viewer, and the correct answer is different depending on the unique number. At this time, if you write the correct answer to all popup quizzes, individual/different correct answers (RESULT 2) is determined by TRUE. Finally, comprehensive judgement uses the results of RESULT 1 and RESULT 2 to determine whether normal watching is completed, and it automatically informs the viewers immediately after they watch the video. After being judged as watching complete, viewers can watch the parts they want intensively and repeatedly without interference from the popup quizzes.

To measure the accuracy of TIDAPQ, the research subjects were recruited among freshmen in domestic engineering universities, and 100 students without prior video knowledge were selected to participate in the verification of the TIDAPQ model. The video used in the experiment contains content of approximately 10 min of learning, but TIDAPQ is not a watching judgement model exclusively for the education sector; the length and content of the video can be different depending on the watching purpose. In addition, the number of popup quizzes used to validate the model is two, but this form is the most basic type of TIDAPQ; the number of popup quizzes can be different depending on the length of the video. As a verification result of TIDAPQ’s model, the performance of normal watching judgement was 93.31%, and the performance of abnormal watching judgement was 85.71%. These results show that TIDAPQ has high performance in video watching judgement.

The existing methods of video watching judgement have a common weakness; false viewers may occur when playing videos and performing other activities [6,7,8,15,16,17,18,19,20,21,22,23]. When you find a video that has been stopped while doing other activities, you can just answer the quizzes [17,18,19,20]. Even if there is a difficult quiz, it is possible to ask your colleague for answers through a chat, allowing you to write the answers any time you want. The video, which has been displayed on the monitor for a long time, becomes a video that the student has focused on [17,18,19,20,22,23], and points will pile up even if there is a face photo in front of the camera [24]. Learner analysis has been well done, but teachers have difficulty determining whether they watched the video through extensive data analysis [17,18,19,20,22,23]. On the other hand, the study justifies developing a TIDAPQ model that can judge the honest watching completion of online videos. This would differentiate it from previous research—to set the allowable time range using the viewer’s event time and to judge watching completion using individual/different correct answers even though it is the same problem. TIDAPQ uses only video and immediately calculates the watching judgement based on the viewer’s events occurring while watching the video, clarifies the criteria for watching judgement, and presents a simple method of watching judgement.

TIDAPQ is a model developed to monitor watching by groups that must watch unwanted videos, but researchers should be aware that applying TIDAPQ to videos that are too long can cause great stress to viewers due to the frequent popup quizzes. This paper has limitations in that the number of participants in the samples was small and that there have not been experiments on the actual video watching monitoring site. We plan to verify the effectiveness of TIDAPQ by applying it to sites where more participants and compulsory watching are required in the future. Through this study, we expect that TIDAPQ will be used properly in areas where watching completion judgements are needed, and we hope there will be more research related to video watching judgement.