Making linear multimedia interactive: questions, solutions and types of reflection

As flipped, blended and online learning increases in popularity, so does the ubiquity of multimedia as a learning tool, especially linear videos, (Davis 2013, Hill et al 2015, Yeung et al 2016). Asking students questions is also pervasive, occurring face-to-face, in exams, and online (Tune et al 2013, Brualdi Timmins 1998, Hargie 1978). Combining the question, solution & reflection framework with online educational video takes a typically linear, non-interactive medium and makes it interactive. Students could be asked to provide their answers as the video pauses, scaffolding student learning, or to provide their answers at the end and compare with the solutions provided (Mayo et al 2009). While comparing, students reflect on their answers (Dewey 1933, Rodgers 2002). Do they notice that their answers are the same as the solution, or not? How do they reconcile their thinking, deciding whether to modify their answers or not? This process, reflective thinking comes in different types and is regarded as an effective mechanism to aid learning (Mezirow 1998, Hubbs and Brand 2005, Lebedev and Sharma 2019). However, the process of reflective thinking is under-researched in the context of online physics education videos.

This study attempts to probe reflective thinking with first year university physics students using question, solution & reflection style online educational videos. The first research question is quantitative, seeking to obtain a sense of the decisions students made.

RQ1: How many students modified, or did not modify their answers with respect to whether their answers to questions shown in a video matched/did not match the solution provided in another video?

The second research question is qualitative, probing how students reconcile their thinking justifying their decisions.

RQ2: What reasons do students give for modifying their answers or not modifying their answers?

2.1. Multimedia

2.2. Reflection

3.1. Context

3.2. Procedure and intervention

To complete the activity, students logged into Canvas where a link took them to a secure university platform (RedCAP) in accordance with university human ethics research protocols. Figure 1 shows the four components to this activity: (a) video one containing four questions, (b) part one of the survey, followed by a page break—students could not go back, (c) video two containing solutions, and (d) part two of the survey. While the activity was open for a week and students could take a long time to complete the activity, they were allowed only one attempt.

Zoom In Zoom Out Reset image size

Page one-questions video

The questions video is from the YouTube channel 'Veritasium' by Dr Derek Muller. It is an embedded YouTube link which can be paused and replayed. The video presents a sequence of four different situations, primarily on mechanics, positing a question for each, see table 1. While the situations and the questions vary in style and content, they are perplexing; inviting explanations in line with Dewey (1933) phases of reflection. Figure 2 illustrates how different parts of the video follow the first three of Dewey's phases of reflection in question one; an experience is shown, there is a pause for the viewers to process the experience, the question is posed, and viewers are requested to generate possible explanations.

Table 1. The four questions presented in video one, and what each question focuses on.

Question number and its focus	What was asked?
Question one	A cylinder is shown to roll down an inclined plane.
Focus on concepts such as:	It rolls, then stops, then rolls again.
rotation,inclined plane,	What is inside the mystery cylinder?
centre of gravity, buoyancy
Question two	A bicycle has a string attached to the back wheel.
Focus on concepts such as:	The bike is gently supported so it does not tip over.
rotation, net force, gear ratios	If you pull backwards on the string, which way will the bikemove?
Question three	When running around a lap of a track, you run at speed v1.
Focus on the calculation of:	At what speed do you need to run the second lap,
time, distance, and average speed	so that the av researchers immersed erage speed of the two laps is
	equal to 2 times?
Question four	A normal train is moving along train tracks.
Focus on knowledge of: wheels,	As the train is moving forwards, what part of the train
train tracks	is moving backwards relative to the ground?

Zoom In Zoom Out Reset image size

Page one-survey part one

The students were asked: 'Have you seen this video before?', with the options being yes or no. In part one of the survey, students were prompted to write down their answers to each question in a free-response text box e.g., 'what is your answer to question x (description of question)'. This was followed by a 'next page' button. If any text box was left vacant, the students could not progress. After students moved to page two, they could not go back to page one, so they could not retroactively change their answers.

Page two-solutions video

The solutions video is also from the YouTube channel Veritasium. In it, Derek Muller presents the solutions, completing Dewey's phases by testing the selected hypothesis, see figure 3.

Zoom In Zoom Out Reset image size

Page two-survey part two

Part two of the survey asks: 'after watching the solutions video, did your answer to question one (description of the question) change?' with two options: 'yes I did change my answer' and 'no I did not change my answer'. If the student selects 'I did change my answer' the next statement reads 'I did change my answer because _______'. If the student selects 'I did not change my answer' the next statement reads 'I did not change my answer because _______'. This was repeated for the four questions.

After removing incomplete responses and those who did not consent, 508 student responses remained. The data were analysed using a mixed-method approach (Wisdom and Creswell 2013). Analysis of the data from the first part of the survey involved identifying if student answers matched the solution using a 'matching protocol' as outlined in table 2. The 'matching protocol' focuses not on the correctness of student answers but on whether those answers matched the solution provided. The protocol was developed by the first author, who considered the exact solution in the video and the common ways of re-phrasing used by students in their answers. A cornerstone of reflection is the act of comparing and distinguishing one's answer from the given solution, which is why the matching protocol focused on the match between the solution and student answer, rather than 'correctness'. The protocol was discussed with and refined in consultations with the other authors.

The analysis from the second part of the survey required coding. One of the researchers immersed themselves in the data, iteratively developing the coding scheme which were discussed with the other researchers. The first group to be coded was where responses did not match and were not modified; referred to as did not match-unmodified. The students' responses were inductively coded, and they aligned with our previous study (Lebedev and Sharma 2019). The coding used in Lebedev and Sharma (2019) had not been pursued from the beginning because the sample was different; public engagement with informal education whose motivations are different to our sample of first year university physics students. The second group, did match-unmodified, was coded deductively, using the earlier coding. The third group, did not match-modified, was inductively coded, as no prior codes have been established for modified responses. The codes for the third group were used for the fourth group, did match-modified.

After all data were coded, a sample of 60 student responses were independently coded by a fourth researcher. The intercoder reliability of 85% affirms the validity and robustness of the codes.

Table 3 displays the data corresponding to the first research question. The number of students whose answers matched or did not match are shown, followed by the number who modified or did not modify their answers.

The numbers of students with matched answers are small for questions one and two, approximately doubling for question four and more than doubling for question three. A substantial percentage of students provided answers that did not match. If we focus on questions one, two and four for not matched answers, most students did modify their answers. However, for not matched answers, the numbers of the students who did not modify their answers despite seeing the solutions video are nontrivial, 24 to 34%. Why they did not modify their answers after watching the solutions video is the subject of research question two.

Turning our attention to the pattern in table 3, we discuss the nature of the questions and our matching protocol. We note that students decided to modify/not modify without being aware of the matching protocol or whether their answers were deemed matched/not matched by the researchers. The matching protocol, table 2 was equally discriminating for all questions in that only particular variants on the solution were accepted, and these were applied consistently. Questions one and two draw on integration of concepts and are experiential in nature. Question one is mysterious, inviting a level of informed guessing. Question two was cheeky asking students 'which way will the bike move?' with the solution, 'usually move backwards, but it depends'. For both these questions, students responded, coming up with possible explanations. Most picked up the clues reflecting that their answers did not match the solution, and modified their answers. On the other hand, a nontrivial percentage did not modify their answers even though their answers did not match. We will probe these reflections further through coding the responses. Question three teases one's intuition; it is mathematical in nature, with a clear solution. Those whose answers matched, did not modify; those whose answers did not match, mostly modified. Question four is mischievous, requiring technical knowledge of trains and tracks as well as knowledge of Newton's third law, with a somewhat obscure solution. This question had the largest percentage of students who had answers which were not matched and they did not modify. Despite the fact that the questions were posed as closed questions, all questions were perplexing and one could offer possible explanations as outlined in Dewey's phases.

The second research question explores the types of reflections. Table 4 contain student quotes illustrating the categories of reflection. Note that in our coding, we had started off as inductive and switched to deductive when data saturation had occurred. So the categories and the similarities are a posteriori. Thus, during the act of comparing and distinguishing their answer from the solution, a cornerstone of reflection, students can and do provide similar reasons for both modifying and not modifying answers. It is important to note that individual students were tracked through the four questions; no tendency for consistently providing the same type of reflection for several questions was found. Table 5 presents the number of students in each category for both modified and unmodified when answers were not matched as these are the prevalent categories, see table 3. Next, we discuss the categories drawing on tables 4 and 5.

Categories correct, wrong and repeating the solution, capture answers in which students notice the clues in the solutions and acknowledge that there is agreement or not. A form of similarity, mirrors of each other, is found between correct and wrong-'I did not change my answer because my answer was correct' is very similar (and mirrors) 'I did change my answer because my answer was incorrect'. In the case of repeating the solution, students have actually taken action and written the solutions from the video. While this can be seen as a low threshold action, in terms of reflective thinking it can be viewed as a meaning-making process that moves a learner from one experience of the disciplinary content into the next. Combining wrong and repeating the solution, 22 to 32% of students, reconciled their thinking, acknowledging that their answers needed modification.

Categories almost correct, almost correct with an explanation or an alternative answer appear in both the modified, and unmodified groups. For example, a typical response is 'I did not change my answer because my answer was almost the same' as compared to 'I did change my answer though my answer was almost the same', both of these responses are coded as almost. Another example would be 'I did not change my answer because my solution would have worked' is very similar to 'I did change my answer; but my answer would also have worked'; both of these responses would be coded as alternative. 'I did change my answers because I was not completely correct and there were two ping pong balls inside too', is categorised as almost correct with explanation, as is, 'I did not change my answer because although I said thick, sticky, heavy substance the ping pong balls added to the mixture change the centre of gravity'. These reflections are mirrors of each other, the students have distinguished their answer from the solution on the video, demonstrating an understanding of the relationships which provides a thread that makes possible the continuity of learning disciplinary content. Table 5 shows that around 15% of the justifications for both modifying and not modifying their answers fell into these categories for question one. For questions two and four, these justifications appeared more for unmodified answers and the justifications were miniscule for question three. The categories discussed thus far fall into the overarching type of reflection captured by Mezirow (1998) as outward focusing where learners are intent on discipline content and tools, reconciling their answers with what is being covered or taught.

The single most popular category of reflections for modifying when answers were not matched is I do not fully understand, 15 to 21% of the students. This is reassuring as these students are able to distinguish that their answer is incongruent with the solution. Sometimes they articulate that they do not know acknowledging gaps in own knowledge, other times that they need to learn more or different things, and yet other times that they have learnt new things. In terms of reflection, they are demonstrating a disciplined way of thinking which requires attitudes that value the personal and intellectual growth of oneself. The category discontented which emerged in both modified and not modified, was not large but should not be neglected. These are responses which express discontent with the question, solution, content or presenter. Given that the questions ranged from somewhat cheeky to inviting a level of informed guessing, the low numbers expressing discontent and falling in this category attests that they rose to the challenge. Again, as students reconcile their thinking, we see the reflections for modified and not modified as mirrors of each other. These categories fall into the overarching type of reflection captured by Mezirow (1998) as inward focusing where learners are intent on their own learning and actions; taking responsibility for their learning or becoming emotionally engaged.

Finally, there were those responses which could not be coded for various reasons and those who had watched the video before.

Note that question 3 again is distinctively different, due to its straightforward mathematical nature, making it the most closed question, that is a question with one direct answer without other possible explanations. Question one is most open, attracting responses such as 'almost correct' to 'alternative solution' for both modified and unmodified. Questions 2 and 4 fall in between. In other words, questions that require integration of concepts and are experiential in nature such as 1 and 2, as well as those which require specific knowledge, such as question 4, are suited for generation of possible explanations in terms of Dewey's phases. Student reflections suggest that these are open questions where multiple plausible answers are offered by students.

The ubiquity and rise in popularity of online multimedia (Davis 2013) makes it important to research its educational value. It is also worth exploring not just conceptual gains or student engagement (Hill et al 2015, Yeung et al 2016), but reflection, which is under-researched in the online educational video context. This study points to some interesting results with regards to the question, solution & reflection style of videos which takes a typically linear, non-interactive medium and makes it interactive.

For the first research question, we note that the vast majority of students' answers are designated as not matched with the solutions. It is also worth noting that when the students were asked if they changed or did not change their answer, most of the students chose the expected option. That is to say, for those students whose answers did not match, most reported that they modified their answers while for those whose answers did match very few modified their answers. Only very few students whose answers did not match the solutions did not change their answers, self-reporting that they were correct. It seems safe to say that most of the students in this study are able to compare, distinguish and notice the differences between their answers and the solutions provided.

In answering our second research question, as students reconcile their thinking, the reasons provided for not modifying and modifying their answers are similar, or 'mirrors' of each other. The categories themselves demonstrate different types of reflections. Correct, wrong and repeating the solution illustrate a low threshold entry for reflective thinking involving meaning-making of the disciplinary content. Almost correct to alternative answer appearing in both the modified and unmodified groups involves reflective thinking of relationships in disciplinary content such that the learner can offer possible alternate explanations. Do not fully understand for learning is reflective thinking in a disciplined way with an attitude that values the growth of oneself. Discontented can be viewed as emotional engagement. The different types of reflective thinking discussed here, regarded as an effective mechanism to aid learning, are not dissimilar to those reported for adult learners, in particular the overarching patterns of inward focused and outward focused (Mezirow 1998, Hubbs and Brand 2005, Lebedev and Sharma 2019).

Different questions in different online physics educational videos serve different purposes (Muller et al 2008, Brualdi Timmins 1998). The questions in our study range from cheeky to inviting a level of informed guessing. Question three, mathematical in nature was perceived by students as a closed question, not inducing different types of reflective thinking. While the other questions were varied, we glean from the categories that they were better aligned with Rodgers' interpretation of Dewey's phases (Rodgers 2002, Dewey 1933). The questions were tricky, somewhat 'cheeky' and mischievous; some were open, some were closed. In our study, such questions induced more reflective activity. Many students rose to the challenge of providing informed guesses or focusing on self learning (Hill et al 2015, Sharma and Bewes 2011). All in all, the lesson for physics educators is that they should be tempted to intentionally create contexts for teaching reflective thinking; to provide some cheeky and mischievous formative questions which align with Dewey's phases to engage students in reflective thinking. Despite this task being worth 0.125% of their total grade for the course, the fact that students were prepared to spend time on their responses for this task, is worth noting. In particular, the question, solution & reflection style is rare and it would be worthwhile running similar studies in different contexts and with different topics.

The authors would like to thank the students who have participated in this study, and The University of Sydney Human Ethics Research Committee for approving this research. The authors would also like to acknowledge Dr Derek Muller, members of SUPER group and Jules Rankin for independently coding. We also thank the reviewers for their thoughtful feedback on this paper. The authors have confirmed that any identifiable participants in this study have given their consent for publication.