Skip to main content
Log in

Relationships between Facial Expressions, Prior Knowledge, and Multiple Representations: a Case of Conceptual Change for Kinematics Instruction

  • Published:
Journal of Science Education and Technology Aims and scope Submit manuscript

Abstract

Kinematics is an important but challenging area in physics. In previously published works of the current research project, it was revealed that there is a significant relationship between facial microexpression states (FMES) changes and conceptual conflict-induced conceptual change. Consequently, the current study integrated FMES into a kinematics multiple representation instructional scenario to investigate if FMES could be used to help construct students’ conceptual paths, and help predict students’ learning outcome. Analysis revealed that types of students’ FMES (neutral, surprised, positive, and negative) were important in helping instructors predict students’ learning outcomes. Findings showed that exhibiting negative FMES through all three major representation segments of the instructional process (i.e., scientific demonstration, textual instruction, and animated instruction) suggests a higher probability of conceptual change among students with sufficient background knowledge on the topic. For students with insufficient prior knowledge, the result was the opposite. Moreover, animated representation was found to be critical to the prediction of student conceptual change. In sum, the results showed FMES as a viable indicator for conceptual change in kinematics, and also reaffirmed the importance of prior knowledge and representations of scientific concepts.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

Similar content being viewed by others

Availability of Data and Material

The data and relevant material may be accessed with authorization from the author(s) or National Taiwan Normal University.

References

  • Ainsworth, S. (1999). The functions of multiple representations. Computers in Education, 33(2), 131–152. https://doi.org/10.1016/S0360-1315(99)00029-9.

    Article  Google Scholar 

  • Ainsworth, S. (2008). How should we evaluate multimedia learning environments? In J.-F. Rouet, R. Lowe, & W. Schnotz (Eds.), Understanding multimedia documents (pp. 249–265). Boston: Springer US.

    Chapter  Google Scholar 

  • Ainsworth, S. (2014). The multiple representation principle in multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (2nd ed., pp. 464–486). Cambridge: Cambridge University Press.

    Chapter  Google Scholar 

  • Anderson, J. L., & Wall, S. D. (2016). Kinecting physics: conceptualization of motion through visualization and embodiment. Journal of Science Education and Technology, 25(2), 161–173. https://doi.org/10.1007/s10956-015-9582-4.

    Article  Google Scholar 

  • Barrett, T. J., Stull, A. T., Hsu, T. M., & Hegarty, M. (2015). Constrained interactivity for relating multiple representations in science: when virtual is better than real. Computers in Education, 81, 69–81.

    Article  Google Scholar 

  • Bellocchi, A., & Ritchie, S. M. (2015). “I was proud of myself that I didn't give up and I did it”: experiences of pride and triumph in learning science. Science Education, 99(4), 638–668. https://doi.org/10.1002/sce.21159.

    Article  Google Scholar 

  • Bellocchi, A., Ritchie, S. M., Tobin, K., King, D., Sandhu, M., & Henderson, S. (2014). Emotional climate and high quality learning experiences in science teacher education. Journal of Research in Science Teaching, 51(10), 1301–1325. https://doi.org/10.1002/tea.21170.

    Article  Google Scholar 

  • Braasch, J. L. G., & Goldman, S. R. (2010). The role of prior knowledge in learning from analogies in science texts. Discourse Processes, 47(6), 447–479. https://doi.org/10.1080/01638530903420960.

    Article  Google Scholar 

  • Breiman, L., Friedman, J. H., Stone, C. J., & Olshen, R. A. (1984). Classification and regression trees. Pacific Grove: Wadsworth.

    Google Scholar 

  • Chiu, M.-H., Chou, C.-C., Wu, W.-L., & Liaw, H. (2014). The role of facial microexpression state (FMES) change in the process of conceptual conflict. British Journal of Educational Technology, 45(3), 471–486. https://doi.org/10.1111/bjet.12126.

    Article  Google Scholar 

  • Chiu, M.-H., Liaw, H. L., Yu, Y.-R., & Chou, C.-C. (2019). Facial micro-expression states as an indicator for conceptual change in students' understanding of air pressure and boiling points. British Journal of Educational Technology, 50(1), 469–480. https://doi.org/10.1111/bjet.12597.

    Article  Google Scholar 

  • Chiu, M.-H., Yu, Y.-R., Liaw, H. L., & Lin, C.-H. (2015). The use of facial micro-expression state and Tree-Forest Model for predicting conceptual-conflict based conceptual change. Helsinki: Paper presented at the European Science Education Research Association Conference.

    Google Scholar 

  • Cook, M., Wiebe, E. N., & Carter, G. (2008). The influence of prior knowledge on viewing and interpreting graphics with macroscopic and molecular representations. Science Education, 92(5), 848–867.

    Article  Google Scholar 

  • De Ambrosis, A., Malgieri, M., Mascheretti, P., & Onorato, P. (2015). Investigating the role of sliding friction in rolling motion: a teaching sequence based on experiments and simulations. European Journal of Physics, 36(3), 21. https://doi.org/10.1088/0143-0807/36/3/035020.

    Article  Google Scholar 

  • Ekman, P. (1970). Universal facial expressions of emotion. California Mental Health Research Digest, 8, 151–158.

    Google Scholar 

  • Fang, N., & Uziak, J. (2018). Student misconceptions of general plane motion in rigid-body kinematics. Journal of Professional Issues in Engineering Education and Practice, 144(3), 03118001. https://doi.org/10.1061/(ASCE)EI.1943-5541.0000366.

    Article  Google Scholar 

  • Flores, R., Koontz, E., Inan, F. A., & Alagic, M. (2015). Multiple representation instruction first versus traditional algorithmic instruction first: Impact in middle school mathematics classrooms. Educational Studies in Mathematics, 89(2), 267–281.

    Article  Google Scholar 

  • Fredricks, J. A., & McColskey, W. (2012). The measurement of student engagement: a comparative analysis of various methods and student self-report instruments. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of research on student engagement (pp. 763–782). Boston: Springer US.

    Chapter  Google Scholar 

  • Fyfe, E. R., Rittle-Johnson, B., & DeCaro, M. S. (2012). The effects of feedback during exploratory mathematics problem solving: prior knowledge matters. Journal of Education & Psychology, 104(4), 1094–1108. https://doi.org/10.1037/a0028389.

    Article  Google Scholar 

  • Gasper, K. (2003). When necessity is the mother of invention: mood and problem solving. Journal of Experimental Social Psychology, 39(3), 248–262 Retrieved from http://www.sciencedirect.com/science/article/pii/S0022103103000234. Accessed 10 Sept 2020.

  • Gilbert, J. K., & Treagust, D. F. (2009). Introduction: macro, submicro and symbolic representations and the relationship between them: Key models in chemical education. In Gilbert, J. K. & Treagust, D. F. (Eds.), Multiple representations in chemical education (pp. 1-8): Springer.

  • Grayson, D. J., & McDermott, L. C. (1996). Use of the computer for research on student thinking in physics. American Journal of Physics, 64(5), 557–565.

    Article  Google Scholar 

  • Hailikari, T., Katajavuori, N., & Lindblom-Ylanne, S. (2008). The relevance of prior knowledge in learning and instructional design. American Journal of Pharmaceutical Education, 72(5), 113. https://doi.org/10.5688/aj7205113.

    Article  Google Scholar 

  • Hewitt, P. (1999). Average speed of balls. Physics Teacher, 37(7), 423–423.

    Article  Google Scholar 

  • Hwang, G.-J., Chu, H.-C., Shih, J.-L., Huang, S.-H., & Tsai, C.-C. (2010). A decision-tree-oriented guidance mechanism for conducting nature science observation activities in a context-aware ubiquitous learning environment. Journal of Educational Technology & Society, 13(2), 53–64 Retrieved from http://www.jstor.org/stable/jeductechsoci.13.2.53.

    Google Scholar 

  • Kalyuga, S. (2008). Relative effectiveness of animated and static diagrams: an effect of learner prior knowledge. Computers in Human Behavior, 24(3), 852–861. https://doi.org/10.1016/j.chb.2007.02.018.

    Article  Google Scholar 

  • Kenekayoro, P., Buckley, K., & Thelwall, M. (2014). Automatic classification of academic web page types. Scientometrics, 101(2), 1015–1026.

    Article  Google Scholar 

  • Kennedy, G., Coffrin, C., Barba, P. d., & Corrin, L. (2015). Predicting success: how learners’ prior knowledge, skills and activities predict MOOC performance. Poughkeepsie: Paper presented at the Proceedings of the Fifth International Conference on Learning Analytics and Knowledge.

    Book  Google Scholar 

  • Kim, N.-G., & Son, H. (2015). How facial expressions of emotion affect distance perception. Frontiers in Psychology, 6(1825). https://doi.org/10.3389/fpsyg.2015.01825.

  • King, D., Ritchie, S., Sandhu, M., & Henderson, S. (2015). Emotionally intense science activities. International Journal of Science Education, 37(12), 1886–1914. https://doi.org/10.1080/09500693.2015.1055850.

    Article  Google Scholar 

  • Kirby, N. F., & Dempster, E. R. (2014). Using decision tree analysis to understand foundation science student performance. Insight gained at one South African university. International Journal of Science Education, 36(17), 2825–2847.

    Article  Google Scholar 

  • Kozma, R. B. (2000). The use of multiple representations and the social construction of understanding in chemistry. In Jacobson, M. & Kozma, R. (Eds.), Innovations in science and mathematics education: advanced designs for technologies of learning (pp. 11-46): Routledge.

  • Kozma, R. B., & Russell, J. (1997). Multimedia and understanding: expert and novice responses to different representations of chemical phenomena. Journal of Research in Science Teaching, 34(9), 949–968 Retrieved from https://onlinelibrary.wiley.com/doi/abs/10.1002/%28SICI%291098-2736%28199711%2934%3A9%3C949%3A%3AAID-TEA7%3E3.0.CO%3B2-U.

    Article  Google Scholar 

  • Leonard, W. J., & Gerace, W. J. (1996). The power of simple reasoning. Physics Teacher, 34(5), 280–283.

    Article  Google Scholar 

  • Lewinski, P., den Uyl, T. M., & Butler, C. (2014). Automated facial coding: Validation of basic emotions and FACS AUs in FaceReader. Journal of Neuroscience, Psychology, and Economics, 7(4), 227–236. https://doi.org/10.1037/npe0000028.supp.(Supplemental).

    Article  Google Scholar 

  • Liaw, H. L., Chiu, M.-H., & Chou, C.-C. (2014). Using facial recognition technology in the exploration of student responses to conceptual conflict phenomenon. Chemistry Education Research and Practice, 15(4), 824–834. https://doi.org/10.1039/c4rp00103f.

    Article  Google Scholar 

  • Limón, M., & Carretero, M. (1997). Conceptual change and anomalous data: a case study in the domain of natural sciences. European Journal of Psychology of Education, 12(2), 213–230. https://doi.org/10.1007/bf03173085.

    Article  Google Scholar 

  • Linenberger, K. J., & Bretz, S. L. (2012). Generating cognitive dissonance in student interviews through multiple representations. Chemistry Education Research and Practice, 13(3), 172–178.

    Article  Google Scholar 

  • Loyens, S. M. M., Jones, S. H., Mikkers, J., & van Gog, T. (2015). Problem-based learning as a facilitator of conceptual change. Learning and Instruction, 38, 34–42. https://doi.org/10.1016/j.learninstruc.2015.03.002.

    Article  Google Scholar 

  • Madu, B. C., & Orji, E. (2015). Effects of cognitive conflict instructional strategy on students’ conceptual change in temperature and heat. SAGE Open, 5(3), 215824401559466. https://doi.org/10.1177/2158244015594662.

    Article  Google Scholar 

  • Maison, D., & Pawłowska, B. (2017). Using the Facereader method to detect emotional reaction to controversial advertising referring to sexuality and homosexuality. In Neuroeconomic and Behavioral Aspects of Decision Making (pp. 309-327): Springer.

  • Matsumoto, D., & Hwang, H. (2011). Evidence for training the ability to read microexpressions of emotion. Motivation and Emotion, 35(2), 181–191. https://doi.org/10.1007/s11031-011-9212-2.

    Article  Google Scholar 

  • Mayer, R. E. (2003). The promise of multimedia learning: using the same instructional design methods across different media. Learning and Instruction, 13(2), 125–139. https://doi.org/10.1016/S0959-4752(02)00016-6.

    Article  Google Scholar 

  • Mayer, R. E., & Moreno, R. (2005). A cognitive theory of multimedia learning: implications for design principles 91.

  • McNamara, D. S., & Kintsch, W. (1996). Learning from texts: effects of prior knowledge and text coherence. Discourse Processes, 22(3), 247–288.

    Article  Google Scholar 

  • Moore, J. C. (2009). Two tracks demonstrate average speed. Physics Education, 44(5), 456–458.

    Article  Google Scholar 

  • Ninaus, M., Greipl, S., Kiili, K., Lindstedt, A., Huber, S., Klein, E., Karnath, H. O., & Moeller, K. (2019). Increased emotional engagement in game-based learning—a machine learning approach on facial emotion detection data. Computers in Education, 142, 103641. https://doi.org/10.1016/j.compedu.2019.103641.

    Article  Google Scholar 

  • Ozuru, Y., Dempsey, K., & McNamara, D. S. (2009). Prior knowledge, reading skill, and text cohesion in the comprehension of science texts. Learning and Instruction, 19(3), 228–242. https://doi.org/10.1016/j.learninstruc.2008.04.003.

    Article  Google Scholar 

  • Pekrun, R. (2019). Inquiry on emotions in higher education: progress and open problems. Studies in Higher Education, 44(10), 1806–1811. https://doi.org/10.1080/03075079.2019.1665335.

    Article  Google Scholar 

  • Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: a program of qualitative and quantitative research. Educational Psychologist, 37(2), 91–105. https://doi.org/10.1207/s15326985ep3702_4.

    Article  Google Scholar 

  • Pekrun, R., Lichtenfeld, S., Marsh, H. W., Murayama, K., & Goetz, T. (2017). Achievement emotions and academic performance: longitudinal models of reciprocal effects. Child Development, 88(5), 1653–1670. https://doi.org/10.1111/cdev.12704.

    Article  Google Scholar 

  • Rau, M. A. (2018). Making connections among multiple visual representations: how do sense-making skills and perceptual fluency relate to learning of chemistry knowledge? Instructional Science, 46(2), 209–243. https://doi.org/10.1007/s11251-017-9431-3.

    Article  Google Scholar 

  • Rimoldini, L. G., & Singh, C. (2005). Student understanding of rotational and rolling motion concepts. Physical Review Special Topics - Physics Education Research, 1(1), 010102.

    Article  Google Scholar 

  • Skiendziel, T., Rösch, A. G., & Schultheiss, O. C. (2019). Assessing the convergent validity between the automated emotion recognition software Noldus FaceReader 7 and Facial Action Coding System Scoring. PLoS One, 14(10), 1–18. https://doi.org/10.1371/journal.pone.0223905.

    Article  Google Scholar 

  • Staus, N. L., & Falk, J. H. (2017). The role of emotion in informal science learning: testing an exploratory model. Mind, Brain, and Education, 11(2), 45–53. https://doi.org/10.1111/mbe.12139.

    Article  Google Scholar 

  • Trevors, G., Duffy, M., & Azevedo, R. (2014). Note-taking within MetaTutor: interactions between an intelligent tutoring system and prior knowledge on note-taking and learning. Educational Technology Research and Development, 62(5), 507–528. https://doi.org/10.1007/s11423-014-9343-8.

    Article  Google Scholar 

  • Tyng, C. M., Amin, H. U., Saad, M. N. M., & Malik, A. S. (2017). The influences of emotion on learning and memory. Frontiers in Psychology, 8(1454). https://doi.org/10.3389/fpsyg.2017.01454.

  • Ubben, I., Salisbury, S. L., & Daniel, K. L. (2019). Combining visual and verbal data to diagnose and assess modeling competence. In A. Upmeier zu Belzen, D. Krüger, & J. van Driel (Eds.), Towards a competence-based view on models and modeling in science education (pp. 99–115). Cham: Springer International Publishing.

    Chapter  Google Scholar 

  • van Heuvelen, A., & Zou, X. (2001). Multiple representations of work-energy processes. American Journal of Physics, 69(2), 184–196.

    Article  Google Scholar 

  • Wang, J., Berzins, K., Hicks, D., Malkers, J., Xiao, F., & Pinheiro, D. (2012). A boosted-tree method for name disambiguation. Scientometrics, 93(2), 391–411.

    Article  Google Scholar 

  • White, R., & Gunstone, R. (1992). Prediction-observation-explanation. In R. White & R. Gunstone (Eds.), Probing Understanding (pp. 44–64). London: Falmer Press.

    Google Scholar 

  • Wu, C.-H., Huang, Y.-M., & Hwang, J.-P. (2016). Review of affective computing in education/learning: trends and challenges. British Journal of Educational Technology, 47(6), 1304–1323. https://doi.org/10.1111/bjet.12324.

    Article  Google Scholar 

  • Xie, Z., Yu, X., Niu, J., & Li, Y. (2019). Facial microexpression recognition based on adaptive key frame representation. Journal of Electronic Imaging, 28(3), 033015.

    Article  Google Scholar 

  • Zhai, X. (2019). Applying machine learning in science assessment: opportunity and challenge. Journal of Science Education and Technology 1-4.

  • Zhai, X., Yin, Y., Pellegrino, J. W., Haudek, K. C., & Shi, L. (2020). Applying machine learning in science assessment: a systematic review. Studies in Science Education, 56(1), 111–151.

    Article  Google Scholar 

Download references

Acknowledgments

This project was supported by grants from the Ministry of Science and Technology in Taiwan (MOST104-2511-S-003-047-MY2 and 106-2511-S-003-009-MY2; Post-Doc 102-2811-S-003-006 and Post-Doc 102-2811-S-003-007). The authors would like to express their appreciation for to the Ministry funding this project and to the students who participated in this research.

Funding

This project was supported by grants from the Taiwanese Ministry of Science and Technology (MOST104-2511-S-003-047-MY2 and 106-2511-S-003-009-MY2; Post-Doc 102-2811-S-003-006 and Post-Doc 102-2811-S-003-007).

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mei-Hung Chiu.

Ethics declarations

The current study had been reviewed and approved by the Internal Review Board of the authors’ university.

Conflict of Interest

The authors declare that they have no conflict of interest.

Code Availability

The code may be accessed with authorization from the author(s) or National Taiwan Normal University.

Informed Consent

Informed consent was obtained from all participants.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

Sample Questions from the Screening Test and the Post-Test

(All questions translated from Traditional Chinese)

Screening Test:

What would you see when a speeding car runs off the cliff?

  1. 1

    It will fall perpendicularly.

  2. 2

    It will fall diagonally.

  3. 3

    It will fall in a curve.

  4. 4

    It will not fall.

figure a

Post-Test:

In the figure below, how would the horizontal velocity of the metal ball change just before it leaves the horizontal surface?

  1. 1

    Increase

  2. 2

    Decrease

  3. 3

    Remain Unchanged

figure b

In the figure below, how would the horizontal velocity of the metal ball change just before it slides down the slope?

  1. 1

    Increase

  2. 2

    Decrease

  3. 3

    Remain Unchanged

figure c

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Liaw, H., Yu, YR., Chou, CC. et al. Relationships between Facial Expressions, Prior Knowledge, and Multiple Representations: a Case of Conceptual Change for Kinematics Instruction. J Sci Educ Technol 30, 227–238 (2021). https://doi.org/10.1007/s10956-020-09863-3

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10956-020-09863-3

Keywords

Navigation