Lost in transition – Learning analytics on the transfer from knowledge acquisition to knowledge application in complex problem solving

doi:10.1016/j.chb.2020.106594

Computers in Human Behavior

Volume 115, February 2021, 106594

https://doi.org/10.1016/j.chb.2020.106594 Get rights and content

Highlights

•
Students should be able to transition from knowledge acquisition to its application.
•
Complex problem solving performance can be used to investigate knowledge transition.
•
A great proportion of students fails to engage in successful knowledge transition.
•
Rates of successful knowledge transition vary according to item complexity.
•
Indications that mental models play a key role for successful knowledge transition.

Abstract

Since Complex Problem Solving (CPS) skills represent a key competence for educational success, they are of great relevance for learning analytics. More specifically, CPS serves as a pertinent showcase for addressing a crucial existing gap contemporary education is facing, the gap between students' ability to acquire and subsequently apply knowledge in uncertain situations, which are increasingly important in the 21st century. While the CPS process incorporates both the acquisition and application of knowledge, many earlier studies have focused on identifying the factors relevant for success in knowledge acquisition. Given the dearth of existing research on factors influencing a successful transition between both CPS phases, we investigated the rates of successful and unsuccessful knowledge transition over the course of nine CPS items in a sample of N = 1151 students in 9th grade. Results showed that many participants were unable to transition their knowledge from the acquisition to the application phase, which was presumably due to an inefficient mental model transfer. Furthermore, the likelihood of students being ‘lost in transition’ was higher in more complex items. Implications are discussed in light of learning analytics, and particularly with regard to the factors to be taken into account by future CPS training programs.

Section snippets

Complex problem solving and knowledge transition

One 21st century skill that has received considerable attention in recent educational research is CPS (Greiff et al., 2013; Herde, Wüstenberg, & Greiff, 2016; Schweizer, Wüstenberg, & Greiff, 2013). Complex problem solving can be defined as the ability to solve problems with dynamic, hidden, or intertwined features. Such problems can be encountered on any societal level, from those with global impact such as, for instance, climate change (e.g., Urry, 2008), to individual ones referring to the

Sample characteristics

We analyzed the log files of a large-scale dataset of 9th grade students in Finland.¹

RQ1: How many students who solved the knowledge acquisition phase fail in the knowledge application phase?

Our first RQ addresses the question to what extent the problem of being ‘lost in transition’ is generally occurring. Therefore, we analyzed the relative frequencies of students who, after successfully solving the knowledge acquisition phase, did not manage to perform successfully in the knowledge application phase. The results of this analysis can be seen in Fig. 3 below, indicating that, across all nine MicroDYN items, a significant proportion of students fell into this group (previously

Discussion

The overarching aim of this study was twofold. To begin with, we wanted to uncover the magnitude of what we call the ‘lost in transition’ phenomenon, which refers to the case that a participant successfully explores the hidden variable relations in the knowledge acquisition phase, but fails in reaching the predefined target goals in the following knowledge application phase (i.e., Group B as shown in Fig. 2), in the particular showcase of CPS. In addition, we wanted to investigate potential

Conclusion

The primary aim of this study was to uncover the magnitude of the ‘lost in transition’ phenomenon, referring to the case that a participant who successfully acquires knowledge fails to subsequently apply this knowledge, in CPS. In addition, we discussed some potential reasons for why this phenomenon arises. Overall, we can conclude that a significant amount of students experiences being ‘lost in transition’, and that the probability of this phenomenon occurring particularly depends on the

Credit author statement

Björn Nicolay, Conceptualization, Methodology, Formal analysis, Writing - original draft, Writing - review & editing, Visualization. Florian Krieger, Conceptualization, Methodology, Formal analysis, Writing - review & editing, Resources, Visualization. Matthias Stadler, Conceptualization, Methodology, Validation, Writing - review & editing. Janice Gobert, Conceptualization, Resources, Writing - review & editing. Samuel Greiff, Conceptualization, Software, Resources, Writing - review & editing,

Declaration of competing interest

Samuel Greiff is one of two authors of the commercially available COMPRO-test that is based on the multiple complex systems approach, which employs the same assessment principle as MicroDYN. He receives royalty fees for COMPRO.

Acknowledgments

This research was supported by a grant from the Fonds National de la Recherche Luxembourg (Luxembourgish National Research Found) [CORE ‘TRIOPS’]. We would like to explicitly thank our Finnish colleagues for the possibility to analyze this large-scale dataset, which is based on Vainikainen (2014).

References (86)

J.R. Anderson et al.
Extending problem-solving procedures through reflection
Cognitive Psychology
(2014)
R.H. Baayen et al.
Mixed-effects modeling with crossed random effects for subjects and items
Journal of Memory and Language
(2008)
M. Bühner et al.
Working memory, visual–spatial-intelligence and their relationship to problem-solving
Intelligence
(2008)
A.A. Darabi et al.
Progression of mental models throughout the phases of a computer-based instructional simulation: Supportive information, practice, and performance
Computers in Human Behavior
(2009)
B. Eichmann et al.
The role of planning in complex problem solving
Computers & Education
(2019)
J.R. Frederiksen et al.
An approach to training based upon principled task decomposition
Acta Psychologica
(1989)
J. Funke
Microworlds based on linear equation systems: A new approach to complex problem solving and experimental results
D. Gopher et al.
Practice under changing priorities: An approach to the training of complex skills
Acta Psychologica
(1989)
S. Greiff et al.
Students' exploration strategies in computer-simulated complex problem environments: A latent class approach
Computers & Education
(2018)
S. Greiff et al.
Understanding students' performance in a computer-based assessment of complex problem solving: An analysis of behavioural data from computer-generated log files
Computers in Human Behavior
(2016)

S. Wüstenberg et al.

Complex problem solving — more than reasoning?

Intelligence

(2012)

Xi Zhang et al.

Learning analytics in collaborative learning supported by Slack: From the perspective of engagement

Computers in Human Behavior

(2019)

R. Adams et al.

Automatic coding procedures for collaborative problem solving

J.R. Anderson

Acquisition of cognitive skill

Psychological Review

(1982)

J.R. Anderson et al.

Learning rapid and precise skills

Psychological Review

(2019)

J.D. Barrett et al.

Thinking about applications: Effects on mental models and creative problem-solving

Creativity Research Journal

(2013)

D. Bates et al.

Fitting linear mixed-effects models using lme4

Journal of Statistical Software

(2015)

A. Buchner

Basic topics and approaches to the study of complex problem solving

J. Canas et al.

Cognitive flexibility and adaptability to environmental changes in dynamic complex problem-solving tasks

Ergonomics

(2003)

P. Charland et al.

Developing and assessing ERP competencies: Basic and complex knowledge

Journal of Computer Information Systems

(2016)

Z. Chen et al.

All other things being equal: Acquisition and transfer of the control of variables strategy

Child Development

(1999)

S. Everwijn et al.

Ability- or competence-based education: Bridging the gap between knowledge acquisition and ability to apply

Higher Education

(1993)

J. Funke

Dynamic systems as tools for analysing human judgement

Thinking & Reasoning

(2001)

J. Funke

Complex problem solving

J.D. Gobert et al.

Operationalizing and detecting disengagement within online science microworlds

Educational Psychologist

(2015)

J.D. Gobert et al.

From log files to assessment metrics: Measuring students' science inquiry skills using educational data mining

The Journal of the Learning Sciences

(2013)

S.P. Gott et al.

Problem solving in technical domains: How mental models and metacognition affect performance

Instruction: Theoretical and applied perspectives

(1991)

A.C. Graesser et al.

Advancing the science of collaborative problem solving

Psychological Science in the Public Interest

(2018)

Cited by (11)

Understanding transitions in complex problem-solving: Why we succeed and where we fail
2023, Thinking Skills and Creativity
Problem-solving is an essential transversal and learning skill in the 21st century. To better support it, we need a deeper understanding of its processes, i.e., problem exploration, problem representation and knowledge use. Understanding the mechanism behind the transitions between these processes plays an important role in supporting its development. With a sample of N = 1828 first-year university students, this study is the first to measure how students transition between the different problem-solving processes and how this influences their overall problem-solving performance as well as how this is related to their other test-taking behaviours. Results indicate that mastering the first transition – correctly understanding and depicting the problem structure – plays a crucial role in problem-solving. Students who failed to master it regularly failed in the next transition. Problem complexity strongly influenced transitions during the entire problem-solving process. Based on students’ behavioural patterns, we distinguished four qualitatively different latent transition classes: expert transitioners, advanced transitioners, beginner transitioners and non-transitioners. The number of interactions proved to be a more effective profile characteristic, especially on high-complexity problems, than time spent on the problem-solving process. The results of the current study provide important insights into how students’ transition between the different problem-solving processes and how their test-taking behaviour indicates their transition skills.
Focusing on eigendynamic effects promotes students’ performance in complex problem solving: A log-file analysis of strategic behavior
2022, Computers and Education
Citation Excerpt :
This pattern of findings suggests that more intelligent (compared to less intelligent) students achieved higher CPS performance scores in the second CPS phase by adequately identifying eigendynamics (first comparison), but regardless of whether they applied the strategic behavior to identify eigendynamic effects early or not (second comparison). Possibly, the strategic behavior to adequately identify eigendynamics and integrate the gained information in the first CPS phase helped more intelligent students to transfer their knowledge acquired in the first CPS phase to the second CPS phase: a process at which surprisingly many students fail (Nicolay et al., 2021). An explanation for the non-significant mediation effect of the second comparison could be that the strategic behavior to identify eigendynamic effects early might be too distant to the second CPS phase to mediate the intelligence-CPS relationship in this CPS phase.
Human beings are frequently confronted with complex systems that change dynamically over time. Correspondingly, so called eigendynamic effects are an important feature in complex problem solving (CPS) tasks. Eigendynamic effects describe increases or decreases in outcome variables over time without the individual's intervention. Identifying the eigendynamic effects from the start while exploring the system is reasonably an effective strategic behavior when systems are comprised of different effect types (i.e., eigendynamic effects alongside other effects). In this study, we hypothesized that the strategic behavior to identify eigendynamic effects early when exploring the system was related to CPS performance and mediated the relationship between intelligence and CPS performance. In a sample of N = 262 German high school students, the investigated strategic behavior predicted CPS performance when exploring the system (knowledge acquisition phase) and when controlling the system (knowledge application phase) in single CPS tasks and in a set of CPS tasks comprising of different effect types. In mediation models the strategic behavior to identify eigendynamic effects early mediated the intelligence-CPS performance relation in the knowledge acquisition phase, but not in the knowledge application phase. We discuss these findings in the light of how students interact with complex and dynamic systems, pointing to the importance of this strategic behavior for the relationship between intelligence and CPS.
Towards strengthening links between learning analytics and assessment: Challenges and potentials of a promising new bond
2022, Computers in Human Behavior
Learning analytics uses large amounts of data about learner interactions in digital learning environments to understand and enhance learning. Although measurement is a central dimension of learning analytics, there has thus far been little research that examines links between learning analytics and assessment. This special issue of Computers in Human Behavior highlights 11 studies that explore how links between learning analytics and assessment can be strengthened. The contributions of these studies can be broadly grouped into three categories: analytics for assessment (learning analytic approaches as forms of assessment); analytics of assessment (applications of learning analytics to answer questions about assessment practices); and validity of measurement (conceptualization of and practical approaches to assuring validity in measurement in learning analytics). The findings of these studies highlight pressing scientific and practical challenges and opportunities in the connections between learning analytics and assessment that will require interdisciplinary teams to address: task design, analysis of learning progressions, trustworthiness, and fairness – to unlock the full potential of the links between learning analytics and assessment.
Interdisciplinary frontiers: computer-based process data analysis in educational measurement
2022, International Encyclopedia of Education: Fourth Edition
In recent years, the collection and analysis of technology-based process data in the educational measurement realm has steadily increased. Process data are usually generated automatically while a student is working on a computer-based task or exercise, and contain all relevant pieces of information on how the student interacts with this task. Such information includes but is not limited to task duration, number of clicks, and task performance. Given the informational richness of computer-based process data, its analysis is of great interest to educational stakeholders and researchers alike. Notably, scientific results obtained from process data analysis have been used to adapt assessment techniques and institutional curricula, and have uncovered areas for intervention to foster integral skills that provide an educational benefit to students. This article presents a thorough overview of historical and contemporary applications of technology-based process data analysis in the educational field. Specifically, it provides in-depth coverage of the role of process data analysis with respect to psychometrical considerations, formative and summative assessment. Moreover, we describe process data analysis for large-scale educational assessments such as PISA and PIAAC, as well as publicly available process data repositories. Additionally, process data analysis in educational measurement is showcased using the example of complex problem solving, a skill that has proven to be relevant for students' educational success and beyond. Finally, this article addresses several current challenges to consider when engaging in computer-based process data analysis for educational purposes, before concluding with an outlook on potential future developments in the field.
How we explore, interpret, and solve complex problems: A cross-national study of problem-solving processes
2022, Heliyon
Citation Excerpt :
Sentences which were subject to different interpretations were further discussed among the researchers and translators. The CPS approach and the CPS tasks have been employed extensively at both the national and international levels (see Csapó and Funke, 2017; Eichmann et al., 2020; Greiff et al., 2013b; Greiff et al., 2015b; Mustafic et al., 2019; Nicolay et al., 2021; OECD, 2014a). The psychometric indices of the test proved to be good, independent of the cultures and nations (see Wüstenberg et al., 2014; Wu and Molnár, 2021).
Complex problem solving (CPS) is considered an important educational outcome in the 21^st century. Despite its importance, we have only little only knowledge of its measurability, development, or comparability in some countries, in particular in those with a short history of computer-based assessment. The results of the current study provide insights into the validity of CPS measurements and shed light on the different behavioral patterns and test-taking behavior in two convenience samples with different sample characteristics of Jordanian (N = 431) and Hungarian (N = 1844) students as they solve complex problems. CPS proved to be measurement-invariant in Jordan and Hungary among university students. Analyzing log data, we identified differences in students' test-taking behavior in terms of the effectiveness of their exploration strategy, time-on-task, and number of trials. Based on the students’ exploration strategy behavior, we identified four latent classes in both samples. The tested process indicators proved to be non-invariant over the different latent profiles; that is, there are differences in the role of the number of manipulations executed, time-on-task, and type of strategy used in actual problem-solving achievement between students that fall within different profiles. This study contributes to our understanding of how students from different educational contexts behave while solving complex problems.
The Influence of Emotion and Emotion Regulation on Complex Problem-Solving Performance
2023, Systems

View all citing articles on Scopus

^☆: This research was supported by a grant from the Fonds National de la Recherche Luxembourg (Luxembourgish National Research Found) [CORE ‘TRIOPS’].

View full text

Full length articleLost in transition – Learning analytics on the transfer from knowledge acquisition to knowledge application in complex problem solving☆

Highlights

Abstract

Section snippets

Complex problem solving and knowledge transition

Sample characteristics

RQ1: How many students who solved the knowledge acquisition phase fail in the knowledge application phase?

Discussion

Conclusion

Credit author statement

Declaration of competing interest

Acknowledgments

Cognitive Psychology

Journal of Memory and Language

Intelligence

Computers in Human Behavior

Computers & Education

Acta Psychologica

Acta Psychologica

Computers & Education

Computers in Human Behavior

Intelligence

Computers & Education

Computers & Education

Computers in Human Behavior

Computers & Education

Intelligence

Computers in Human Behavior

Intelligence

Learning and Individual Differences

Developmental Review

Computers in Human Behavior

Intelligence

Computers in Human Behavior

Intelligence

International Journal of Man-Machine Studies

Intelligence

Computers in Human Behavior

Automatic coding procedures for collaborative problem solving

Acquisition of cognitive skill

Psychological Review

Learning rapid and precise skills

Psychological Review

Thinking about applications: Effects on mental models and creative problem-solving

Creativity Research Journal

Fitting linear mixed-effects models using lme4

Journal of Statistical Software

Basic topics and approaches to the study of complex problem solving

Cognitive flexibility and adaptability to environmental changes in dynamic complex problem-solving tasks

Ergonomics

Developing and assessing ERP competencies: Basic and complex knowledge

Journal of Computer Information Systems

All other things being equal: Acquisition and transfer of the control of variables strategy

Child Development

Ability- or competence-based education: Bridging the gap between knowledge acquisition and ability to apply

Higher Education

Dynamic systems as tools for analysing human judgement

Thinking & Reasoning

Complex problem solving

Operationalizing and detecting disengagement within online science microworlds

Educational Psychologist

From log files to assessment metrics: Measuring students' science inquiry skills using educational data mining

The Journal of the Learning Sciences

Problem solving in technical domains: How mental models and metacognition affect performance

Instruction: Theoretical and applied perspectives

Advancing the science of collaborative problem solving

Psychological Science in the Public Interest

Full length article
Lost in transition – Learning analytics on the transfer from knowledge acquisition to knowledge application in complex problem solving☆