Elsevier

Intelligence

Volume 83, November–December 2020, 101502
Intelligence

The moderating effect of prior knowledge on the relationship between intelligence and complex problem solving – Testing the Elshout-Raaheim hypothesis

https://doi.org/10.1016/j.intell.2020.101502Get rights and content

Highlights

  • Prior knowledge moderates the relation of intelligence and complex problem solving.

  • The Elshout-Raaheim hypothesis predicts a curvilinear moderation.

  • Prior knowledge increased over the course of complex problem solving (CPS) tasks.

  • Over CPS tasks, intelligence-CPS-correlations increased and then decreased.

  • The effect occurred in the acquisition and in the application phase of CPS.

Abstract

Although the relation between intelligence and complex problem solving (CPS) has been investigated repeatedly, the moderating effect of prior knowledge on this relation still remains an open question. The Elshout-Raaheim hypothesis (see Leutner, 2002) predicts a higher correlation between intelligence and CPS at a medium level of problem specific prior knowledge and lower correlations at higher as well as lower levels of prior knowledge, thus a curvilinear moderation. We investigated this hypothesis in a sample of N = 495 high school students by using general intelligence (g) and a minimal complex systems approach (MCS) of CPS. Strategic prior knowledge in the sense of the relative frequency of the vary-one-thing-at-a-time strategy (VOTAT; Tschirgi, 1980) increased across MCS tasks in the knowledge acquisition phase of CPS. With increasing prior knowledge, correlations followed the predicted inverted U-shaped pattern in the knowledge acquisition phase and the knowledge application phase of CPS, thus supporting the Elshout-Raaheim hypothesis. The moderating effect of strategic prior knowledge for the intelligence–CPS relation and its relevance are discussed.

Introduction

Human problem solving takes place whenever a given state is transformed into a goal state with no available routine solution (Mayer & Wittrock, 2006). Corresponding problems differ substantially regarding the transformations that are necessary to reach a desired goal state. In complex problem solving research, real-life problems were transferred into computer-simulated problem scenarios (so called microworlds) to examine the human process and performance when engaging in complex problem solving (CPS; e.g., Dörner, 1986; Dörner, Kreuzig, Reither, & Stäudel, 1983; Süß & Kretzschmar, 2018). Any type of human problem solving can be described as a core component of human intellect, and the relation between intelligence and CPS performance has been investigated in numerous studies (e.g., Dörner & Kreuzig, 1983; Kretzschmar, Neubert, Wüstenberg, & Greiff, 2016; Kröner, Plass, & Leutner, 2005; Lotz, Sparfeldt, & Greiff, 2016; Stadler, Becker, Gödker, Leutner, & Greiff, 2015). Nevertheless, the role of prior knowledge and its moderating influence on the relation between intelligence and CPS performance has not been conclusively resolved.

The Elshout-Raaheim hypothesis (Leutner, 2002; see Elshout, 1987; Raaheim, 1988) postulates that prior knowledge moderates the relation between intelligence and CPS. Specifically with increasing prior knowledge, the correlation coefficients between intelligence and CPS are hypothesized to result in an inverted U-shaped curve. Recent advances in the assessment of CPS and particularly the emergence of the minimal complex systems approach (MCS) allow an advanced reconsideration of the Elshout-Raaheim hypothesis. Therefore, this study examined the Elshout-Raaheim hypothesis again, thereby considering the two established CPS phases knowledge acquisition and knowledge application of a test in the MCS approach (i.e., MicroDYN; Greiff, Wüstenberg, & Funke, 2012). In accordance with prior studies, the variation of prior knowledge was operationalized by the assumption of an increase of prior knowledge from CPS task to CPS task in both phases. In extension to prior studies (Leutner, 2002), this study investigated the Elshout-Raaheim hypothesis with a psychometrically strong CPS assessment. In contrast to Leutner's study, the CPS assessment in our study was realized in one single session, circumventing a split of trials across days and corresponding confounds. Furthermore, we inspected the Elshout-Raaheim–hypothesis for both CPS phases. Additionally, we investigated the assumed increase of prior knowledge empirically by examining the application of the VOTAT-strategy (vary-one-thing-at-a-time strategy; Tschirgi, 1980) in the knowledge acquisition phase for each task.

Complex problem solving is characterized as a none-routine process and a special type of problem solving, in which a system is transformed from its current state to a goal state (Jonassen, 2000; see also Mayer & Wittrock, 2006). Complex problem scenarios designed to assess complex problem solving can be described by five characteristics (Dörner et al., 1983): (a) a system consisting of a number of input as well as outcome variables (complexity), (b) various connections between input variables and outcome variables (connectivity), (c) changes of the problem over time (dynamics), (d) the need to explore the problem (intransparency), and (e) multiple and sometimes even contradictory goals (polytely). One widely accepted specific definition to conceptualize CPS is provided by Buchner (in Frensch & Funke, 1995) as follows:

“The successful interaction with task environments that are dynamic (i.e., change as a function of user's intervention and/or as a function of time) and in which some, if not all, of the environment's regularities can only be revealed by successful exploration and integration of the information gained in that process” (p. 14).

The mentioned process-character of complex problem solving as well as the aim to investigate real-life problem solving resulted in CPS tasks that simulated real world problems in the laboratory by using computer models of complex scenarios. The so-called microworlds deal with, for example, the job of a small-town mayor (Lohhausen; Dörner et al., 1983) or a business manager of a factory (Tailorshop; Putz-Osterloh, 1981). Such microworlds featuring a high degree of complexity and interconnectivity can claim a high degree of face validity (e.g., Stadler et al., 2015). Nevertheless, these early assessment procedures faced severe difficulties in meeting the standards of satisfactory psychometric measurement, as has been mentioned before by different authors (e.g., Greiff & Funke, 2010; Kröner, 2001). Specifically, single task simulations (e.g., Tailorshop) are time consuming and do not allow a variation in task difficulty; the corresponding estimates of reliability were low or even unknown (see Kröner, 2001; Süß, 1996), and single (even random) errors at the beginning of the solving process of such a CPS task can heavily compound overall test performance.

To overcome the shortcomings of single task scenarios and, especially, to improve the psychometric properties, the minimal complex systems (MCS) approach was introduced, in which extensive single task scenarios were replaced by several, relatively short scenarios with reduced complexity (e.g., Greiff et al., 2012; Sonnleitner et al., 2012). MCS scenarios like MicroDYN (Greiff et al., 2012) are based on a linear structural equations approach (LSE; Funke, 2001) and consist of a number of single problem-solving tasks. Each CPS task contains a set of input variables and a set of output variables that are related to each other (see Fig. 1). These relations are composed of direct and indirect effects. Direct effects can be described as a direct link between an input- and an output-variable. Indirect or dynamic effects imply a connection of an output-variable with itself, leading to an increase or a decrease of the variable as a function of time. Theoretically, two phases of problem solving are differentiated (e.g., Funke, 2001; Greiff, Fischer, Stadler, & Wüstenberg, 2015): In a first step, problem solvers construct a mental representation of the problem. In a second step, the problem solvers apply this knowledge about the problem in order to reach a desired goal state. In accordance with these theoretical considerations, single problem solving tasks of MCS assessments such as MicroDYN comprise two different phases (Greiff, Fischer, et al., 2015): the knowledge acquisition phase and the knowledge application phase. In the knowledge acquisition phase, participants are asked to explore the system in order to identify the non-transparent relations of input and output variables. After the knowledge acquisition phase, participants are given the correct causal diagram providing them with the full information about the variables' relations and dynamics. In the following knowledge application phase participants have to reach given target values in output variables within a limited number of manipulations of the input variables. The information given to participants after the knowledge acquisition phase allows them to work on the knowledge application task independent of how they performed in the previous knowledge acquisition phase. Despite this conceptual independence of both phases, a correlation was repeatedly reported between them of at least medium size (Greiff et al., 2012).

More recently developed CPS assessments based on the MCS approach display a number of important characteristics. Within the MCS approach, the problem of one-item-testing inherent to classical scenarios was solved, allowing a variation of the item difficulties between tasks (Stadler, Niepel, & Greiff, 2016). Furthermore, the psychometric properties were usually convincing. For example, the MicroDYN scenario revealed at least good reliability estimates (for the knowledge acquisition phase as well as for the knowledge application phase; e.g., α ≥ .85; Greiff et al., 2012). The theoretically derived CPS dimensions knowledge acquisition and knowledge application were supported in confirmatory factor analyses (Greiff & Neubert, 2014). Additionally and regarding the relations with other important variables, evidence for convergent construct validity of MCS with intelligence (see Stadler et al., 2015) as well as incremental predictive validity of MCS in explaining school grades beyond, for example, figural reasoning (as a narrow indicator for intelligence) was found (e.g., Wüstenberg, Greiff, & Funke, 2012). To conclude, MCS scenarios allow psychometrically convincing assessments of CPS.

The conceptual overlap of CPS and intelligence is well illustrated by Gottfredson (1997), defining intelligence as “a very general mental capability that, among other things, involves the ability to reason, plan, solve problems, think abstractly, comprehend complex ideas, learn quickly and learn from experience” (p. 13). Accordingly, problem solving is understood as a key element of intelligence. Whereas intelligence is usually assessed with static, well defined, and transparent problems, CPS is assessed with computer based simulated scenarios that lack several of these static elements and include instead dynamic components (see e.g., Süß & Kretzschmar, 2018). The debate, whether CPS constitutes a construct distinct from intelligence or whether CPS is conceptualized as part of or can be integrated in an extended construct of intelligence has not yet been resolved (e.g., Kretzschmar et al., 2016; Stadler et al., 2015; Stadler, Niepel, & Greiff, 2019). On a conceptual level, intelligence and CPS overlap substantially regarding, for example, the ability to solve problems, as mentioned (see Gottfredson, 1997). However, there are also important conceptual differences such as the interactive aspects and the dynamic elements, which are characteristic of CPS but not of classical intelligence tests (Wüstenberg, Greiff, & Funke, 2012). On an empirical level, a recent meta-analysis reported an uncorrected mean relationship of M(g) = .43 that increased to M(g) = .59 if only psychometrically more convincing MCS assessments of CPS were considered (Stadler et al., 2015). Furthermore, the few studies that used MCS scenarios as CPS indicators as well as broad intelligence operationalizations (in the sense of g; Jensen & Weng, 1994) reported numerically even higher correlations of .65 < r < .85 (Danner, Hagemann, Schankin, Hager, & Funke, 2011; Kretzschmar et al., 2016; Kröner et al., 2005; Lotz, Sparfeldt, & Greiff, 2016). Correlations of this magnitude can be interpreted as an argument to subsume CPS under a possibly extended conceptualization of intelligence. However, CPS indicators revealed in a nested factor modeling approach substantial loadings on both, an intelligence g-factor as well as an independent CPS-factor, leading the authors of that study to argue for the distinctness of CPS from g (Kretzschmar et al., 2016; see also Sonnleitner, Keller, Martin, & Brunner, 2013).

However, up until now the relevance of prior knowledge as potential moderator of these correlations between CPS and intelligence has often been neglected. Prior knowledge has been shown to be an important variable affecting CPS achievements (Süß, 1996; Süß, Kersting, & Oberauer, 1991; Süß & Kretzschmar, 2018). Furthermore, the level of prior knowledge might explain variance in the fuzzy correlation pattern between CPS and intelligence, as outlined in the Elshout-Raaheim hypothesis (see Leutner, 2002).

According to the Elshout-Raaheim hypothesis (Leutner, 2002; see Elshout, 1987; Raaheim, 1988), a moderating effect of prior knowledge on the relation between problem solving and intelligence is assumed: For low levels of prior knowledge, the hypothesis predicts lower correlations between both constructs; for medium levels of prior knowledge a higher correlation is predicted; and for high levels of prior knowledge, the hypothesis predicts again lower correlations between problem solving and intelligence. The described distribution of correlations results, with increasing prior knowledge, in an inverted U-shaped curve of correlation coefficients which can be mathematically characterized as a negative quadratic equation. Conceptually, the hypothesis forecasts that intelligence is particularly important for solving complex problems at a medium level of prior knowledge. Elshout (1987) even described three different zones of prior knowledge: If a person's problem specific prior knowledge is insufficient, the impact of cognitive abilities (i.e., intelligence) is low due to a lack of conceptual understanding of the problem situation, thus leading to a weak relationship between problem solving and intelligence. Above a certain threshold, Elshout described a zone of problematicity in which a person's cognitive abilities interact with prior knowledge towards solving the problem. This process can be described as typical successful problem solving concurring with a stronger relationship between problem solving and intelligence. Above a threshold of high prior knowledge, Elshout postulated a zone of routine with internalized and relatively fast activity, thus leading again to a weak relationship between problem solving and intelligence. Thus, in this zone of routine, former problems changed to routine tasks.

Regarding these theoretical considerations, Leutner (2002, second study) examined the Elshout-Raaheim hypothesis empirically relying on a repeatedly administered CPS scenario and assuming that prior knowledge increased from administration trial to administration trial. CPS was operationalized by a classical computer-based, single-trial scenario called Hunger in the Sahel (Leutner & Schrettenbrunner, 1989) lasting about 50 min. Participants were tested on three trials per day on three consecutive days followed by a delayed trial, summing up to a total of 10 trials. The trial's position (1 to 10) was taken as indicator for different levels of prior knowledge. For each trial, CPS performance was correlated with intelligence. It was hypothesized that the pattern of correlation coefficients would follow an inverted U-shaped curve as predicted by the Elshout-Raaheim hypothesis. However, the results were partially inconclusive: The pattern of correlation coefficients was in accordance with the hypothesis within each of the three days (first day: r = −.20 to r = .05, second day: r = .30 to r = −.05; third day: r = .25 to r = −.15; delayed trial: r = .15). Nevertheless, the overall pattern of the correlation coefficients of Leutner's study was not in accordance with the Elshout-Raaheim hypothesis. Furthermore, seven of ten correlation coefficients were equal to or smaller than zero, although zero or positive correlation coefficients were predicted for all levels of prior knowledge. Possibly, the distribution of trials within and between days as well as the rather poor psychometric properties of traditional CPS scenarios might have contributed to the inconclusive results.

In addition, the assumption of an increase of specific prior knowledge across the 10 trials was not inspected by Leutner (2002). However, there have been studies indicating an increase in strategic prior knowledge across trials in the field of scientific reasoning as well as for MCS scenarios of CPS (Lotz, Scherer, Greiff, & Sparfeldt, 2017; Schauble, 1996; Vollmeyer, Burns, & Holyoak, 1996). In contrast to domain specific prior knowledge, such as general economic knowledge when working on the CPS scenario Tailorshop (Süß & Kretzschmar, 2018), strategic prior knowledge focuses on strategies that facilitate the problem solving process. One such strategy is to vary-one-thing-at-a-time (VOTAT; Tschirgi, 1980; also known as control of variables strategy, Chen & Klahr, 1999; see also Inhelder & Piaget, 1958). For direct, non-dynamic effects, the VOTAT strategy is the most appropriate and efficient exploration strategy (Tschirgi, 1980; Vollmeyer et al., 1996). A reanalysis of the PISA 2012 data (Greiff, Wüstenberg, & Avvisati, 2015) showed that the application of the VOTAT strategy was not only related to the problem solving proficiency of the specific item in which the strategy was applied (r = .67), but also to the problem solving proficiency of a larger set of problems requiring different exploration strategies (r = .61). This relationship of the VOTAT strategy with the problem solving proficiency of a larger set of problems points to the interpretation of VOTAT as “a good indicator of general strategic knowledge” (Greiff, Niepel, Scherer, & Martin, 2016, p. 45).

To assume an increase in strategic behavior across tasks, these tasks need to share some structural characteristics, so that the same strategies can be applied in the tasks and that participants benefit from applying strategic behavior while working on each of the single tasks. As mentioned, for a series of structurally equivalent scientific reasoning problems consisting only of direct effects, an increase in the use of the VOTAT strategy from trial to trial was shown (Schauble, 1996; see also Vollmeyer et al., 1996). In line with these findings, a recent study found a steady increase of the relative frequency of the use of VOTAT in the knowledge acquisition phase across five CPS tasks (Lotz, Scherer, Greiff, & Sparfeldt, 2017). Similar to scientific reasoning problems, these MCS tasks consisted only of direct effects, which made the VOTAT strategy the most efficient way to explore the systems. In conclusion, the VOTAT strategy can be seen as an adequate indicator for strategic prior knowledge and the assumption that prior strategic knowledge increases across a set of structurally equivalent CPS tasks seems reasonable.

Although the relationship between CPS and intelligence has been investigated repeatedly, the role of prior knowledge on the relation between the two constructs still remains largely an open question. Theoretically, the Elshout-Raaheim hypothesis fills in this missing link by predicting a moderating effect through prior knowledge on the relation between CPS and intelligence resulting in an inverted U-shaped pattern of correlation coefficients. The emergence of the MCS approach with a sequence of short tasks of structural equivalence and with strong psychometric properties established good prerequisites to reinvestigate the Elshout-Raaheim hypothesis and to shed some more light on the relation between CPS and intelligence.

Specifically, the present study investigated the Elshout-Raaheim hypothesis (Leutner, 2002; see also Elshout, 1987; Raaheim, 1988) using the MCS approach MicroDYN (Greiff et al., 2012) for the knowledge acquisition and the knowledge application phase. Hypothesis (1) examined the distribution of correlation coefficients between intelligence and single MicroDYN tasks for the knowledge acquisition phase. Following the Elshout-Raaheim hypothesis, we predicted that item-based correlation coefficients ordered in the administered CPS task sequence followed an inverted U-shaped curve. Hypothesis (2) examined the distribution of correlation coefficients between intelligence and single CPS tasks in the knowledge application phase. Again, correlation coefficients ordered in the administered item sequence were predicted to follow an inverted U-shaped curve. In this study, prior strategic knowledge was operationalized by the sequential order of conceptually similar MicroDYN tasks. As an indicator of prior strategic knowledge, the application of the VOTAT-strategy in the knowledge acquisition phase was inspected for each task. We expected an increase across tasks. To conclude, this study extended prior research investigating the Elshout-Raaheim Hypothesis by (1) relying on a psychometrically strong CPS assessment, (2) realizing the CPS assessment in one single session, (3) inspecting both CPS phases, and (4) investigating the assumed increase of strategic prior knowledge empirically.

Section snippets

Sample and procedure

The sample consisted of N = 495 German high school students (n = 264 females, n = 228 males, n = 3 without gender specification; age M = 16.40, SD = 0.94 years).1 The students attended two academic-tracked school types from either 10th grade (Gymnasium, graduation after 12th grade; 12 classes out of 5

Results

Before testing the Elshout-Raaheim hypothesis, we analyzed the relative frequency of the VOTAT use across the five CPS tasks to investigate whether the relative frequency of VOTAT use increased across tasks indicating an increasing level of strategic prior knowledge. The results showed a steady increase of the relative VOTAT frequencies across all five MicroDYN tasks (task 1/2/3/4/5: 47%/58%/63%/67%/71%; see Lotz, Scherer, Greiff, & Sparfeldt, 2017, for details3

Discussion

The present study investigated the moderating effect of strategic prior knowledge on the relation between CPS and intelligence. Relying on the Elshout-Raaheim hypothesis (Leutner, 2002; see Elshout, 1987; Raaheim, 1988), we postulated an inverted U-shaped pattern of the correlation coefficients of CPS and intelligence with increasing strategic prior knowledge. Strategic prior knowledge was indicated by the relative frequency of the VOTAT strategy in each task of the knowledge acquisition phase.

Acknowledgments

We are particularly grateful to Christin Lotz for her valuable conceptual and empirical work regarding effective strategic behavior while solving complex problems and coordinating the data scollection. We are thankful to Sascha Ludwig who supported the analyses by extracting the relevant data from the log files. We thank Anna Auth, Elena Groh, and Kerstin Mayer, who participated in an excellent manner on the data collection as well as Johannes Schult for statistical support. Furthermore, we are

References (51)

  • J. Scharfen et al.

    Retest effects in cognitive ability tests: A meta-analysis

    Intelligence

    (2018)
  • M. Schwichow et al.

    Teaching the control-of-variables strategy: A meta-analysis

    Developmental Review

    (2016)
  • T. Seufert

    Supporting coherence formation in learning from multiple representations

    Learning and Instruction

    (2003)
  • P. Sonnleitner et al.

    Students' complex problem-solving abilities: Their structure and relations to reasoning ability and educational success

    Intelligence

    (2013)
  • M. Stadler et al.

    Complex problem solving and intelligence: A meta-analysis

    Intelligence

    (2015)
  • M. Stadler et al.

    Easily too difficult: Estimating item difficulty in computer simulated microworlds

    Computers in Human Behavior

    (2016)
  • M. Stadler et al.

    Differentiating between static and complex problems: A theoretical framework and its empirical validation

    Intelligence

    (2019)
  • S. Valerius et al.

    Consistent g- as well as consistent verbal-, numerical- and figural-factors in nested factor models? Confirmatory factor analyses using three test batteries

    Intelligence

    (2014)
  • R. Vollmeyer et al.

    The impact of goal specificity on strategy use and the acquisition of problem structure

    Cognitive Science

    (1996)
  • S. Wüstenberg et al.

    Complex problem solving – More than reasoning?

    Intelligence

    (2012)
  • M. Brunner et al.

    Analyzing the reliability of multidimensional measures: An example from intelligence research

    Educational and Psychological Measurement

    (2005)
  • O. Chen et al.

    The expertise reversal effect is a variant of the more general element interactivity effect

    Educational Psychology Review

    (2017)
  • Z. Chen et al.

    All other things being equal: Acquisition and transfer of the control of variables strategy

    Child Development

    (1999)
  • D. Dörner

    Diagnostik der operativen Intelligenz [diagnostics of operative intelligence]

    Diagnostica

    (1986)
  • D. Dörner et al.

    Problemlösefähigkeit und Intelligenz [Problem solving ability and intelligence]

    Psychologische Rundschau

    (1983)
  • Cited by (6)

    • Assessment in STEM Problem-Solving: A Systematic Review

      2022, International Journal of Assessment and Evaluation
    View full text