The contribution of argument knowledge to the comprehension and critical evaluation of argumentative text

Highlights

• Argument knowledge facilitates comprehension and argument evaluation outcomes.

• Comprehension of argumentative text contributes to argument evaluation.

• Adolescents can recognize common informal reasoning fallacies in arguments.

Abstract

The purpose of this study was to develop a comprehensive test to estimate argument knowledge and examine directly the influence of this knowledge on the comprehension of argumentative text and the critical evaluation of its arguments. Also, the possible contribution of comprehension to argument evaluation was explored. Sixth and ninth graders read two completely balanced two-sided argumentative texts on controversial issues before answering comprehension questions and evaluating the quality of the text-based arguments. Argument quality was varied systematically according to the presence or absence of the ad populum and ad ignorantiam fallacies. Results documented the facilitating role of argument knowledge on both comprehension and evaluation and the contribution of comprehension to argument evaluation. The study highlights the potential of argument knowledge as a target for instructional intervention and of the argumentative text as a fertile context for promoting both comprehension and critical thinking objectives.

Introduction

Argumentative text is a primary medium of learning about theories, interpretations, and issues in scientific and social domains (Larson et al., 2009, Larson et al., 2004, Norris and Philips, 2003). Its core content involves arguments in support of a claim. Structurally, however, naturalistic argumentative text can vary in complexity depending on the degree to which it also presents counter-arguments (in support of an opposing claim) and/or rebuttals (Buehl et al., 2001, Hynd, 2001). Therefore, the ability to comprehend argumentative text is an important one to develop, warranting further examination of the factors that may support it. Prior research, however, has shown the comprehension of argumentative text to present a challenge (Diakidoy et al., 2015, Diakidoy et al., 2017, Larson et al., 2004) that may not always be attributable to lack of prior topic knowledge (Kamalski, Sanders, & Lentz, 2008).

The contribution of argumentative text to belief formation and change (Alexander et al., 2001, Britt and Larson, 2003, Buehl et al., 2001, Kardash and Scholes, 1996, Murphy and Alexander, 2004) also highlights the importance of the ability to evaluate critically the arguments presented in such text. Evaluating arguments and rejecting fallacious ones is an integral part of critical thinking skill (Klaczynski et al., 1997, Neuman, 2003, Walton, 2010). The ability to evaluate critically arguments is crucial given the availability of a wealth of often-conflicting information and dubious claims. As a result, the need to better equip students with the necessary critical thinking skills in school has been identified as an urgent one (e.g., Bensley and Spero, 2014, Byrnes and Dunbar, 2014). Critical thinking research, however, has documented the difficulty of bypassing prior beliefs and biases in order to analyze and evaluate arguments objectively (Klaczynski et al., 1997, Macpherson and Stanovich, 2007, Stanovich and West, 1997, Stanovich and West, 1998).

One factor that has not been examined in previous work is prior knowledge about the function, the content, the structure and the quality of an argument. In its simplest form, an argument consists of a claim and a reason supporting it (Toulmin, 1958, Voss and Means, 1991). Typically, however, an argumentative text promotes a claim by presenting several supporting reasons forming, thus, an argument line. This argument line is often elaborated with additional information that may be peripheral or argument-relevant (such as qualifiers and warrants) and may run in parallel with a counter-argument line within a single text. Therefore, argument knowledge may provide an appropriate framework for organizing text content that supports comprehension and allows readers to focus on the relations between claims and reasons that form the object of critical evaluation. The primary goal of this study was to examine the main and interactive effects of argument knowledge on both the comprehension and the evaluation of the content of argumentative text.

The ability to comprehend argumentative texts is important in and out of school as it contributes to completing class assignments, making decisions in everyday life, and participating in a democratic society (Britt and Larson, 2003, Larson et al., 2004, Larson et al., 2009). However, the comprehension of an extended argumentative text is a demanding task, requiring readers to mentally represent an entire line of argument – often in relation to a parallel line of counterargument – and in connection to one or more claims (Diakidoy et al., 2015, Diakidoy et al., 2017). Specifically, the task requires that readers identify and represent critical argument elements, starting with the claim that the set of reasons presented in text aim to support (Larson et al., 2004, Reznitskaya et al., 2007).

Representing the claim promoted by a text is a prerequisite for argumentative text comprehension as it occupies the top of a hierarchical argument structure, subsuming all of the reasons offered in its support (Britt and Larson, 2003, Reznitskaya and Anderson, 2002, Wolfe et al., 2009). However, prior research has demonstrated that even skilled, young adult readers have difficulty representing the main claim that an argumentative text promotes. For example, the results of Larson et al. (2004) indicated only a 30% accuracy in identifying the main claim and supporting reasons in a reading/search task. Similarly, Diakidoy et al., 2015, Diakidoy et al., 2017 found that only about half of their young adult readers were able to remember with accuracy the main claim that a naturalistic argumentative text promoted. Notably, accuracy in this task improved slightly when the reading task was to evaluate the argumentative text content as opposed to comprehend the text (Diakidoy et al., 2017). The authors interpreted this finding to indicate that a goal to evaluate the text served to focus readers’ attention and processing on key argument elements as opposed to all other peripheral topic information.

In contrast, Chambliss (1995) showed that advanced high school readers were able to use claim statements to identify argument parts and to maintain a text’s argument structure in their summaries. In a subsequent study with a younger sample, Chambliss and Murphy (2002) showed that fifth graders were more likely to construct hierarchical than non-hierarchical, list-like representations, when compared to fourth graders. Moreover, these representations were equally likely to reflect the hierarchical structure of the argument as that of the topic. Although these findings are rather optimistic given the comprehension difficulties manifested by young adults in the Diakidoy et al., 2015, Diakidoy et al., 2017 studies, differences in the materials can account for differences in the results. Chambliss (1995) found that high-school students’ performance, as indicated through written response tasks and think-aloud protocols, was higher when a text was both clearly structured and signaled as an argument and its argument structure was simple rather than complex. However, the argumentative texts used in the Diakidoy et al., 2015, Diakidoy et al., 2017 studies were the opposite. They were naturalistic texts that were only implicitly persuasive. The fact that these texts were not clearly structured as argumentative and left it to the reader to infer the main claim promoted compounded readers’ difficulty in representing their argument line with accuracy. Therefore, these results taken together imply that the extent to which younger and older readers experience difficulty in comprehending argumentative texts depends in part on the simplicity and the explicitness of the structure of their argument line.

Nevertheless, Chambliss and Murphy (2002) interpreted their findings to indicate the use of an argument schema, as opposed to a topic schema, by some of the older children. The importance of an argument schema has been further implicated in a wide-range of studies examining the effects of structural characteristics, tutorials, and dialogic argumentation in the processing and evaluation of simple arguments (Britt and Larson, 2003, Larson et al., 2009) and the writing of argumentative essays (Reznitskaya et al., 2007, Wolfe et al., 2009). According to the argument schema theory (Reznitskaya & Anderson, 2002), a well-developed argument schema supports the organization and retrieval of argument-relevant information. Thus, it is reasonable to assume that the availability of a well-developed argument schema would similarly facilitate the automatic activation of argument-relevant knowledge about the function, the parts, the structure and the quality of arguments. This knowledge should support the representation of text-based argument-relevant parts and their interconnections facilitating, thereby, comprehension. Despite its obvious implications for comprehension, no research, to our knowledge, has examined the argument schema theory in relation to the comprehension of argumentative text. In this study, we developed a measure of argument knowledge in order to examine directly the effects of this knowledge on comprehension.

Only a few studies have examined directly the interconnection between comprehension and critical evaluation of arguments and have yielded conflicting results (Diakidoy et al., 2015, Diakidoy et al., 2017, Neuman, 2003, Neuman and Weizman, 2003). It is reasonable to hypothesize that comprehension of an argumentative text is a prerequisite for the critical evaluation of an argument embedded in it or its entire argument line as it provides the representational basis for the evaluation to take place. Further, it is well documented that deep comprehension requires the generation of inferences that specify implicit connections between text parts and support the integration of text information with background knowledge in working memory (Coté, Goldman, & Saul, 1998; Graesser, Millis, & Zwaan, 1997; McNamara & Magliano, 2009). Neuman (2003) results are in line with this hypothesis, indicating that high-school students’ comprehension skill and, more specifically, their ability to generate inferences, was a positive predictor of their ability to identify and eventually reject fallacious arguments. However, Diakidoy et al. (2015) showed the generation of knowledge-based inferences to contribute to positive evaluations of fallacious argument lines in text and to their persuasive impact. Therefore, knowledge-based inferencing that contributes to deep comprehension was found to hinder critical evaluation.

These contrasting results may be explainable on a methodological and theoretical basis. Neuman (2003) employed short argument scenarios and measured comprehension skill separately through performance on the reading comprehension section of a standardized aptitude test. In contrast, Diakidoy et al. (2015) employed an extended text with a structurally complex argument line and measured participants’ comprehension of that text. Therefore, whereas Neuman (2003) has shown that high general comprehension ability facilitates the critical evaluation of simple arguments in short texts, Diakidoy et al. (2015) has shown that the deep comprehension of an argumentative text may not necessarily facilitate the critical evaluation of its arguments or argument line. From a theoretical perspective, it must be noted that although inferencing is associated with deep comprehension, not all inferencing is productive, as it may involve irrelevant associations or even inaccurate prior knowledge (e.g., Coté, Goldman, & Saul, 1998; Kendeou & van den Broek, 2005). In fact, the automatic activation of prior knowledge and beliefs that gives rise to inferences included in the representational outcome has been considered a source of bias (Stanovich & West, 2008) resulting in uncritical evaluations.

Nevertheless, a dissociation between comprehension and critical evaluation is problematic since the comprehension outcome, that is, the resulting representation, provides the basis upon which the evaluation processes operate (Glöckner & Witteman, 2010). However, shifting the focus from the comprehension process to its outcome may help resolve this problem: Several studies have suggested that an accurate representation, as opposed to an elaborated one, may be more conducive for critical evaluation, as it allows readers to focus on the connection between a claim and a reason (Britt et al., 2008, Neuman and Weizman, 2003). For example, Neuman and Weizman (2003) showed that students who were able to recall the fallacy-containing clause in short argument scenarios (in verbatim or paraphrase form) were more likely to identify the fallacy and correctly reject the argument on the basis of that fallacy than students who had not recalled it. The implication of these results is that an accurate and relatively complete text-base representation of an argumentative text may be more conducive to critical evaluation than a highly-elaborated situation model (Kintsch, 1988; McNamara & Magliano, 2009). Therefore, in the present study we measured both literal and inferential comprehension in order to further examine the contribution of memory for text separately from the ability to generate inferences on the evaluation outcomes.

Argumentative text, in printed or electronic form is wide-spread and contributes to belief formation and change (Alexander et al., 2001, Britt and Larson, 2003, Buehl et al., 2001, Kardash and Scholes, 1996). However, not all texts are equal in terms of the quality of the arguments they present in support of claims. One or more arguments (claim-reason pairs) within an argument line can be irrelevant, insufficient, and/or unacceptable (Blair & Johnson, 1987). Violations of any or all of the above criteria (relevance, sufficiency, acceptability) render an individual argument or an entire argument line invalid and fallacious. Nevertheless, fallacious arguments can be highly persuasive (Copi and Burgess-Jackson, 1996; Walton, 2010) contributing to misinformation, unsubstantiated beliefs, and faulty decision making. Therefore, in addition to understanding, readers need to be able to evaluate critically the arguments that comprise the content of an argumentative text in order to identify and reject the fallacious arguments.

Argument evaluation is an integral part of critical thinking skill (Alexander, 2014, Neuman, 2003; Walton, 2010), reflecting the ability to evaluate the quality of arguments and reject fallacious ones independent of prior beliefs or knowledge (Byrnes & Dunbar, 2014). However, the pervasive influence of prior beliefs in argument evaluation (West, Toplak, & Stanovich, 2008) has been well-documented, with studies showing that fallacious but belief-consistent arguments in short scenarios are more likely to be accepted as valid in contrast to belief-inconsistent arguments (Klaczynski et al., 1997, Macpherson and Stanovich, 2007). Research that has examined the evaluation of argumentative text has shown similar uncritical evaluation and acceptance of fallacious argument lines, even in the absence of strong prior beliefs (Diakidoy et al., 2015, Diakidoy et al., 2017).

From a dual-process perspective, these results reflect fast and intuitive processing of information (Evans, 2007, Sloman, 1996). Dual-process models of thinking draw a distinction between intuitive and strategic or deliberate processes in evaluation and decision making (Evans, 2007, Glöckner and Witteman, 2010). Intuitive processing, which is also known as associative or memory-based processing, places fewer demands on a limited-resource cognitive system as the evaluation of arguments relies on automatically activated prior topic knowledge and personal beliefs. As a result, intuitive processing is taken to underlie a number of heuristics and biases some of which may be traced to faulty or incomplete knowledge and/or prior beliefs (Stanovich & West, 2008). In contrast, overcoming belief bias and identifying fallacies in arguments requires the application of more effortful and analytic processes. These processes are more strategic in the sense that they involve the monitoring and control of any automatically-activated products of associative memory and the conscious, sequential application of analytic processes in the consideration of arguments (Evans, 2007; Stanovich, Toplak, & West, 2008). Therefore, critical-analytic processing is often considered as necessary to override intuitive processing and its outcomes (Stanovich, West, & Toplak, 2011).

Considering the influence of prior beliefs (Stanovich and West, 2008, Wolfe and Kurby, 2017) and the fact that fallacious arguments can appear to be well-reasoned and highly persuasive (Copi and Burgess-Jackson, 1996, Walton, 2010), ratings of quality or strength cannot be trusted to provide an accurate indicator of argument quality. Therefore, in this study we operationalized and manipulated argument quality systematically by focusing on two common fallacies: ad populum and ad ignorantiam. The ad populum fallacy promotes that something is true because all or a large portion of the population believes or does so (Ricco, 2007). It appeals to a sympathetic feeling of group solidarity in order to prove a point (Neuman, 2003). Nevertheless, what most people believe doesn’t necessarily establish that this belief is true. On the other hand, the ad ignorantiam fallacy capitalizes on the use of ignorance or lack of knowledge (Neuman, 2003, Ricco, 2007). Such arguments suggest that if something has not been proven to be true, then it is false and vice versa. Prior research has shown that both fallacies can be recognized by adolescents when embedded in short argument scenarios, even when students are not able to explain the nature of the fallacy (Neuman, 2003, Neuman and Weizman, 2003, Weinstock et al., 2006, Weinstock et al., 2004). Moreover, Weinstock et al. (2006) found that 7th graders performed better than 9th graders on ad ignorantiam scenarios and equally with 9th graders on ad populum scenarios. Therefore, although one would expect developmental trends in favor of older students and adults, prior findings present a more complex picture.

The primary aim of the study was to examine the effects of argument knowledge on the comprehension of argumentative text and the evaluation of its individual arguments with a sample of 6th and 9th graders. We targeted these grade levels strategically for three reasons. First, prior research has documented younger students’ ability to represent the hierarchical structure of an argument line when the text follows a simple and explicit argument structure (Chambliss, 1995, Chambliss and Murphy, 2002). The argumentative texts used in the current study were similarly simple and explicit in terms of their structure. Second, reading and writing argumentative text with a focus on the two-sided balanced text is part of the objectives of the national curriculum for 6th grade. Third, 6th grade is the last grade of elementary school while 9th grade is the last grade of middle school in the Cypriot school system. Therefore, differences in the results between these two grade levels may carry educational implications, as well.

On the basis of prior work (Chambliss, 1995, Larson et al., 2009, Reznitskaya et al., 2007), we developed a multi-part, paper-and-pencil questionnaire to measure knowledge of the function, the content, the structure, and the quality of arguments. Students in Grades 6 and 9 were pretested on their knowledge of argument, their verbal ability and their prior beliefs before reading two argumentative texts on different topics. Each text was clearly argumentative presenting explicitly two opposing claims followed by a set of supporting reasons. The quality of the arguments justifying each claim, that is, each claim-reason pair, was varied systematically according to the presence or absence of the ad populum and ad ignorantiam fallacies. These two fallacies are relevant in adolescence when popularity or lack of knowledge may contribute to uncritical acceptance of dubious claims. Although there are several informal reasoning fallacies (e.g. Walton, 1987) that could be potentially useful to study with samples of adolescents, we chose these specific two fallacies because there is prior empirical work that has shown these fallacies to be detectable by adolescent participants (Neuman, 2003, Neuman and Weizman, 2003, Weinstock et al., 2006, Weinstock et al., 2004). As a result, each claim within each text was supported by a set of high-quality arguments and two sets of fallacious arguments. Post-reading tasks included measures of prior topic knowledge, text comprehension, and argument evaluation. It must be noted, however, that only comprehension and evaluation comprised the major outcomes of interest. A measure of prior topic knowledge was included because of its well-known influence on comprehension and as a control in order to account for any topic knowledge differences influencing the results of the study.

Based on prior research (e.g. Chambliss, 1995, Chambliss and Murphy, 2002, Britt et al., 2008, Larson et al., 2004), we hypothesized that a well-developed argument schema would facilitate both the comprehension of argumentative text [Hypothesis 1] and the critical evaluation of its arguments [Hypothesis 2]. Although prior research has shown a general dissociation between comprehension and critical evaluation (Diakidoy et al., 2015, Diakidoy et al., 2017), several studies have also highlighted the importance of a complete and accurate representation of the content of an argument or argument line in its evaluation (Britt et al., 2008, Larson et al., 2009, Neuman and Weizman, 2003). Therefore, we hypothesized that memory for text, as indicated by performance on literal comprehension questions, would facilitate critical evaluation more than inferential comprehension [Hypothesis 3]. Regarding argument evaluation and in light of previous findings (e.g., Neuman, 2003, Neuman and Weizman, 2003, Weinstock et al., 2006), we expected participants to show awareness of argument quality by evaluating fallacious arguments lower than non-fallacious ones [Hypothesis 4]. Finally, we also explored potential age differences in argument knowledge, comprehension and critical evaluation.