Evidence of fairness: Twenty-five years of research in Assessing Writing

Highlights

• Five fairness reseach trends identified: bias, validity, social context, legal responsibility, and ethical obligation.

• Findings show researchers have diverse categories in gathering evidence of fairness.

• Findings reveal that we do not share a single taxonomy for researching fairness.

• Individual score use and contextual factors invite future research that works to structure opportunity for students.

Abstract

When Assessing Writing (ASW) was founded 25 years ago, conversations about fairness were very much in the air and illustrated sharp divides between teachers and educational measurement researchers. For teachers, fairness was typically associated with consistency and access. For educational measurement researchers, fairness was a technical issue: an assessment that did not identify the presence of β (the bias factor) was fair. Since its founding, ASW has continued to be a space where evolving discussions about fairness play out. In this article, we examine a selection of 73 ASW research studies published from 1994 to 2018 that use fairness as a category of evidence. In tracing the use of fairness and related terms across these research articles, our goal is to understand how the conversation about fairness has changed in the last quarter century. Following a literature review that situates fairness within generational, standards-based, and evidential scholarship, we analyze five trends in the journal: fairness as the elimination of bias; fairness as the pursuit of validity; fairness as acknowledgement of social context; fairness as legal responsibility; and fairness as ethical obligation. A tidy narrative that theoretical conceptualization of fairness has deepened over the ASW lifespan is not born out by our findings. Instead, evidence suggests that the disparate stances and methodological challenges that informed early research on fairness remain. As well, the textual record suggests that we have not developed or shared taxonomies for systematically investigating questions of fairness. In our desire to make the research we present actionable, we close by calling attention to the need for theorization of fairness, the advantages nuanced of research methods, and the benefits of non-Western perspectives.

Introduction

In the first editorial to Assessing Writing (ASW) in 1994, Brian Huot wrote about the genesis of the journal, which stemmed from the 1992 New Directions in Portfolios Conference (Black, Daiker, Sommers, & Stygal, 1994; Black, Helton, & Sommers, 1994). As Huot explained in his introduction, there were few venues for writing researchers to publish on assessment despite the wealth of scholarship presented at conferences. For Huot and co-editor Kathleen Blake Yancey, the journal was a space where “the crucial relationship between pedagogy and assessment” would be valued (Huot & Yancey, 1994, p. 143). There would not be a sole focus on educational measurement perspectives; instead, as the editors established in their first editorial of the second volume, ASW would “contribute to the increasingly interesting and divergent conversations about assessment currently taking place” (Huot & Yancey, 1995, p. 1).

Indeed, early issues of the journal created a space where teachers of writing, writing program administrators, and writing researchers could find a place to work through the issues that interested them in the ways they believed were viable for their programs and students. The traditions of measurement and composition would be in conversation. Notions of fairness would be important here. There was a sense of justice—that what researchers were publishing in the journal was not solely about assessing writing more efficiently or accurately; rather, the research was about making the teaching and assessment of writing fairer. In other words, the conversation was not just about technical advances in-and-of themselves but about examining the social conditions created by and through writing assessment. In fact, one might argue that because of its resonances with justice, democracy, and social good, the very notion of fairness has been integral to ASW since its inception.

Two articles from the first issue of ASW illustrate how fairness informed the journal from its inception. In “Validity in High Stakes Writing Assessment: Problems and Possibilities,” Pamela Moss, a trained psychometrician then assistant professor at University of Michigan, wrote: “Extensive research has been conducted on how to develop and score standardized writing tasks to provide reliable, valid, and fair estimates of students’ writing abilities (e.g., Breland, Camp, Jones, Morris, & Rock, 1987; Huot, 1990; Ruth & Murphy, 1988)” (1994, p. 109). In citing Hunter Breland, a former engineer and then researcher at Educational Testing Service (ETS), in relation to Brian Huot, then assistant professor of English at University of Louisville who advocated for linking writing assessment with learning, Moss was clearly making connections between the measurement and composition communities. Moss went on to connect questions of fairness to “consequential decisions about individuals and programs,” opportunity to learn, and research on “the cognitive and social aspects of learning” (p. 110, p. 109).

If Moss was attempting to put competing notions of fairness in dialog, Michael Williamson (1994), an English professor who specialized in writing assessment and had studied with the psychometrician Michael J. Zieky at ETS, was interested in thinking about the history of education and the implications of fairness in testing beyond educational settings. In “The Worship of Efficiency: Untangling Theoretical and Practical Considerations in Writing Assessment,” Williamson associated fairness with rise of rationalist methods at the beginning of the 20th century: “The positivist science of psychometrics that developed in the late nineteenth century connected to [a] shift in education began to provide assessment tools believed to be objective and fair because they were seen as independent of the bias of the human decisions of individual teachers” (Williamson, 1994, p. 151). Here, Williamson identified a concept of fairness associated with assessment methods designed to be distributed across settings—and, hence, often more reliable than valid or fair. He went on to locate another concept of fairness in the “bureaucratic model” of contemporary education, one in which there is a need for fairness in terms of equitable impact. In his description, fairness is related to treatment and the downstream implications of decisions related to access or exclusion (p. 151). Clearly, Williamson was thinking about the ramifications of assessment beyond a single practice and wanted teachers and researchers to think more expansively about the implications of experimental design and social consequence.

We provide as examples these articles in the first two issues of ASW to illustrate the complex discussions about fairness that researchers were already having in the field at the inception of the journal. Note that terminology about race, class, gender, or linguistic difference are not prominent in the quoted articles but that concerns about social inequality are certainly present. Emphasis on these two early articles is a useful lesson that current conversations about fairness, and the very meaning of fairness itself, are not new; rather, discussions of fairness in writing assessment are rooted in much deeper philosophical and methodological discussions in the field. These two articles, both published in 1994, therefore, identify an initial and enduring concern with evidence of fairness that continue to the present writing. That concern has taken many forms, as we will show: from the U.S. founding of the journal in 1994 under Huot and Yancey until 2000; through its internationalization under the editorship of Liz Hamp-Lyons from 2002 to 2017; and continuing under the current editorship of David Slomp and Martin East and their focus on the consequential dimensions of validity, reliability and fairness in international settings.

In the following article, written in response to the call from Slomp and East in their first issue as editors as they aimed to frame the future of writing assessment, we trace this evolving conversation in the pages of ASW from 73 selected research articles that use fairness and related terms as key words. After establishing our literature review (§2), research questions (§3) and methods (§4), we present five trends we identified in the 73 articles: fairness as the elimination of bias (§5.1); fairness as the pursuit of validity (§5.2); fairness as acknowledgement of social context (§5.3); fairness as legal responsibility (§5.4); and fairness as ethical obligation (§5.5). We then discuss our findings in terms of our research questions (§6) and conclude with recommendations (§7) and conclusions (§8) that may prove useful to writing assessment researchers.