Stealth assessment of creativity in a physics video game

https://doi.org/10.1016/j.chb.2020.106647Get rights and content

Highlights

  • Stealth assessment was used to embed creativity measurement in a learning game.

  • The stealth assessment of creativity was validated using external measures of creativity.

  • Creativity directly predicted game performance and enjoyment controlling for incoming knowledge.

  • Creativity did not predict learning when controlling for the incoming knowledge.

  • An indirect relationship between creativity and learning through game performance was proposed.

Abstract

Creativity has been of research interest to psychologists dating back many decades, and is currently recognized as one of the essential skills needed to succeed in our complex, interconnected world. One medium that has affordances to assess and support creativity in young people is video games. In this paper, we briefly discuss the literature on video games and creativity and provide an example of current work being done relative to measuring creativity in the context of a game called Physics Playground using stealth assessment. To validate the stealth assessment of creativity, we conducted a one-group pretest-posttest study with 167 8th and 9th graders from a K-12 school in Florida. Results suggest that our stealth assessment of creativity is valid (i.e., our stealth assessment estimate significantly correlated with our external performance-based measures of creativity). Additional analyses revealed that creativity (i.e., estimated using our stealth assessment of creativity) significantly predicts in-game performance (e.g., number of levels solved), game enjoyment, and learning of physics content. We conclude with a discussion of future directions in this line of creativity research.

Introduction

Most of us know creativity when we see it. For instance, consider a group of people living in a remote village in Africa with no electricity or the financial means to buy washing machines. A product design student, Richard Hewitt, came up with a creative idea to couple a bicycle and a large container. Then, with just a bit of re-engineering, he had the SpinCycle Washing Machine (Hewitt, 2012; see Fig. 1). This idea came to Richard's mind during a visit to Burundi in Central Africa, after washing about 30 loads of clothes by hand.

Going beyond the final product (like a piece of art, musical score, or clever invention), we want to focus on the processes that provide evidence for creativity. In line with this aim, we define creativity as the abilities needed to produce ideas or solutions that are novel yet appropriate for the problem at hand. Moreover, we argue that well-designed games provide an excellent vehicle for capturing and analyzing these processes that can evolve into creative solutions.

Creativity has been of research interest to psychologists dating back many decades, and is currently recognized as one of the essential skills needed to succeed in our complex, interconnected world (e.g., the Partnership for 21st Century Learning, 2019). That is, we are living in a creative society where one's success is based on the ability to think and act creatively. However, despite the recognized importance of creativity, current school systems do not adequately prepare younger people to become creative thinkers (Sawyer, 2011).

As mentioned, one medium with affordances to assess and support creativity in young people is video games. Playing video games is one of the most popular activities for people of all ages. According to the Global Games Market Report (Wijman, 2019) there are more than 2.5 billion gamers across the world. Another recent report analyzed responses to gameplay-related questions from 4500 gamers (ages 18 and older) residing in nine countries (i.e., France, Germany, India, Italy, Japan, Singapore, South Korea, the United Kingdom, and the United States). On average, gamers spend 7 h and 7 min playing video games per week (The State of Online Gaming, 2019). A study on media usage in the U.S. reported that 67% of youth (ages 8 to 18) spend an average of 73 min daily playing video games, compared with only 38 min daily reading print materials (Rideout, Foehr, & Roberts, 2010). Another indicator of the popularity of gaming is that annually, Americans spend $43.4 billion on video/game-related purchases (Entertainment Software Association, 2019); and globally, people spend about $150 billion on games (Wijman, 2019).

So, how can video games cultivate creativity? Will Wright (2006), a renowned game designer, argues that video games are “dream machines” that have the ability to unleash human imagination. He explains that a game is a “possibility space” in which video games start at a well-defined state and end when a specific state is reached. How players reach a specific goal is open-ended, and each player can navigate this possibility space by making continuous choices and actions.

Gee (2005) similarly describes how a well-designed game incorporates good learning principles that can support players’ creativity. First, players are not simply consumers of the game but producers by making their own actions and choices. At a fundamental level, what players do and create in the game to progress through levels is a form of production. For example, in the popular “god” game called Spore, players create their own species, and then the species evolve into more intelligent creatures and civilization. Some games, such as LittleBigPlanet, Portal 2, and Physics Playground have built-in level editor functionality that allows players to modify the games and even create their own levels. Second, good games often encourage players to take risks, explore and try new things, and learn by failing. Failing is not a bad thing in games as it is in traditional education. In fact, failing is a great way to get feedback about progress. Third, video games are “pleasantly frustrating.” That is, tasks in a well-designed game are challenging, but reside within a range of difficulty levels. This gives players a great sense of accomplishment upon completing the task.

Despite this inherent link between creativity and video games, there is limited and rather mixed evidence for relationships between playing video games and creativity. For example, Hamlen (2009) investigated the relationship between self-reported time spent playing video games per week and performance on the Torrance Tests of Creative Thinking (TTCT; Torrance, 1972) in 4th and 5th graders. She reported that the number of hours of gameplay does not significantly predict TTCT performance controlling for gender and grade. In contrast, Jackson and colleagues (Jackson et al., 2012) investigated the relationship between gameplay time (i.e., participants’ response to how often do you play videogames?) and creativity using the TTCT, and they reported that playing video games is significantly associated with creativity.

Although investigating relationships between video gameplay and creativity may be interesting, this line of research does not directly help educators and practitioners to use video games to foster creativity. That is, studies that have investigated the relationship between playing video games and creativity in general (using correlational analyses) are often based on the assumption that creativity is a “general” construct, and do not consider the possible interplay with or dependence on domains. However, Baer and Kaufman (2005), using their Amusement Park Theory (APT) of creativity, suggest that for any creative work to happen, there are some requirements that need to be present. For example, a person working within a particular domain must have at least some basic knowledge about that domain before creative work may emerge. Similarly, Csikszentmihalyi's (1997) theory of creativity includes domain (e.g., math, arts, science, etc.) as one of the components of the creativity model, and creativity occurs at the intersection of person, domain, and field (i.e., experts in the field who can judge the creative work).

Another problem with some existing studies on games and creativity is that they do not clearly state how creativity is defined (see Plucker, Beghetto, & Dow, 2004 for more on this issue). Also, the way in which creativity is assessed in these studies is problematic, where many studies view creativity as unidimensional. Moreover, correlational studies do not systematically examine how specific aspects of creativity manifest in video games. Finally, such general studies (e.g., Hamlen, 2009; Jackson et al., 2012) typically treat all game genres as equal, but that can be misleading as some genres have more potential to enhance creativity than others.

There are, however, some studies reported in the literature (using experimental and quasi-experimental research design) that do use games that are domain relevant and/or have potential for enhancing creativity (e.g., Minecraft, Portal 2). These games allow players to co-create the gaming environment (using the game's create mode or level editor); hence, they permit players to be creative rather than passively receiving game problems to solve. For instance, Fessakis and Lappas (2013) used a physics-related puzzle game called Crayon Physics Deluxe (Kloonigames, 2014) to investigate the effects of this game on students' creativity. Similarly, Moffat, Crombie, and Shabalina (2017), and Inchamnan, Wyeth, and Johnson (2013) used another popular game called Portal 2 (with a high potential to enhance creativity) to investigate the effects of playing this game on participants' creativity, compared to playing two other puzzle games that were had low potential for enhancing creativity (i.e., I-Fluid, and Braid). Blanco-Herrera, Gentile, and Rokkum (2019) used Minecraft (another game with high potential to enhance creativity) to compare its effects on creativity compared to a racing game called NASCAR, and watching a TV show. The findings of all of the aforementioned studies showed positive results indicating that certain games (e.g., Crayon Physics Deluxe, Portal 2, Minecraft) that engage players in solving interesting problem, creating virtual environments in the games, and designing new game levels, can enhance people's creativity compared to other video games (i.e., racing or shooting games)—see (in press) Rahimi & Shute, for a full review on the effects of videos games on creativity.

To support creativity using video games in the broader education community, we need to understand the affordances of video games in relation to the multidimensional aspects of creativity. That is, the first question we should ask is: What are some of the cognitive and noncognitive dimensions of creativity that are part of playing video games? In addition, attention needs to be paid to assessment methods that use creative behaviors and products that players create in and outside of video games (Plucker & Makel, 2010). Such behaviors and products are believed to be more valid indicators of creativity than commonly used self-report measures of creativity (McClelland, 1973; Shute, Ventura, & Kim, 2013).

The purpose of this paper is to threefold: (1) review the current literature of creativity and link the literature with the mechanics of popular games that foster creative endeavors; (2) describe a methodology called stealth assessment as a way to assess creativity in the context of a learning video game called Physics Playground (Shute & Ventura, 2013); and (3) provide empirical support for the construct and criterion validity of our stealth assessment measure of creativity.

Section snippets

Multiple Dimensions of Creativity

There have been countless arguments over the accepted definition of creativity among psychologists across the decades. Despite this lack of agreement, there are some common notions of creativity that run through the literature. First, creativity is generally defined as the ability to produce solutions, ideas, or products that are both novel and effective (Lubart, 1994). Kaufman and Sternberg (2007) similarly have noted that most definitions of creativity consist of three components: novelty,

Participants and research design

The participants of this study consisted of 167 8th and 9th graders (76 male and 91 female; 13-15 years old) from a K-12 school in Florida. Upon the completion of the study, each student received a $25 gift card. We used a one-group, pretest-posttest research design.

Procedure

The total gameplay time was about 4 h (across six 45-min sessions in a week). Thirty computers, in one of the school's two computer labs, were used for this study. Separators were used between the computers to make sure that

Results

To address research question 1 regarding the criterion-related validity of our stealth assessment estimate of creativity (computed using BNs), we conducted several correlational analyses. As expected, our creativity estimate correlated with our external, performance-based measures of creativity as follows: (a) Alternative Uses test—fluency (r = 0.18, p = .02); (b) Alternative Uses test—originality (r = 0.18, p = .02); and (c) Student-made levels (r = 0.23, p = .01). There was no significant

Discussion and future research

As playing video games has become a key part of everyday life for today's youth, the broader education community has been exploring affordances of video games to measure and support competencies that are valuable to success in the 21st Century. In this paper, we discussed how one such game—Physics Playground—can be used as a vehicle to measure creativity. We examined research questions related to the validity of our stealth assessment estimate of creativity, as well as the effects of creativity

Declaration of interest

We wish to confirm that there are no known conflicts of interest associated with this manuscript.

CRediT authorship contribution statement

Valerie J. Shute: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editing. Seyedahmad Rahimi: Formal analysis, Investigation, Methodology, Resources, Software, Validation, Writing - original draft, Writing - review & editing.

Acknowledgments

This work was supported by the US National Science Foundation [award number #037988] and the US Department of Education [award number #039019]. We'd also like to thank team members who helped in this project—Russell Almond, Mathew Ventura, Yoon Jeon Kim, Lubin Wang, and Weinan Zhao.

References (89)

  • A.M. Abdulla et al.

    After six decades of systematic study of creativity: What do teachers need to know about what it is and how it is measured?

    Roeper Review

    (2017)
  • B. Akram et al.

    Improving stealth assessment in game-based learning with LSTM-based analytics

    (2018)
  • R.G. Almond

    An IRT-based parameterization for conditional probability tables

  • R.G. Almond et al.

    Bayesian networks in educational assessment

    (2015)
  • T.M. Amabile

    The social psychology of creativity: A componential conceptualization

    Journal of Personality and Social Psychology

    (1983)
  • J. Baer et al.

    Bridging generality and specificity: The amusement park theoretical (APT) model of creativity

    Roeper Review

    (2005)
  • R.S. Baker et al.

    Towards general models of effective science inquiry in virtual performance assessments

    Journal of Computer Assisted Learning

    (2016)
  • J. Barojas et al.

    Physics and creativity: Problem solving and learning contexts

    Industry and Higher Education

    (2001)
  • M.D. Berzonsky et al.

    Social-cognitive aspects of identity style: Need for cognition, experiential openness, and introspection

    Journal of Adolescent Research

    (1992)
  • V.M. Cheng

    Developing physics learning activities for fostering student creativity in Hong Kong context

  • P.C. Cheung et al.

    Creative potential of school children in Hong Kong: Norms of the Wallach-Kogan Creativity Tests and their implications

    Creativity Research Journal

    (2004)
  • A.J. Cropley

    A note on the Wallach‐Kogan tests of creativity

    British Journal of Educational Psychology

    (1968)
  • A.J. Cropley

    Originality scores under timed and untimed conditions

    Australian Journal of Psychology

    (1972)
  • M. Csikszentmihalyi

    Creativity: Flow and the psychology of discovery and invention

    (1997)
  • M.E. DeRosier et al.

    Zoo U: A stealth approach to social skills assessment in schools

    Advances in Human-Computer Interaction

    (2012)
  • T. Dewett

    Linking intrinsic motivation, risk taking, and employee creativity in an R&D environment

    R & D Management

    (2007)
  • I.A.N. Diakidoy et al.

    Creativity in physics: Response fluency and task specificity

    Creativity Research Journal

    (2001)
  • C. Dippo

    Evaluating the alternative uses test of creativity

  • Essential facts about the computer and video game industry

    (2019)
  • G.J. Feist

    The influence of personality on artistic and scientific creativity

  • G. Fessakis et al.

    Cultivating preschoolers creativity using guided interaction with problem solving computer games

  • J.P. Gee

    Learning by design: Good video games as learning machines

    (2005)
  • J.A. Glover

    Risky shift and creativity

    Social Behavior and Personality: International Journal

    (1977)
  • J.A. Glover et al.

    Relation of four components of creativity to risk-taking preferences

    Psychological Reports

    (1977)
  • J.D. Gobert et al.

    From log files to assessment metrics: Measuring students' science inquiry skills using educational data mining

    The Journal of the Learning Sciences

    (2013)
  • J.P. Guilford

    The structure of intellect

    Psychological Bulletin

    (1956)
  • K.R. Hamlen

    Relationships between computer and video game play and creativity among upper elementary school students

    Journal of Educational Computing Research

    (2009)
  • R. Hewitt

    SpinCycle project [Weblog]

    (2012)
  • W. Inchamnan et al.

    Does activity in computer game play have an impact on creative behaviour? 2013

  • P.W. Jackson et al.

    The person, the product, and the response: Conceptual problems in the assessment of creativity

    Journal of Personality

    (1965)
  • A.M. Kanar et al.

    Guiding learners through technology-based instruction: The effects of adaptive guidance design and individual differences on learning over time

    Journal of Educational Psychology

    (2013)
  • J.C. Kaufman et al.

    Do people recognize the four Cs? Examining layperson conceptions of creativity

    Psychology of Aesthetics, Creativity, and the Arts

    (2013)
  • J.C. Kaufman et al.

    Resource review: Creativity

    Change

    (2007)
  • F. Ke et al.

    Assessing game-based mathematics learning in action

  • Cited by (53)

    • Toward Asset-based Instruction and Assessment in Artificial Intelligence in Education

      2024, International Journal of Artificial Intelligence in Education
    • Assessment Framework for Immersive Learning: Application and Evaluation

      2024, Communications in Computer and Information Science
    View all citing articles on Scopus
    View full text