Stealth assessment of creativity in a physics video game
Introduction
Most of us know creativity when we see it. For instance, consider a group of people living in a remote village in Africa with no electricity or the financial means to buy washing machines. A product design student, Richard Hewitt, came up with a creative idea to couple a bicycle and a large container. Then, with just a bit of re-engineering, he had the SpinCycle Washing Machine (Hewitt, 2012; see Fig. 1). This idea came to Richard's mind during a visit to Burundi in Central Africa, after washing about 30 loads of clothes by hand.
Going beyond the final product (like a piece of art, musical score, or clever invention), we want to focus on the processes that provide evidence for creativity. In line with this aim, we define creativity as the abilities needed to produce ideas or solutions that are novel yet appropriate for the problem at hand. Moreover, we argue that well-designed games provide an excellent vehicle for capturing and analyzing these processes that can evolve into creative solutions.
Creativity has been of research interest to psychologists dating back many decades, and is currently recognized as one of the essential skills needed to succeed in our complex, interconnected world (e.g., the Partnership for 21st Century Learning, 2019). That is, we are living in a creative society where one's success is based on the ability to think and act creatively. However, despite the recognized importance of creativity, current school systems do not adequately prepare younger people to become creative thinkers (Sawyer, 2011).
As mentioned, one medium with affordances to assess and support creativity in young people is video games. Playing video games is one of the most popular activities for people of all ages. According to the Global Games Market Report (Wijman, 2019) there are more than 2.5 billion gamers across the world. Another recent report analyzed responses to gameplay-related questions from 4500 gamers (ages 18 and older) residing in nine countries (i.e., France, Germany, India, Italy, Japan, Singapore, South Korea, the United Kingdom, and the United States). On average, gamers spend 7 h and 7 min playing video games per week (The State of Online Gaming, 2019). A study on media usage in the U.S. reported that 67% of youth (ages 8 to 18) spend an average of 73 min daily playing video games, compared with only 38 min daily reading print materials (Rideout, Foehr, & Roberts, 2010). Another indicator of the popularity of gaming is that annually, Americans spend $43.4 billion on video/game-related purchases (Entertainment Software Association, 2019); and globally, people spend about $150 billion on games (Wijman, 2019).
So, how can video games cultivate creativity? Will Wright (2006), a renowned game designer, argues that video games are “dream machines” that have the ability to unleash human imagination. He explains that a game is a “possibility space” in which video games start at a well-defined state and end when a specific state is reached. How players reach a specific goal is open-ended, and each player can navigate this possibility space by making continuous choices and actions.
Gee (2005) similarly describes how a well-designed game incorporates good learning principles that can support players’ creativity. First, players are not simply consumers of the game but producers by making their own actions and choices. At a fundamental level, what players do and create in the game to progress through levels is a form of production. For example, in the popular “god” game called Spore, players create their own species, and then the species evolve into more intelligent creatures and civilization. Some games, such as LittleBigPlanet, Portal 2, and Physics Playground have built-in level editor functionality that allows players to modify the games and even create their own levels. Second, good games often encourage players to take risks, explore and try new things, and learn by failing. Failing is not a bad thing in games as it is in traditional education. In fact, failing is a great way to get feedback about progress. Third, video games are “pleasantly frustrating.” That is, tasks in a well-designed game are challenging, but reside within a range of difficulty levels. This gives players a great sense of accomplishment upon completing the task.
Despite this inherent link between creativity and video games, there is limited and rather mixed evidence for relationships between playing video games and creativity. For example, Hamlen (2009) investigated the relationship between self-reported time spent playing video games per week and performance on the Torrance Tests of Creative Thinking (TTCT; Torrance, 1972) in 4th and 5th graders. She reported that the number of hours of gameplay does not significantly predict TTCT performance controlling for gender and grade. In contrast, Jackson and colleagues (Jackson et al., 2012) investigated the relationship between gameplay time (i.e., participants’ response to how often do you play videogames?) and creativity using the TTCT, and they reported that playing video games is significantly associated with creativity.
Although investigating relationships between video gameplay and creativity may be interesting, this line of research does not directly help educators and practitioners to use video games to foster creativity. That is, studies that have investigated the relationship between playing video games and creativity in general (using correlational analyses) are often based on the assumption that creativity is a “general” construct, and do not consider the possible interplay with or dependence on domains. However, Baer and Kaufman (2005), using their Amusement Park Theory (APT) of creativity, suggest that for any creative work to happen, there are some requirements that need to be present. For example, a person working within a particular domain must have at least some basic knowledge about that domain before creative work may emerge. Similarly, Csikszentmihalyi's (1997) theory of creativity includes domain (e.g., math, arts, science, etc.) as one of the components of the creativity model, and creativity occurs at the intersection of person, domain, and field (i.e., experts in the field who can judge the creative work).
Another problem with some existing studies on games and creativity is that they do not clearly state how creativity is defined (see Plucker, Beghetto, & Dow, 2004 for more on this issue). Also, the way in which creativity is assessed in these studies is problematic, where many studies view creativity as unidimensional. Moreover, correlational studies do not systematically examine how specific aspects of creativity manifest in video games. Finally, such general studies (e.g., Hamlen, 2009; Jackson et al., 2012) typically treat all game genres as equal, but that can be misleading as some genres have more potential to enhance creativity than others.
There are, however, some studies reported in the literature (using experimental and quasi-experimental research design) that do use games that are domain relevant and/or have potential for enhancing creativity (e.g., Minecraft, Portal 2). These games allow players to co-create the gaming environment (using the game's create mode or level editor); hence, they permit players to be creative rather than passively receiving game problems to solve. For instance, Fessakis and Lappas (2013) used a physics-related puzzle game called Crayon Physics Deluxe (Kloonigames, 2014) to investigate the effects of this game on students' creativity. Similarly, Moffat, Crombie, and Shabalina (2017), and Inchamnan, Wyeth, and Johnson (2013) used another popular game called Portal 2 (with a high potential to enhance creativity) to investigate the effects of playing this game on participants' creativity, compared to playing two other puzzle games that were had low potential for enhancing creativity (i.e., I-Fluid, and Braid). Blanco-Herrera, Gentile, and Rokkum (2019) used Minecraft (another game with high potential to enhance creativity) to compare its effects on creativity compared to a racing game called NASCAR, and watching a TV show. The findings of all of the aforementioned studies showed positive results indicating that certain games (e.g., Crayon Physics Deluxe, Portal 2, Minecraft) that engage players in solving interesting problem, creating virtual environments in the games, and designing new game levels, can enhance people's creativity compared to other video games (i.e., racing or shooting games)—see (in press) Rahimi & Shute, for a full review on the effects of videos games on creativity.
To support creativity using video games in the broader education community, we need to understand the affordances of video games in relation to the multidimensional aspects of creativity. That is, the first question we should ask is: What are some of the cognitive and noncognitive dimensions of creativity that are part of playing video games? In addition, attention needs to be paid to assessment methods that use creative behaviors and products that players create in and outside of video games (Plucker & Makel, 2010). Such behaviors and products are believed to be more valid indicators of creativity than commonly used self-report measures of creativity (McClelland, 1973; Shute, Ventura, & Kim, 2013).
The purpose of this paper is to threefold: (1) review the current literature of creativity and link the literature with the mechanics of popular games that foster creative endeavors; (2) describe a methodology called stealth assessment as a way to assess creativity in the context of a learning video game called Physics Playground (Shute & Ventura, 2013); and (3) provide empirical support for the construct and criterion validity of our stealth assessment measure of creativity.
Section snippets
Multiple Dimensions of Creativity
There have been countless arguments over the accepted definition of creativity among psychologists across the decades. Despite this lack of agreement, there are some common notions of creativity that run through the literature. First, creativity is generally defined as the ability to produce solutions, ideas, or products that are both novel and effective (Lubart, 1994). Kaufman and Sternberg (2007) similarly have noted that most definitions of creativity consist of three components: novelty,
Participants and research design
The participants of this study consisted of 167 8th and 9th graders (76 male and 91 female; 13-15 years old) from a K-12 school in Florida. Upon the completion of the study, each student received a $25 gift card. We used a one-group, pretest-posttest research design.
Procedure
The total gameplay time was about 4 h (across six 45-min sessions in a week). Thirty computers, in one of the school's two computer labs, were used for this study. Separators were used between the computers to make sure that
Results
To address research question 1 regarding the criterion-related validity of our stealth assessment estimate of creativity (computed using BNs), we conducted several correlational analyses. As expected, our creativity estimate correlated with our external, performance-based measures of creativity as follows: (a) Alternative Uses test—fluency (r = 0.18, p = .02); (b) Alternative Uses test—originality (r = 0.18, p = .02); and (c) Student-made levels (r = 0.23, p = .01). There was no significant
Discussion and future research
As playing video games has become a key part of everyday life for today's youth, the broader education community has been exploring affordances of video games to measure and support competencies that are valuable to success in the 21st Century. In this paper, we discussed how one such game—Physics Playground—can be used as a vehicle to measure creativity. We examined research questions related to the validity of our stealth assessment estimate of creativity, as well as the effects of creativity
Declaration of interest
We wish to confirm that there are no known conflicts of interest associated with this manuscript.
CRediT authorship contribution statement
Valerie J. Shute: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editing. Seyedahmad Rahimi: Formal analysis, Investigation, Methodology, Resources, Software, Validation, Writing - original draft, Writing - review & editing.
Acknowledgments
This work was supported by the US National Science Foundation [award number #037988] and the US Department of Education [award number #039019]. We'd also like to thank team members who helped in this project—Russell Almond, Mathew Ventura, Yoon Jeon Kim, Lubin Wang, and Weinan Zhao.
References (89)
I can name that Bayesian network in two matrixes!
International Journal of Approximate Reasoning
(2010)- et al.
The dynamic componential model of creativity and innovation in organizations: Making progress, making meaning
Research in Organizational Behavior
(2016) - et al.
Learning through playing virtual age: Exploring the interactions among student concept learning, gaming performance, in-game behaviors, and the use of in-game characters
Computers & Education
(2015) - et al.
Four ways five factors are basic
Personality and Individual Differences
(1992) - et al.
Aesthetics and creativity
- et al.
Information technology use and creativity: Findings from the children and technology project
Computers in Human Behavior
(2012) - et al.
Psychometric analysis of the performance data of simulation-based assessment: A systematic review and a bayesian network example
Computers in Education
(2015) - et al.
The effectiveness of adaptive difficulty adjustments on students' motivation and learning in an educational computer game
Computers & Education
(2013) - et al.
Modeling how incoming knowledge, persistence, affective states, and in-game progress influence student learning from an educational game
Computers & Education
(2015) - et al.
Measuring problem solving skills via stealth assessment in an engaging video game
Computers in Human Behavior
(2016)
After six decades of systematic study of creativity: What do teachers need to know about what it is and how it is measured?
Roeper Review
Improving stealth assessment in game-based learning with LSTM-based analytics
An IRT-based parameterization for conditional probability tables
Bayesian networks in educational assessment
The social psychology of creativity: A componential conceptualization
Journal of Personality and Social Psychology
Bridging generality and specificity: The amusement park theoretical (APT) model of creativity
Roeper Review
Towards general models of effective science inquiry in virtual performance assessments
Journal of Computer Assisted Learning
Physics and creativity: Problem solving and learning contexts
Industry and Higher Education
Social-cognitive aspects of identity style: Need for cognition, experiential openness, and introspection
Journal of Adolescent Research
Developing physics learning activities for fostering student creativity in Hong Kong context
Creative potential of school children in Hong Kong: Norms of the Wallach-Kogan Creativity Tests and their implications
Creativity Research Journal
A note on the Wallach‐Kogan tests of creativity
British Journal of Educational Psychology
Originality scores under timed and untimed conditions
Australian Journal of Psychology
Creativity: Flow and the psychology of discovery and invention
Zoo U: A stealth approach to social skills assessment in schools
Advances in Human-Computer Interaction
Linking intrinsic motivation, risk taking, and employee creativity in an R&D environment
R & D Management
Creativity in physics: Response fluency and task specificity
Creativity Research Journal
Evaluating the alternative uses test of creativity
Essential facts about the computer and video game industry
The influence of personality on artistic and scientific creativity
Cultivating preschoolers creativity using guided interaction with problem solving computer games
Learning by design: Good video games as learning machines
Risky shift and creativity
Social Behavior and Personality: International Journal
Relation of four components of creativity to risk-taking preferences
Psychological Reports
From log files to assessment metrics: Measuring students' science inquiry skills using educational data mining
The Journal of the Learning Sciences
The structure of intellect
Psychological Bulletin
Relationships between computer and video game play and creativity among upper elementary school students
Journal of Educational Computing Research
SpinCycle project [Weblog]
Does activity in computer game play have an impact on creative behaviour? 2013
The person, the product, and the response: Conceptual problems in the assessment of creativity
Journal of Personality
Guiding learners through technology-based instruction: The effects of adaptive guidance design and individual differences on learning over time
Journal of Educational Psychology
Do people recognize the four Cs? Examining layperson conceptions of creativity
Psychology of Aesthetics, Creativity, and the Arts
Resource review: Creativity
Change
Assessing game-based mathematics learning in action
Cited by (53)
Learning experience assessment through players chat content in multiplayer online games
2024, Computers in Human BehaviorAssociations between Computational Thinking and Figural, Verbal Creativity
2023, Thinking Skills and CreativityToward Asset-based Instruction and Assessment in Artificial Intelligence in Education
2024, International Journal of Artificial Intelligence in EducationAssessment Framework for Immersive Learning: Application and Evaluation
2024, Communications in Computer and Information Science