Advancing our understanding and assessment of cognitive effort in the cognitive fit theory and data visualization context: Eye tracking-based approach

doi:10.1016/j.dss.2022.113862

Decision Support Systems

Volume 163, December 2022, 113862

https://doi.org/10.1016/j.dss.2022.113862 Get rights and content

Highlights

•
We highlight the need in CFT research to focus on advancing our more direct assessment and measurement of cognitive effort
•
Eye fixation-based metrics can be useful in advancing our understanding of cognitive fit and its cognitive effort mechanism
•
In all tasks except the complex-symbolic task, users experience change in cognitive effort based on cognitive fit
•
Physiological indicators can provide insights that may be missed if using only perceptual measures of cognitive effort

Abstract

In Cognitive Fit Theory (CFT) based research, there is a consensus about cognitive effort as the underlying mechanism impacting performance. Although critical to the theory, cognitive effort and its direct empirical assessment remain a challenge. In this repeated measures experimental study, we introduce a research model and develop hypotheses based on the fundamental relationships underlying CFT while integrating eye tracking as an approach for assessing cognitive effort. Our study finds that eye tracking technology, specifically fixation-based metrics, can be used in the understanding of cognitive processes initiated by our data representation choices. Specifically, we find that in all tasks except the complex-symbolic task, users experience meaningful change in the physiological assessment of cognitive effort based on the condition of cognitive fit. We contrast our findings to existing research and find that physiological indicators of cognitive effort can provide critical insights often missed in traditional CFT research.

Introduction

Cognitive Fit Theory (CFT) [1,2], even after over 30 years, continues to significantly impact our understanding of how visual data representations, through their matching with problem tasks and users' mental models, impact decision performance. In the extant CFT research, there is a consensus on the critical role of cognitive effort (sometimes called cognitive workload, burden, strain, or load), yet the research largely fails to integrate cognitive effort measurement [3]. Instead, empirical studies typically only assume cognitive effort's theoretical existence as a function of cognitive fit and as the mechanism that drives decision speed and accuracy. Without a greater understanding of the theorized mechanism, the research could be exposed to criticism that challenges the fundamental underpinnings of substantial portions of CFT literature [4]. One possible reason for the existing state of the research may be found in the difficulty of measuring cognitive effort. Therefore, there is a clear need to enhance current research methods and theory by offering and testing ways cognitive effort could be measured and integrated into CFT and the broader data visualization literature.

To address this opportunity, in this research effort, we turn our attention to a physiological indicator of cognitive effort by deploying eye tracking technology through its ability to collect biometric data that describes users' visual attention. The scarcity of physiological response research in CFT is surprising, given that research findings suggest that evaluating visual interface performance without measuring cognitive processes, such as workload, may lead to incorrect conclusions about the efficiency of an interface. Driven partly by the ‘eye-mind hypothesis’ [5], eye tracking is considered effective in assessing cognitive processes as it reveals how the user reads and scans the displayed information by capturing users' eye fixation-derived metrics [6,7]. These metrics have been successfully linked to cognitive processes indicating various forms of attention, interest, mental effort, and cognitive load in psychology [8], neuroscience [9], art, human factors, marketing, and computer science [10]. In recent years we have also seen the adoption of eye tracking technology in IS research [[11], [12], [13], [14], [15]]. Given the mature and validated use of eye-tracking-based metrics across fields, the application of eye tracking to evaluate cognitive effort in the CFT context is timely and appropriate.

The remainder of the paper is organized as follows. We first highlight the relevant background associated with CFT and the role of cognitive effort and introduce eye tracking as an approach for measuring cognitive effort. We then present a research model and develop hypotheses based on the fundamental relationships underlying CFT. The following sections present our experimental design and data analysis. We then discuss our findings and contextualize these results by discussing implications, limitations, and potential next steps.

Section snippets

Cognitive fit theory

A series of studies, starting with the Minnesota experiments [16], compared decision performance in various tasks by offering subjects information in tabular and graphical formats. These early studies yielded inconclusive results, motivating the introduction of CFT [1,2] to interpret those results and provide a theoretical basis to predict decision performance. CFT introduced the idea of cognitive fit or a match between the problem representation and the problem-solving task that occurs when

Hypothesis development

According to CFT, if an external problem representation does not match that emphasized in the task, there is nothing to guide the decision-maker in working toward a task solution. As a result, a greater cognitive effort is required to transform the information into a form suitable for solving that particular type of problem [1,2]. The lens of describing cognitive effort as the emergent property of cognitive fit and as the underlying mechanism that influences decision performance dominates the

Experimental design

The experiment was a fully randomized within-subject three-factor design: Task Complexity (simple and complex), Task Type (spatial and symbolic), and Representation (table and graph). This resulted in 8 cell, 2 by 2 by 2 factorial design. All participants performed all experimental tasks associated with eight cells in random order. To avoid the potential bias of using the same answer from the same task and different representation influencing the answer in another representation, a slightly

Cognitive effort - average fixation duration (CE_FD)

Repeated measures Analysis of Variance (Table 1) showed a main effect of Task Complexity (F(1,31) = 8.352, p = 0.007, η_p² = 212) and Task Type (F(1,31) = 4.282, p = 0.047, η_p² = 0.121). The Analysis of Variance also showed interaction effect of Task Complexity * Task Type (F(1,31) = 51.939, p = 0.000, η_p² = 0.626), Task Complexity * Representation (F(1,31) = 7.522, p = 0.01, η_p² = 0.195), Task Type * Representation (F(1,31) = 10.58, p = 0.003, η_p² = 254), and Task Complexity * Task Type *

Discussion

Our study finds that, in most tested tasks (all except complex-symbolic task), users demonstrate meaningful changes in the cognitive effort, measured using eye-tracking fixation metrics based on the condition of cognitive fit. We detected the impact beyond the individual effects of task complexity, task type, or representation.

Implications, limitations and next steps

There are important implications, questions, and next steps emerging from this research. We identified a consensus around the role of cognitive effort in the context of the visual representation of data. Cognitive effort is the underlying mechanism and one of the key ingredients that makes user response prediction and visual representation recommendation choices possible. Yet IS literature that depends on a deeper understanding of that cognitive effort is scarce, mostly theoretical, and lacking

Conclusion

In this study, we identify and elevate the need for CFT research to focus on advancing our understanding and assessment of cognitive effort. Recognizing the difficulty in measuring cognitive effort, we offer the potential of eye-tracking technology to assess cognitive effort physiologically in the CFT context. Several important implications for phenomenon measurement, theory and empirical research, and practice emerged from our study: (i) fixation-based metrics can be useful in advancing our

Author credit statement

We declare that this manuscript has not been published previously and that it is not under consideration for publication elsewhere. In addition, we assure that both authors (listed on the Title Page document and in that particular order) have fully participated in the research and the article preparation and both of us have approved the submission.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

References (60)

D. Bačić et al.
Business information visualization intellectual contributions: an integrative framework of visualization capabilities and dimensions of visual intelligence
Decis. Support. Syst.
(2016)
M.A. Just et al.
Eye fixations and cognitive processes
Cogn. Psychol.
(1976)
M.K. Eckstein et al.
Beyond eye gaze: what else can eyetracking reveal about cognition and cognitive development?
Dev. Cogn. Neurosci.
(2017)
Q. Wang et al.
An eye-tracking study of website complexity from cognitive load perspective
Decis. Support. Syst.
(2014)
M. Shojaeizadeh et al.
Detecting task demand via an eye tracking machine learning system
Decis. Support. Syst.
(2019)
J.S. Giboney et al.
User acceptance of knowledge-based system recommendations: explanations, arguments, and fit
Decis. Support. Syst.
(2015)
X. Liu et al.
Novice digital service designers’ decision-making with decision aids — a comparison of taxonomy and tags
Decis. Support. Syst.
(2020)
C. Speier
The influence of information presentation formats on complex task decision-making performance
Int. J. Hum. Comput. Stud.
(2006)
A. Chandra et al.
Representational congruence and information retrieval: towards an extended model of cognitive fit
Decis. Support. Syst.
(1999)
F.-Y. Kuo et al.
An exploratory study of cognitive effort involved in decision under framing—an application of the eye-tracking technology
Decis. Support. Syst.
(2009)

J. Sweller

Cognitive load during problem solving: effects on learning

Cogn. Sci.

(1988)

G.S. Hubona et al.

Mental representations of spatial language

Int. J. Hum. Comput. Stud.

(1998)

R.E. Wood

Task complexity: definition of the construct

Organ. Behav. Hum. Decis. Process.

(1986)

I. Vessey

The effect of information presentation on decision making: a cost-benefit analysis

Inf. Manag.

(1994)

L. Paquette et al.

The effect of decision strategy and task complexity on decision performance

Organ. Behav. Hum. Decis. Process.

(1988)

I. Vessey

Cognitive fit: a theory-based analysis of the graphs versus tables literature

Decis. Sci.

(1991)

I. Vessey et al.

Cognitive fit: an empirical study of information acquisition

Inf. Syst. Res.

(1991)

D. Bačić et al.

Task-representation fit’s impact on cognitive effort in the context of decision timeliness and accuracy: a cognitive fit perspective

AIS Trans. Human-Computer Interact.

(2018)

K. Rayner

Eye movements in reading and information processing: 20 years of research

Psychol. Bull.

(1998)

D. Bačić

Biometrics and business information visualization: Research review, agenda and opportunities

P. Ayres et al.

Cognitive load theory: new directions and challenges

Appl. Cogn. Psychol.

(2012)

A.T. Duchowski

Eye Tracking Methodology: Theory and Practice

(2017)

P.-M. Léger et al.

Precision is in the eye of the beholder: application of eye fixation-related potentials to information systems research

J. Assoc. Inf. Syst.

(2014)

P. Bera et al.

Using eye tracking to expose cognitive processes in understanding conceptual models

MIS Q.

(2019)

A. Vance et al.

Tuning out security warnings: a longitudinal examination of habituation through fMRI, eye tracking, and field experiments

MIS Q.

(2018)

G.W. Dickson et al.

Research in management information systems: the Minnesota experiments

Manag. Sci.

(1977)

V. Khatri et al.

Understanding conceptual schemas: exploring the role of application and IS domain knowledge

Inf. Syst. Res.

(2006)

S. Goswami et al.

The role of visualization tools in spreadsheet error correction from a cognitive fit perspective

J. Assoc. Inf. Syst.

(2008)

Z. Bin et al.

Visualization of network concepts: the impact of working memory capacity differences

Inf. Syst. Res.

(2010)

B.E. Mennecke et al.

Is a map more than a picture? The role of SDSS technology, subject characteristics, and problem complexity on map reading and problem solving

MIS Q.

(2000)

Cited by (10)

A machine learning study for predicting driver goals in contingencies with leading and lagging features during goal determination
2024, Expert Systems with Applications
Many studies have focused on decision support systems that enhance both the efficiency and safety of driving. They have also explored the potential of real-time psychological data and machine learning in predicting drivers’ cognitive state, such as their fatigue levels, drowsiness, or workload. However, few studies have investigated prediction of driving goals as a cognitive outcome. Early prediction plays an essential role in providing active decision support during driving events under time pressure conditions. In this study, machine learning algorithms and features associated with different phases of decision-making were used to predict two common driving goals: defensive driving in emerging scenarios and urgent reactions in nonroutine scenarios. The effects of perception-, reflex-, control-, and kinetic-related features and how they contribute to prediction in the context of decision-making were analyzed. A total of 49 individuals were recruited to complete simulated driving tasks, with 237 events of defensive driving and 271 events of urgent reactions identified. The results revealed premium recall with a naïve Bayes classifier, indicating the onset of decision-making, with extreme gradient boosting and random forests exhibiting superior precision in predicting defensive driving and urgent reactions, respectively. Additionally, the cutoff of the initial 0.4 s of the events was identified. Before the cutoff, the leading features were reflex- and control-related features, which were the drivers’ immediate reactions before scenario evaluation and goal determination. These leading features contributed to superior prediction results for the two types of driving goals, indicating the likelihood of early detection. After the cutoff, model performance decreased, and lagging features came into play. These lagging features comprised perception- and kinetic-related features, reflecting observation of cues and outcomes of inputs delivered to vehicles. In the first 2 s, predictive models recovered and stabilized.
Selling myself: Anthropomorphic products in electronic commerce
2024, Decision Support Systems
What if a product was able to sell itself? Suppose a product could display itself in the form of an animated caricature that walked, talked, and described itself to potential buyers. Such a display may trigger anthropomorphism, the automatic process of attributing human characteristics to a non-human object, but past research suggests that the nature of the product may influence this process. We argue that displaying a product as an animated caricature is more likely to trigger anthropomorphism when the underlying product itself has some human-like features than when the product does not (e.g., appearance, movement, speech, behavior). We tested this form of presentation with a laptop, a camera, and a TV, theorizing that the effects would be stronger for the laptop than either the TV or the camera because the computer has more human-like features. Our results show that participants bid 20% more for a computer when it was presented in the form of an animated caricature compared to when it was presented on a traditional Web page, and as theorized, there were no differences for the TV or camera when presented in this form. Electroencephalogram (EEG) results show that participants engaged in similar cognitive activities for the computer displayed on Web pages and as an animated caricature. However, there was a significant difference in results between the two display formats for the camera and TV, which we interpret as the cognitive conflict that occurs when the animated product display does not fit the underlying nature of the product.
All eyes on me: Predicting consumer intentions on social commerce platforms using eye-tracking data and ensemble learning
2023, Decision Support Systems
Understanding what information is important for consumers when making a purchase-related decision has been a key question for researchers and practitioners ever since the advent of empirical research in commerce. Nevertheless, our knowledge of what information is important has been formed primarily through post-purchase conscious capturing approaches, such as surveys and questionnaires. To overcome these limitations, we ground this research on an exploratory study that captures eye-tracking data during a decision-making task of product selection. Grounded on the dynamic attention theory, we utilize different information types and formats present on a popular social commerce platform, to identify elements which are important when deciding about online product purchase decision. Specifically, we employ a series of prediction algorithms and use an ensemble learning setup to predict the aspects that contribute to product selection by consumers. Our analysis highlights the most important informational cues to accurately predict product selection among alternatives. In addition, the results showcase how such elements shift in importance during the temporal sequence of comparing different product alternatives. Our results provide insight into how we can understand the journey of decision-making for social commerce customers when navigating through information to select a product. In addition, it opens the discussion about the shifts that eye-tracking in combination with machine learning can create for researchers and marketers.
Do atmospheric cues matter in live streaming e-commerce? An eye-tracking investigation
2023, Electronic Commerce Research and Applications
This study developed a stimulus-organism-response (SOR) framework based on the cognitive load theory to investigate the influencing mechanisms of atmospheric cues, such as background visual complexity (BVC) and music tempo (MT), in customers’ live-streaming shopping. To test these hypotheses, a laboratory experiment was conducted using eye tracking. The experiment had a 3 (BVC: high vs. medium vs. low) × 3 (MT: fast vs. slow vs. no) within-subject factor design. The outcomes indicate that both BVC and MT significantly affect the participants’ arousal states, which in turn influences their behavioral intentions. Specifically, high arousal was triggered among participants when they were viewing live streams with complex background images and fast-tempo background music. The results also indicate that high BVC induces less intrinsic cognitive load, which is positively correlated with purchase intention and continuous watching intention.
Incorporating evolutionary adaptions into the cognitive fit model for data visualization
2023, Decision Support Systems
With the proliferation of data has arisen an array of information visualization tools to help humans convert data to information for better decision-making. This ever-expanding visualization toolset has outrun the development of theories of data visualization usage for decision support. In the current research, we contribute to the development of visualization theories in two ways. First, we incorporate our understanding of evolutionary adaptations into the cognitive fit theory to explain why 3-D surface graphs often outperform traditional 2-D graphs in various sensemaking tasks. Drawing on geon theory to interpret the 3-D advantage, we develop the concept of geonic/non-geonic information representations and tasks. We tested and found support for our hypotheses that geonic representations have a speed advantage for the tasks we studied, derived from the human evolved capacity for geographic orienteering, but accuracy depends on the match of representation and task, as geonic or non-geonic. Our experiment compared the geonic representation (3-D surface graph) to non-geonic planar representations (cluster bar, heat map, and line) in geonic (integration, proportion) and non-geonic atomic (acquisition, comparison) tasks. For our second contribution, we build on this result to expand the cognitive fit model to incorporate classic human information processes and evolutionary adaptations. We suggest that the cognitive fit model can expand to include new advances in neuroscience. We discuss the theoretical and practical implications of our experiment and our expanded model.
Create the best first glance: The cross-cultural effect of image background on purchase intention
2023, Decision Support Systems
As globalization drives more firms toward cross-border e-commerce (CBEC), a well-designed decision support system becomes crucial to gain a competitive edge in the international market. Product images, a vital aspect of the system interface, play a significant role in shaping users' first impressions, facilitating seller-buyer information interaction, and ultimately enhancing users' decisions making in the system. Across a series of studies, this research investigates the effect of cultural differences (thinking style: holistic vs. analytic) on image background and reveals the underlying mechanism. Results show that online consumers from cultures characterized by holistic thinking style (Chinese sample) are more prone to purchase products presented with contextual backgrounds than those with white backgrounds, while this effect is absent for online consumers from cultures that tend to think in an analytic way (American sample). This effect is also observed when the thinking style is primed within the culture in separate samples from the United States and China. Study 3 employs eye-tracking technology and shows that holistic thinking, compared to analytic thinking, results in an asymmetry in cognitive effort to purchase the same products framed with contextual and white background images. Specifically, contextual (vs. white) background information greatly assists holistic thinking consumers in understanding the product, enabling them to spend less cognitive effort on product information processing. Instead, the cognitive effort that analytic thinking consumers spare in the product information is not affected by the background. Finally, we discuss theoretical contributions and practical insights for CBEC retailers and system designers that the findings indicate.

View all citing articles on Scopus

Dinko Bačić is an Assistant Professor of Information Systems and the founder of the UX & Biometrics lab in the Quinlan School of Business at Loyola University Chicago. He holds a DBA degree in Information Systems from the Cleveland State University. His research interests include information visualization, human-computer interaction, biometrics, cognition, neuroIS, business intelligence & analytics, and pedagogy. He has papers published in premier journals such as Decision Support Systems, Communications of the Association for Information Systems, AIS Transactions on Human-Computer Interaction, Springer Computer Science Lecture Notes, and Leonardo, among others. He has over fifteen years of corporate and consulting experience in business intelligence, finance, project management, and human resources.

Raymond M. Henry is Professor of Information Systems and Associate Dean in the Monte Ahuja College of Business at Cleveland State University. He received his PhD in Information Systems from the University of Pittsburgh. His research explores topics related to information systems, human-computer interaction, knowledge management and supply chain management. His work has been published in premier journals including Information Systems Research, Journal of Management Information Systems, Journal of the Association of Information Systems, and Journal of Operations Management, among others.

View full text

Advancing our understanding and assessment of cognitive effort in the cognitive fit theory and data visualization context: Eye tracking-based approach

Highlights

Abstract

Introduction

Section snippets

Cognitive fit theory

Hypothesis development

Experimental design

Cognitive effort - average fixation duration (CEFD)

Discussion

Implications, limitations and next steps

Conclusion

Author credit statement

Declaration of Competing Interest

Decis. Support. Syst.

Cogn. Psychol.

Dev. Cogn. Neurosci.

Decis. Support. Syst.

Decis. Support. Syst.

Decis. Support. Syst.

Decis. Support. Syst.

Int. J. Hum. Comput. Stud.

Decis. Support. Syst.

Decis. Support. Syst.

Cogn. Sci.

Int. J. Hum. Comput. Stud.

Organ. Behav. Hum. Decis. Process.

Inf. Manag.

Organ. Behav. Hum. Decis. Process.

Cognitive fit: a theory-based analysis of the graphs versus tables literature

Decis. Sci.

Cognitive fit: an empirical study of information acquisition

Inf. Syst. Res.

Task-representation fit’s impact on cognitive effort in the context of decision timeliness and accuracy: a cognitive fit perspective

AIS Trans. Human-Computer Interact.

Eye movements in reading and information processing: 20 years of research

Psychol. Bull.

Biometrics and business information visualization: Research review, agenda and opportunities

Cognitive load theory: new directions and challenges

Appl. Cogn. Psychol.

Eye Tracking Methodology: Theory and Practice

Precision is in the eye of the beholder: application of eye fixation-related potentials to information systems research

J. Assoc. Inf. Syst.

Using eye tracking to expose cognitive processes in understanding conceptual models

MIS Q.

Tuning out security warnings: a longitudinal examination of habituation through fMRI, eye tracking, and field experiments

MIS Q.

Research in management information systems: the Minnesota experiments

Manag. Sci.

Understanding conceptual schemas: exploring the role of application and IS domain knowledge

Inf. Syst. Res.

The role of visualization tools in spreadsheet error correction from a cognitive fit perspective

J. Assoc. Inf. Syst.

Visualization of network concepts: the impact of working memory capacity differences

Inf. Syst. Res.

Is a map more than a picture? The role of SDSS technology, subject characteristics, and problem complexity on map reading and problem solving

MIS Q.

Cognitive effort - average fixation duration (CE_FD)