Abstract
Validity and reliability are important psychometric properties for specialists who try to understand human behavior with measurement tools. There are many factors that affect psychometric properties, and one of them is aberrant response behavior. Person-fit statistics are one way to detect aberrant individuals. The general purpose of the study was to specify the effects of possible aberrant individuals on fit indices in the Confirmatory Factor Analysis and their effects on estimated parameters by using the Mokken Homogeneity Model and the Graded Response Model. Satisfaction with Life Scale (SWLS) and the Positive and Negative Affect Scale (PANAS) were used in accordance with the purpose of this research. Flagged individuals were determined according to parametric and non-parametric person-fit statistics for both SWLS and PANAS. These individuals were removed from the data sets, and the trimmed data sets were created. Trimmed and untrimmed data sets were compared. It is seen that individuals who were flagged as aberrant generally affected the statistical results. Especially from the data sets trimmed by the parametric person-fit statistic more discriminative items were obtained. According to the research results, searching for aberrant responses can be suggested in scale development and scale adaptation studies.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
The data that support the findings of this study are available from LISS (Longitudinal Internet Studies for the Social Sciences) panel administered by CentERdata (Tilburg University, The Netherlands) but restrictions apply to the availability of these data, which were used under license for the current study and so they are not publicly available. Data are however available from the author(s) upon reasonable request and with permission of LISS.
References
Chalmers, P. (2012). Mirt: A multidimensional item response theory package for the r environment. Journal of Statistical Software, 48(6), 1–29. https://doi.org/10.18637/jss.v048.i06.
Conijn, J. M., Emons, W. H., & Sijtsma, K. (2014). Statistic lz-based person-fit methods for noncognitive multiscale measures. Applied Psychological Measurement, 38(2), 122–136. https://doi.org/10.1177/0146621613497568.
Conijn, J. M., Emons, W. H., De Jong, K., & Sijtsma, K. (2015). Detecting and explaining aberrant responding to the outcome Questionnaire-45. Assessment, 22(4), 513–524. https://doi.org/10.1177/1073191114560882.
Crișan, D. R., Tendeiro, J., & Meijer, R. (2019, September 24). The crit value as an effect size measure for violations of model assumptions in Mokken scale analysis for binary data. https://doi.org/10.31234/osf.io/8ydmr.
Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Holt: Rinehart and Winston.
Curtis, D. D. (2004). Person misfit in attitude surveys: Influences, impacts and implications. International Education Journal, 5(2), 125–143 https://files.eric.ed.gov/fulltext/EJ903843.pdf.
De Ayala, R. J. (2009). The theory and practice of item response theory. New York: Guilford Publications.
De Koning, E., Sijtsma, K., & Hamers, J. H. M. (2002). Comparison of four irt models when analyzing two tests for inductive reasoning. Applied Psychological Measurement, 26(3), 302–320. https://doi.org/10.1177/0146621602026003005.
Diener, E. D., Emmons, R. A., Larsen, R. J., & Griffin, S. (1985). The satisfaction with life scale. Journal of Personality Assessment, 49(1), 71–75. https://doi.org/10.1207/s15327752jpa4901_13.
DiStefano, C., & Hess, B. (2005). Using confirmatory factor analysis for construct validation: An empirical review. Journal of Psychoeducational Assessment, 23(3), 225–241. https://doi.org/10.1177/073428290502300303.
Egberink, I. J. L. (2010). Applications of item response theory to non-cognitive data. [doctoral thesis]. Groningen, The Netherlands: University of Groningen.
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. New Jersey, NJ: Lawrence Erlbaum Associates.
Emons, W. H. M. (2008). Nonparametric person-fit analysis of polytomous item scores. Applied Psychological Measurement, 32(3), 224–247. https://doi.org/10.1177/0146621607302479.
Ferrando, P. J. (2012). Assessing inconsistent responding in e and n measures: An application of person-fit analysis in personality. Personality and Individual Differences, 52(6), 718–722. https://doi.org/10.1016/j.paid.2011.12.036.
Ferrando, P. J., & Chico, E. (2001). Detecting dissimulation in personality test scores: A comparison between person-fit indices and detection scales. Educational and Psychological Measurement, 61(6), 997–1012. https://doi.org/10.1177/00131640121971617.
Floyd, F. J., & Widaman, K. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7(3), 286–299. https://doi.org/10.1037/1040-3590.7.3.286.
Glas, C. A. W., & Khalid, N. (2018). Person fit. In W. J. van der Linden (Ed.), Handbook of item response theory volume three applications (pp. 107–126). Boca Raton, FL: CRC Press/Taylor & Francis Group.
Glass, C. A. W., & Dagohoy, A. V. T. (2007). A person-fit test for irt models for polytomous items. Psychometrika, 72(2), 159–180. https://doi.org/10.1007/s11336-003-1081-5.
Guo, H., Rios, J. A., Haberman, S., Liu, O. L., Wang, J., & Paek, I. (2016). A new procedure for detection of students’ rapid guessing responses using response time. Applied Measurement in Education, 29(3), 173–183. https://doi.org/10.1080/08957347.2016.1171766.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park, CA: Sage publications.
Hambleton, R. K., van der Linden, W. J., & Wells, C. S. (2011). IRT models for the analysis of polytomous scored data: Brief and selected history of model building advances. In M. L. Nering & R. Ostini (Eds.), Handbook of polytomous item response theory models (pp. 21–42). New York: Routledge.
Hemker, B. T., Sijtsma, K., & Molenaar, I. W. (1995). Selection of unidimensional scales from a multidimensional item bank in the polytomous Mokken IRT model. Applied Psychological Measurement, 19(4), 337–352. https://doi.org/10.1177/014662169501900404.
Hopfenbeck, T. N., & Maul, A. (2011). Examining evidence for the validity of PISA learning strategy scales based on student response processes. International Journal of Testing, 11(2), 95–121. https://doi.org/10.1080/15305058.2010.529977.
Karabatsos, G. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16(4), 277–298. https://doi.org/10.1207/S15324818AME1604_2.
Kline, R. B. (2011). Principles and practice of structural equation modeling. New York: Guilford publications.
Liu, T., Sun, Y., Li, Z., & Xin, T. (2019). The impact of aberrant response on reliability and validity. Measurement: Interdisciplinary Research and Perspectives, 17(3), 133–142. https://doi.org/10.1080/15366367.2019.1584848.
Meijer, R. R. (1996). Person-fit research: An introduction. Applied Measurement in Education, 9(1), 3–8. https://doi.org/10.1207/s15324818ame0901_2.
Meijer, R. R. (1997). Person fit and criterion-related validity: An extension of the Schmitt, Cortina, and Whitney study. Applied Psychological Measurement, 21(2), 99–113. https://doi.org/10.1177/01466216970212001.
Meijer, R. R. (2003). Diagnosing item score patterns on a test using item response theory-based person-fit statistics. Psychological Methods, 8(1), 72–87. https://doi.org/10.1037/1082-989X.8.1.72.
Meijer, R. R., & Nering, M. L. (1997). Trait level estimation for nonfitting response vectors. Applied Psychological Measurement, 21(4), 321–336. https://doi.org/10.1177/01466216970214003.
Meijer, R. R., & Tendeiro, J. N. (2012). The use of the lz and lz* person-fit statistics and problems derived from model misspecification. Journal of Educational and Behavioral Statistics, 37(6), 758–766. https://doi.org/10.3102/1076998612466144.
Meijer, R. R., Egberink, I. J., Emons, W. H., & Sijtsma, K. (2008). Detection and validation of unscalable item score patterns using item response theory: An illustration with Harter's self-perception profile for children. Journal of Personality Assessment, 90(3), 227–238. https://doi.org/10.1080/00223890701884921.
Mertler, C. A., & Reinhart, R. V. (2017). Advanced and multivariate statistical methods: practical application and interpretation. New York: Taylor & Francis Group.
Messick, S. (1994). Alternative modes of assessment, uniform standards of validity 1. (ETS research report series). https://doi.org/10.1002/j.2333-8504.1994.tb01634.x.
Mokken, R. J. (1971). A theory and procedure of scale analysis: With applications in political research. The Hague: Mouton.
Mokken, R. J., Lewis, C., & Sijtsma, K. (1986). Rejoinder to “the Mokken scale: A critical discussion”. Applied Psychological Measurement, 10(3), 279–285. https://doi.org/10.1177/014662168601000306.
Reise, S. P., & Revicki, D. A. (2015). Introduction: Age-old problems and modern solutions. In S. P. Reise, & D. A. Revicki (Eds.), Handbook of item response theory modeling: Applications to typical performance assessment (pp. 3–12). New York: Taylor & Francis Group.
Reise, S. P., & Waller, N. G. (1993). Traitedness and the assessment of response pattern scalability. Journal of Personality and Social Psychology, 65(1), 143–151. https://doi.org/10.1037/0022-3514.65.1.143.
Rosseel, Y. (2012). Lavaan: An r package for structural equation modeling. Journal of Statistical Software, 48(2), 1–26. https://doi.org/10.18637/jss.v048.i02.
Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003). Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures. Methods of Psychological Research Online, 8(2), 23–74.
Şengül-Avşar, A. (2019). Comparison of person-fit statistics for polytomous items in different test conditions. Journal of Measurement and Evaluation in Education and Psychology, 10(4), 348–364. https://doi.org/10.21031/epod.525647.
Seo, D. G., & Weiss, D. J. (2013). lz person-fit index to identify misfit students with achievement test data. Educational and Psychological Measurement, 73(6), 994–1016. https://doi.org/10.1177/0013164413497015.
Sijtsma, K. (1998). Methodology review: Nonparametric irt approaches to the analysis of dichotomous item scores. Applied Psychological Measurement, 22(1), 3–31. https://doi.org/10.1177/01466216980221001.
Sijtsma, K., & Molenaar, I. W. (2002). Introduction to nonparametric item response theory. Thousand Oaks, CA: Sage Publications.
Stone, C. A., & Zhu, X. (2015). Bayesian analysis of item response theory models using SAS. Cary, NC: SAS Institute Inc..
Syu, J. J. (2013). Applying person fit-in faking detection-the simulation and practice of nonparametric item response theory. [doctoral dissertation, National Chengchi University]. http://nccur.lib.nccu.edu.tw/bitstream/140.119/58646/1/251501.pdf
Tendeiro, J. N., & Meijer, R. R. (2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics. Journal of Educational Measurement, 51(3), 239–259. https://doi.org/10.1111/jedm.12046.
Tendeiro, J. N., Meijer, R. R., & Niessen, M. S. A. (2016). PerFit: An r package for person-fit analysis in irt. Journal of Statistical Software, 74(5), 1–27. https://doi.org/10.18637/jss.v074.i05.
Toland, M. D. (2014). Practical guide to conducting an item response theory analysis. The Journal of Early Adolescence, 34(1), 120–151. https://doi.org/10.1177/0272431613511332.
Van der Ark, L. A. (2012). New developments in Mokken scale analysis in r. Journal of Statistical Software, 48(5), 1–26. https://doi.org/10.18637/jss.v048.i05.
VandenBos, G. R. (2015). APA dictionary of psychology (2nd ed.). Washington DC: American Psychiatric Pub. https://doi.org/10.1037/14646-000.
Walker, A. A., & Wind, S. A. (2020). Identifying misfitting achievement estimates in performance assessments: An illustration using Rasch and Mokken scale analyses. International Journal of Testing, 20(3), 231–251. https://doi.org/10.1080/15305058.2019.1673758.
Watson, D., Clark, L. A., & Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: The PANAS scales. Journal of Personality and Social Psychology, 54(6), 1063–1070. https://doi.org/10.1037/0022-3514.54.6.1063.
Wise, S. L. (2015). Effort analysis: Individual score validation of achievement test data. Applied Measurement in Education, 28(3), 237–252. https://doi.org/10.1080/08957347.2015.1042155.
Zanon, C., Hutz, C. S., Yoo, H. H., & Hambleton, R. K. (2016). An application of item response theory to psychological test development. Psicologia: Reflexão e Crítica, 29(1), 1–10. https://doi.org/10.1186/s41155-016-0040-x.
Acknowledgments
In this paper we make use of data of the LISS (Longitudinal Internet Studies for the Social Sciences) panel administered by CentERdata (Tilburg University, The Netherlands). The author sincerely thanks the LISS panel for sharing data sets.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Ethical Approval
Data that were used in this study are from a publicly available database. As such, we have depended on the original researchers to comply with the 1964 Declaration of Helsinki and its later addenda and to obtain informed consent from their subjects. We had no interaction with human subjects in the current project.
Informed Consent
Informed consent is not required in this study.
Conflict of Interest
The author declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
ESM 1
(DOCX 51 kb)
Rights and permissions
About this article
Cite this article
Şengül Avşar, A. Aberrant individuals’ effects on fit indices both of confirmatory factor analysis and polytomous IRT models. Curr Psychol 42, 2157–2166 (2023). https://doi.org/10.1007/s12144-021-01563-4
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12144-021-01563-4