Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Article

Using Response Times as Collateral Information About Latent Traits in Psychological Tests

Published Online:December 16, 2019https://doi.org/10.1027/1614-2241/a000181

Abstract

Abstract. In psychological tests, the time needed to respond to the items provides collateral information about the latent traits of the test takers. This, however, requires a measurement model that incorporates the response times in addition to the responses. Such a measurement model is usually based on a full specification of the response time distribution. In the present article, we suggest a novel modeling approach that requires fewer assumptions. In the approach, the responses are modeled with a unidimensional two-parameter logistic model. The single response times are summed to the scale-specific total testing time which is then related to the latent trait of the two-parameter logistic model via a smooth adaptive Gaussian mixture (SAGM) model. The approach can be justified against the background of the bivariate generalized linear item response theory modeling framework (Molenaar, Tuerlinckx, & van der Maas, 2015a). Its utility is investigated in two simulation studies and an empirical example.

References

  • Bartholomew, D., & Tzamourani, P. (1999). The goodness of fit of latent trait models in attitude measurement. Sociological Methods & Research, 27, 524–546. https://doi.org/10.1177/0049124199027004003 First citation in articleCrossref, Google Scholar

  • Bolsinova, M., & Maris, G. (2016). A test for conditional independence between response time and accuracy. British Journal of Mathematical and Statistical Psychology, 69, 62–79. https://doi.org/10.1111/bmsp.12059 First citation in articleCrossref, Google Scholar

  • Bolsinova, M., & Tijmstra, J. (2018). Improving precision of ability estimation: Getting more from response times. British Journal of Mathematical and Statistical Psychology, 71, 13–38. https://doi.org/10.1111/bmsp.12104 First citation in articleCrossref, Google Scholar

  • Bunji, K., & Okada, K. (2019). Item response and response time model for personality assessment via linear ballistic accumulation. Japanese Journal of Statistics and Data Science, 2, 263–297. https://doi.org/10.1007/s42081-019-00040-4 First citation in articleCrossref, Google Scholar

  • Chan, S.-C., Lu, T.-S., & Tsai, R.-C. (2014). Incorporating response time to analyze test data with mixture structural equation modeling. Psychological Testing, 61, 463–488. First citation in articleGoogle Scholar

  • Cheng, Y., & Ke-Hai, Y. (2010). The impact of fallible item parameter estimates on latent trait recovery. Psychometrika, 75, 280–291. https://doi.org/10.1007/s11336-009-9144-x First citation in articleCrossref, Google Scholar

  • Dunn, T., Lushene, R., & O’Neil, H. (1972). Complete automation of the MMPI and a study of its response latencies. Journal of Consulting and Clinical Psychology, 39, 381–387. https://doi.org/10.1037/h0033855 First citation in articleCrossref, Google Scholar

  • Fan, Z., Wang, C., Chang, H., & Douglas, J. (2012). Utilizing response time distributions for item selection in CAT. Journal of Educational and Behavioral Statistics, 37, 655–670. https://doi.org/10.3102/1076998611422912 First citation in articleCrossref, Google Scholar

  • Ferrando, P. (2006). Person-item distance and response time: An empirical study in personality measurement. Psicológica, 27, 137–148. First citation in articleGoogle Scholar

  • Ferrando, P. (2016). An extended multidimensional IRT formulation for the linear item factor analysis model. Methodology, 12, 1–10. https://doi.org/10.1027/1614-2241/a000098 First citation in articleLink, Google Scholar

  • Ferrando, P., & Lorenzo-Seva, U. (2007a). An item response theory model for incorporating response time data in binary personality items. Applied Psychological Measurement, 31, 525–543. https://doi.org/10.1177/0146621606295197 First citation in articleCrossref, Google Scholar

  • Ferrando, P., & Lorenzo-Seva, U. (2007b). A measurement model for Likert responses that incorporates response time. Multivariate Behavioral Research, 42, 675–706. https://doi.org/10.1080/00273170701710247 First citation in articleCrossref, Google Scholar

  • GESIS. (2017). GESIS Panel – Standard Edition. (ZA5665 Data file Version 22.0.0 ed.). GESIS Data Archive. https://doi.org/10.4232/1.12903 First citation in articleGoogle Scholar

  • Goldberg, L., Johnson, J., Eber, H., Hogan, R., Ashton, M., Cloninger, C., & Gough, H. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84–96. https://doi.org/10.1016/j.jrp.2005.08.007 First citation in articleCrossref, Google Scholar

  • Hastie, T., Tibshirani, R., & Friedman, J. (2009). The elements of statistical learning. Berlin, Germany: Springer. First citation in articleCrossref, Google Scholar

  • Hernandez-Camacho, A., Olea, J., & Abad, F. (2017). Comparison of uni- and multidimensional models applied in testlet-based tests. Methodology, 13, 135–143. https://doi.org/10.1027/1614-2241/a000137 First citation in articleLink, Google Scholar

  • Höhne, J., Revilla, M., & Lenzner, T. (2018). Comparing the performance of agree/disagree and item-specific questions across PCs and smartphones. Methodology, 14, 109–118. https://doi.org/10.1027/1614-2241/a000151 First citation in articleLink, Google Scholar

  • Klein Entink, R., van der Linden, W., & Fox, J. (2009). A Box-Cox normal model for response times. British Journal of Mathematical and Statistical Psychology, 62, 621–640. https://doi.org/10.1348/000711008X374126 First citation in articleCrossref, Google Scholar

  • Kuncel, R. (1973). Response processes and relative location of subject and item. Educational and Psychological Measurement, 33, 545–563. https://doi.org/10.1177/001316447303300302 First citation in articleCrossref, Google Scholar

  • Loyes, T., Legrand, C., Schettino, A., & Pourtois, G. (2014). Semi-parametric proportional hazards models with crossed random effects for psychometric response times. British Journal of Mathematical and Statistical Psychology, 67, 304–327. https://doi.org/10.1111/bmsp.12020 First citation in articleCrossref, Google Scholar

  • Molenaar, D., Bolsinova, M., & Vermunt, J. (2017). A semi-parametric within subject mixture approach to the analyses of responses and response times. British Journal of Mathematical and Statistical Psychology, 71, 205–228. https://doi.org/10.1111/bmsp.12117 First citation in articleCrossref, Google Scholar

  • Molenaar, D., Tuerlinckx, F., & van der Maas, H. (2015a). A bivariate generalized linear item response theory modeling framework to the analysis of responses and response times. Multivariate Behavioral Research, 50, 56–74. https://doi.org/10.1080/00273171.2014.962684 First citation in articleCrossref, Google Scholar

  • Molenaar, D., Tuerlinckx, F., & van der Maas, H. (2015b). A generalized linear factor model approach to the hierarchical framework for responses and response times. British Journal of Mathematical and Statistical Psychology, 68, 197–219. https://doi.org/10.1111/bmsp.12042 First citation in articleCrossref, Google Scholar

  • Norets, A. (2010). Approximation of conditional densities by smooth mixtures of regressions. The Annals of Statistics, 38, 1733–1766. https://doi.org/10.1214/09-AOS765 First citation in articleCrossref, Google Scholar

  • Orlando, M., & Thissen, D. (2003). Further investigation of the performance of S–χ2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289–298. https://doi.org/10.1177/0146621603027004004 First citation in articleCrossref, Google Scholar

  • Ranger, J. (2013a). Modeling responses and response times in personality tests with rating scales. Psychological Test and Assessment Modeling, 55, 361–382. First citation in articleGoogle Scholar

  • Ranger, J. (2013b). A note on the hierarchical model for responses and response times in tests of van der Linden (2007). Psychometrika, 78, 538–544. https://doi.org/10.1007/s11336-013-9324-6 First citation in articleCrossref, Google Scholar

  • Ranger, J., & Kuhn, J. (2012). A flexible latent trait model for response times in tests. Psychometrika, 77, 31–47. https://doi.org/10.1007/s11336-011-9231-7 First citation in articleCrossref, Google Scholar

  • Ranger, J., & Kuhn, J. (2018). Modeling responses and response times in rating scales with the linear ballistic accumulator. Methodology, 14, 119–132. https://doi.org/10.1027/1614-2241/a000152 First citation in articleLink, Google Scholar

  • Ranger, J., & Ortner, T. (2012). A latent trait model for response times on tests employing the proportional hazards model. British Journal of Mathematical and Statistical Psychology, 65, 334–349. https://doi.org/10.1111/j.2044-8317.2011.02032.x First citation in articleCrossref, Google Scholar

  • Rouder, J. (2005). Are unshifted distributional models appropriate for response time? Psychometrika, 70, 377–381. https://doi.org/10.1007/s11336-005-1297-7 First citation in articleCrossref, Google Scholar

  • Russell, D., Kahn, J., Altmaier, E., & Spoth, R. (1998). Analyzing data from experimental studies: A latent variable structural equation modeling approach. Journal of Counseling Psychology, 45, 18–29. https://doi.org/10.1037/0022-0167.45.1.18 First citation in articleCrossref, Google Scholar

  • Tuerlinckx, F., Molenaar, D., & van der Maas, H. (2016). Diffusion-based item response theory modeling. In W. van der LindenR. HambletonEds., Handbook of modern item response theory (pp. 283–301). London, UK: Chapman and Hall/CRC Press. https://doi.org/10.1201/9781315374512 First citation in articleGoogle Scholar

  • van der Linden, W. (2007). A hierarchical framework for modeling speed and accuracy on test items. Psychometrika, 72, 287–308. https://doi.org/10.1007/s11336-006-1478-z First citation in articleCrossref, Google Scholar

  • van der Linden, W. (2008). Using response times for item selection in adaptive tests. Journal of Educational and Behavioral Statistics, 31, 5–20. https://doi.org/10.3102/1076998607302626 First citation in articleCrossref, Google Scholar

  • van der Linden, W. (2009a). A bivariate lognormal response-time model for the detection of collusion between test takers. Journal of Educational and Behavioral Statistics, 34, 378–394. https://doi.org/10.3102/1076998609332107 First citation in articleCrossref, Google Scholar

  • van der Linden, W. (2009b). Conceptual issues in response-time modeling. Journal of Educational Measurement, 46, 247–272. https://doi.org/10.1111/j.1745-3984.2009.00080.x First citation in articleCrossref, Google Scholar

  • van der Linden, W., Klein Entink, R., & Fox, J. (2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34, 327–347. https://doi.org/10.1177/0146621609349800 First citation in articleCrossref, Google Scholar

  • van der Maas, H., & Wagenmakers, E. (2005). A psychometric analysis of chess expertise. American Journal of Psychology, 118, 29–60. https://doi.org/10.2307/30039042 First citation in articleGoogle Scholar

  • Villani, M., Kohn, R., & Giordani, P. (2009). Regression density estimation using smooth adaptive Gaussian mixtures. Journal of Econometrics, 153, 155–173. https://doi.org/10.1016/j.jeconom.2009.05.004 First citation in articleCrossref, Google Scholar

  • Wang, C., Chang, H., & Douglas, J. (2013). The linear transformation model with frailties for the analysis of item response times. British Journal of Mathematical and Statistical Psychology, 66, 144–168. https://doi.org/10.1111/j.2044-8317.2012.02045.x First citation in articleCrossref, Google Scholar

  • Wang, C., Fan, Z., Chang, H., & Douglas, J. (2013). A semiparametric model for jointly analyzing response times and accuracy in computerized testing. Journal of Educational and Behavioral Statistics, 38, 381–417. https://doi.org/10.3102/1076998612461831 First citation in articleCrossref, Google Scholar

  • Wang, C., & Xu, G. (2015). A mixture hierarchical model for response times and response accuracy. British Journal of Mathematical and Statistical Psychology, 68, 456–477. https://doi.org/10.1111/bmsp.12054 First citation in articleCrossref, Google Scholar

  • White, H. (1982). Maximum likelihood estimation of misspecified models. Econometrica, 50, 1–25. https://doi.org/10.2307/1912526 First citation in articleCrossref, Google Scholar