Skip to main content
SpringerLink
Log in

Insufficient Effort Responding as a Potential Confound between Survey Measures and Objective Tests

  • Original Paper
  • Published:
Journal of Business and Psychology Aims and scope Submit manuscript

Abstract

Following research that demonstrates insufficient effort responding (IER) may confound survey measures and inflate observed correlations (Huang, Liu, & Bowling, 2015c), a question emerges as to whether and when IER can act as a confound between objective tests and surveys. Using data (N = 243) originally designed to examine training and transfer, study 1 demonstrates that (a) IER is negatively related to performance on tests, and (b) IER’s influence on surveys depends on the sample means of these measures. As a result, IER could inflate a test’s association with other tests and surveys. Study 2 investigates the impact of two parameters—within-person consistency of IER and percentage of IER cases in the sample—by randomly replacing bootstrapped attentive responses (10,000 bootstrapped samples of 200 cases identified from study 1). When predicting the confounding effects of IER, within-person consistency has positive linear and quadratic effects, percentage of IER cases has a positive linear effect, and consistency and percentage have a positive interactive effect. Research and practical implications for the design and evaluation of surveys and tests are discussed.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. We focus on within-person consistency as the percentage of items in a given data collection that a participant responds to with insufficient effort. This narrower focus is distinct from rank-order stability, which is reflected in a correlation coefficient between IER measures obtained from two different survey administrations (see Bowling et al., 2016). Rank-order stability captures the consistency of individuals’ relative standings on their IER behavior across time and situations.

  2. We should note that when attentive responses have an average near the scale midpoint, positive and negative errors tend to cancel each other out, resulting in IER scores near the scale midpoint as well.

  3. See Huang et al. (2012) and Meade and Craig (2012) for procedures to calculate the psychometric antonym index, psychometric synonym index, and individual reliability. Details of all IER indices are available from the first author.

  4. Although it may be tempting to interpret these correlations as if they indicate substantive associations (e.g., respondents with low CSE tended to engage in IER), we encourage readers to interpret these relationships with caution. We do not intend to establish associations between IER and substantive constructs in this paper because respondents who engage in IER produce scores that may not validly indicate their standing on these substantive measures. As a result, these correlations are partially a function of the methodological confound of IER (see Huang, Liu et al., 2015; McGonagle et al., 2016).

  5. The lack of support for Hypotheses 2 and 3 involving strategic knowledge might be attributable to the high difficulty level of the strategic knowledge test. On a difficult test, attentive respondents cannot outperform inattentive respondents by much — a floor effect. As a result, IER will share a small correlation with observed test score. In the case of strategic knowledge, the test was indeed quite difficult (M = 41.89 on a 100-point scale) and its correlation with IER was rather weak (r = −.18).

  6. Syntax available at https://osf.io/cwt7z/

References

  • Aguinis, H., & Vandenberg, R. J. (2014). An ounce of prevention is worth a pound of cure: Improving research quality before data collection. Annual Review of Organizational Psychology and Organizational Behavior, 1, 569–595.

    Article  Google Scholar 

  • Arvey, R. D., Strickland, W., Drauden, G., & Martin, C. (1990). Motivational components of test taking. Personnel Psychology, 43, 695–716.

    Article  Google Scholar 

  • Baer, R. A., Ballenger, J., Berry, D. T. R., & Wetter, M. W. (1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68, 139–151.

    Article  PubMed  Google Scholar 

  • Bell, B. S., & Kozlowski, S. W. J. (2008). Active learning: Effects of core training design elements on self-regulatory processes, learning, and adaptability. Journal of Applied Psychology, 93, 296–316.

    Article  PubMed  Google Scholar 

  • Berry, D. T. R., Wetter, M. W., Baer, R. A., Larsen, L., Clark, C., & Monroe, K. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340–345.

    Article  Google Scholar 

  • Bliese, P. (2016). Package ‘multilevel’ (version 2.6). Retrieved April 2017 from: https://cran.r-project.org/web/packages/multilevel/.

  • Blume, B. D., Ford, J. K., Baldwin, T. T., & Huang, J. L. (2010). Transfer of training: A meta-analytic review. Journal of Management, 36, 1065–1105.

    Article  Google Scholar 

  • Bosco, F. A., Aguinis, H., Singh, K., Field, J. G., & Pierce, C. A. (2015). Correlational effect size benchmarks. Journal of Applied Psychology, 100, 431–449.

    Article  PubMed  Google Scholar 

  • Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111, 218–229.

    Article  PubMed  Google Scholar 

  • Brown, K. G. (2005). An examination of the structure and nomological network of trainee reactions: A closer look at “smile sheets”. Journal of Applied Psychology, 90, 991–1001.

    Article  PubMed  Google Scholar 

  • Callans, M. (2012, December 28). Non-proctored vs. proctored assessments. Retrieved from http://blog.wonderlic.com/nonproctoredvsproctoredassessments

  • Caplan, B., & Miller, S. C. (2010). Intelligence makes people think like economists: Evidence from the General Social Survey. Intelligence, 38, 636–647.

  • Carsey, T. M., & Harden, J. J. (2014). Monte Carlo simulation and resampling methods for social science. Thousand Oaks, CA: Sage.

    Book  Google Scholar 

  • Chang, C. H., Ferris, D. L., Johnson, R. E., Rosen, C. C., & Tan, J. A. (2012). Core self-evaluations: A review and evaluation of the literature. Journal of Management, 38, 81–128.

    Article  Google Scholar 

  • Chen, G., Thomas, B., & Wallace, J. (2005). A multilevel examination of the relationships among training outcomes, mediating regulatory processes, and adaptive performance. Journal of Applied Psychology, 90, 827–841.

    Article  PubMed  Google Scholar 

  • Chernyshenko, O. S., Stark, S., Chan, K., Drasgow, F., & Williams, B. (2001). Fitting item response theory models to two personality inventories: Issues and insights. Multivariate Behavioral Research, 36, 523–562.

    Article  PubMed  Google Scholar 

  • Cheung, J. H., Burns, D. K., Sinclair, R. R., & Sliter, M. (2017). Amazon Mechanical Turk in organizational psychology: An evaluation and practical recommendations. Journal of Business and Psychology, 32, 347–361.

  • Cianci, A. M., Klein, H. J., & Seijts, G. H. (2010). The effect of negative feedback on tension and subsequent performance: The main and interactive effects of goal content and conscientiousness. Journal of Applied Psychology, 95, 618–630.

    Article  PubMed  Google Scholar 

  • Clark, M. E., Gironda, R. J., & Young, R. W. (2003). Detection of back random responding: Effectiveness of MMPI-2 and personality assessment inventory validity indices. Psychological Assessment, 15, 223–234.

    Article  PubMed  Google Scholar 

  • Colquitt, J. A., LePine, J. A., & Noe, R. A. (2000). Toward an integrative theory of training motivation: A meta-analytic path analysis of 20 years of research. Journal of Applied Psychology, 85, 678–707.

    Article  PubMed  Google Scholar 

  • Cor, M. K., Haertel, E., Krosnick, J. A., & Malhotra, N. (2012). Improving ability measurement in surveys by following the principles of IRT: The Wordsum vocabulary test in the General Social Survey. Social Science Research, 41, 1003–1016.

    Article  PubMed  Google Scholar 

  • Cortina, J. M. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6, 415–439.

    Article  Google Scholar 

  • Credé, M. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70, 596–612.

    Article  Google Scholar 

  • Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.

    Article  Google Scholar 

  • DeSimone, J. A., Davison, H. K., Schoen, J. L., & Bing, M. N. (2020). Insufficient effort responding as a partial function of implicit aggression. Organizational Research Methods, 23, 154–180.

    Article  Google Scholar 

  • DeSimone, J. A., DeSimone, A. J., Harms, P. D., & Wood, D. (2018). The differential impacts of two forms of insufficient effort responding. Applied Psychology: An International Review, 67, 309–338.

    Article  Google Scholar 

  • DeSimone, J. A., & Harms, P. D. (2018). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 33, 559–577.

    Article  Google Scholar 

  • DeSimone, J. A., Harms, P. D., & DeSimone, A. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171–181.

    Article  Google Scholar 

  • Dixon, W. J., & Mood, A. M. (1946). The statistical sign test. Journal of the American Statistical Association, 41, 557–566.

    Article  PubMed  Google Scholar 

  • Duckworth, A. L., Quinn, P. D., Lynam, D. R., Loeber, R., & Stouthamer-Loeber, M. (2011). Role of test motivation in intelligence testing. Proceedings of the National Academy of Sciences, 108, 7716–7720.

    Article  Google Scholar 

  • Elliot, A. J., McGregor, H. A., & Gable, S. (1999). Achievement goals, study strategies, and exam performance: A mediational analysis. Journal of Educational Psychology, 91, 549–563.

    Article  Google Scholar 

  • Endler, N. S., & Magnusson, D. (1976). Toward an interactional psychology of personality. Psychological Bulletin, 83, 956–974.

    Article  PubMed  Google Scholar 

  • Fleischer, A., Mead, A. D., & Huang, J. (2015). Inattentive responding in MTurk and other online samples. Industrial and Organizational Psychology, 8, 196–202.

    Article  Google Scholar 

  • Ford, J. K., Smith, E. M., Weissbein, D. A., Gully, S. M., & Salas, E. (1998). Relationships of goal orientation, metacognitive activity, and practice strategies with learning outcomes and transfer. Journal of Applied Psychology, 83, 218–233.

    Article  Google Scholar 

  • Galesic, M., & Bosnjak, M. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73, 349–360.

    Article  Google Scholar 

  • Goldberg, L. R. (1999). A broad-bandwidth, public-domain, personality inventory measuring the lower-level facets of several five-factor models. In I. Mervielde, I. Deary, F. D. Fruyt & F. Ostendorf (Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilburg, The Netherlands: Tilburg University Press.

  • Green, S. B., & Stutzman, T. (1986). An evaluation of methods to select respondents to structured job-analysis questionnaires. Personnel Psychology, 39, 543–564.

    Article  Google Scholar 

  • Guttman, L. (1944). A basis for scaling qualitative data. American Sociological Review, 9, 139–150.

    Article  Google Scholar 

  • Harms, P. D., & DeSimone, J. A. (2015). Caution! MTurk workers ahead—fines doubled. Industrial and Organizational Psychology, 8, 183–190.

    Article  Google Scholar 

  • Harrison, J. R., Lin, Z., Carroll, G. R., & Carley, K. M. (2007). Simulation modeling in organizational and management research. Academy of Management Review, 32, 1229–1245.

    Article  Google Scholar 

  • Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7, 191–205.

    Article  Google Scholar 

  • Hough, L. M., Eaton, N. K., Dunnette, M. D., Kamp, J. D., & McCloy, R. A. (1990). Criterion-related validities of personality constructs and the effect of response distortion on those validities. Journal of Applied Psychology, 75, 581–595.

    Article  Google Scholar 

  • Huang, J. L., Blume, B. D., Ford, J. K., & Baldwin, T. T. (2015a). A tale of two transfers: Disentangling maximum and typical transfer and their respective predictors. Journal of Business and Psychology, 30, 709–732.

    Article  Google Scholar 

  • Huang, J. L., Bowling, N. A., Liu, M., & Li, Y. (2015b). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299–311.

    Article  Google Scholar 

  • Huang, J. L., & Bramble, R. J. (2016). Trait, state, and task-contingent conscientiousness: Influence on learning and transfer. Personality and Individual Differences, 92, 180–185.

    Article  Google Scholar 

  • Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.

    Article  Google Scholar 

  • Huang, J. L., Liu, M., & Bowling, N. A. (2015c). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828–845.

    Article  PubMed  Google Scholar 

  • Jarvis, B. (2008). MediaLab (version 2008) [computer software]. New York: Empirisoft.

    Google Scholar 

  • Judge, T. A., Erez, A., Bono, J. E., & Thoresen, C. J. (2003). The core self-evaluations scale: Development of a measure. Personnel Psychology, 56, 303–331.

    Article  Google Scholar 

  • Kam, C. C. S., & Meyer, J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18, 512–541.

    Article  Google Scholar 

  • Kanfer, R., & Ackerman, P. L. (1989). Motivation and cognitive abilities: An integrative/aptitude-treatment interaction approach to skill acquisition. Journal of Applied Psychology, 74, 657–690.

    Article  Google Scholar 

  • Kanfer, R., Ackerman, P. L., Murtha, T. C., Dugdale, B., & Nelson, L. (1994). Goal setting, conditions of practice, and task performance: A resource allocation perspective. Journal of Applied Psychology, 79, 826–835.

    Article  Google Scholar 

  • Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 252–268.

    Article  Google Scholar 

  • Kotrba, L. M., Nieminen, L., Denison, D., & Carter, N. T. (April, 2014). Respondent versus response screening: Looking beyond the class clowns. Paper presented at the annual meeting of the Society for Industrial and Organizational Psychology: Honolulu, HI.

  • Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328.

    Article  Google Scholar 

  • Kraiger, K., & Jerden, E. (2007). A meta-analytic investigation of learner control: Old findings and new directions. In S. M. Fiore & E. Salas (Eds.), Toward a science of distributed learning (pp. 65–90). Washington, DC: APA.

    Chapter  Google Scholar 

  • Kreft, I. G. G., & de Leeuw, J. (1998). Introducing multilevel modeling. Newbury Park, CA: Sage.

    Book  Google Scholar 

  • Kutner, M. H., Nachtsheim, C. J., Neter, J., & Li, W. (2004). Applied linear statistical models (5th ed.). New York: McGraw-Hill/Irwin.

    Google Scholar 

  • Liu, M., Bowling, N. A., Huang, J. L., & Kent, T. A. (2013). Insufficient effort responding to surveys as a threat to validity: The perceptions and practices of SIOP members. The Industrial-Organizational Psychologist, 51, 32–38.

    Google Scholar 

  • Longstaff, H. P., & Porter, J. P. (1928). Speed and accuracy as factors in objective tests in general psychology. Journal of Applied Psychology, 12, 636–642.

    Article  Google Scholar 

  • Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.

    Article  Google Scholar 

  • McGonagle, A. K., Huang, J. L., & Walsh, B. M. (2016). Insufficient effort survey responding: An under-appreciated problem in work and organizational health psychology research. Applied Psychology: An International Review, 65, 287–321.

    Article  Google Scholar 

  • McGrath, R. E., Mitchell, M., Kim, B. H., & Hough, L. (2010). Evidence for response bias as a source of error variance in applied assessment. Psychological Bulletin, 136, 450–470.

    Article  PubMed  Google Scholar 

  • Meyer, R. D., Dalal, R. S., & Hermida, R. (2010). A review and synthesis of situational strength in the organizational sciences. Journal of Management, 36, 121–140.

    Article  Google Scholar 

  • Noe, R. A. (1986). Trainees’ attributes and attitudes: Neglected influences on training effectiveness. Academy of Management Review, 11, 736–749.

    Article  Google Scholar 

  • Nunnally, J. C. (1978). Psychometric theory (2nd ed.). New York: McGraw-Hill.

    Google Scholar 

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716.

    Article  Google Scholar 

  • Oppenheimer, D. M., Meyvis, T., & Davidenko, N. (2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45, 867–872.

    Article  Google Scholar 

  • Pearson, K. (1903). Mathematical contributions to the theory of evolution-XI. On the influence of natural selection on the variability and correlation of organs. Philosophical Transactions of the Royal Society, London, Series A, 200, 1–66.

    Article  Google Scholar 

  • Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., & Podsakoff, N. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88, 879–903.

    Article  PubMed  Google Scholar 

  • Ran, S., Liu, M., Marchiondo, L. A., & Huang, J. L. (2015). Difference in response effort across sample types: Perception or reality? Industrial and Organizational Psychology, 8, 202–208.

    Article  Google Scholar 

  • Schmitt, N., & Stults, D. M. (1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9, 367–373.

    Article  Google Scholar 

  • Schonlau, M., & Toepoel, V. (2015). Straightlining in Web survey panels over time. Survey Research Methods, 9, 125–137.

    Google Scholar 

  • Singer, J. D. (1998). Using SAS PROCMIXED to fit multilevel models, hierarchical models, and individual growth models. Journal of Educational and Behavioral Statistics, 24, 323–355.

    Article  Google Scholar 

  • Spearman, C. (1904). The proof and measurement of association between two things. American Journal of Psychology, 15, 72–101.

    Article  Google Scholar 

  • Stanhope, D. S., Pond III, S. B., & Surface, E. A. (2013). Core self-evaluations and training effectiveness: Prediction through motivational intervening mechanisms. Journal of Applied Psychology, 98, 820–831.

    Article  PubMed  Google Scholar 

  • Stevens, C. K., & Gist, M. E. (1997). Effects of self-efficacy and goal-orientation training on negotiation skill maintenance: What are the mechanisms? Personnel Psychology, 50, 955–978.

    Article  Google Scholar 

  • Thorndike, R. L. (1949). Personnel selection: Test and measurement techniques. New York: Wiley.

    Google Scholar 

  • Ward, M. K., & Pond III, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on Internet-based surveys. Computers in Human Behavior, 48, 554–568.

  • Warr, P., Allan, C., & Birdi, K. (1999). Predicting three levels of training outcome. Journal of Occupational and Organizational Psychology, 72, 351–375.

    Article  Google Scholar 

  • Williams, L. J., Hartman, N., & Cavazotte, F. (2010). Method variance and marker variables: A review and comprehensive CFA marker technique. Organizational Research Methods, 13, 477–514.

    Article  Google Scholar 

  • Wise, S. L. (2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19, 95–114.

    Article  Google Scholar 

  • Wise, S. L., & DeMars, C. E. (2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43, 19–38.

    Article  Google Scholar 

  • Wise, S. L., & Kong, X. (2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18, 163–183.

    Article  Google Scholar 

  • Wood, D., Harms, P. D., Lowman, G. H., & DeSimone, J. A. (2017). Response speed and response consistency as mutually validating indicators of data quality in online samples. Social Psychological and Personality Science, 8, 454–464.

    Article  Google Scholar 

  • Zimmerman, B. J. (2000). Attaining self-regulation: A social cognitive perspective. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 13–39). San Diego, CA: Academic Press.

    Chapter  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. School of Human Resources and Labor Relations, Michigan State University, 368 Farm Lane, 437 SKH, East Lansing, MI, 48824, USA

    Jason L. Huang

  2. Culverhouse College of Business, University of Alabama, Tuscaloosa, AL, USA

    Justin A. DeSimone

Authors
  1. Jason L. Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Justin A. DeSimone
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Jason L. Huang.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Huang, J.L., DeSimone, J.A. Insufficient Effort Responding as a Potential Confound between Survey Measures and Objective Tests. J Bus Psychol 36, 807–828 (2021). https://doi.org/10.1007/s10869-020-09707-2

Download citation

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10869-020-09707-2

Keywords

Search

Navigation