Abstract
This review updates previous meta-analytical findings on validity indicators and provides new evidence on moderators of invalid performance, by investigating differences between noncredible and credible performances of clinical and non-clinical participants. Data from 133 studies (50 from previous meta-analyses and 83 new articles) were extracted and analyzed regarding types of research design, coaching, stimuli, and detection strategies. Overall effects were largest for experimental studies with non-clinical simulators vs community controls (Mean d = 1.648, 95% CI = 1.46–1.835, k = 41), and clinical simulators vs clinical controls (Mean d = 1.728, 95% CI = 1.224–2.232, k = 6), followed by known-groups comparisons (Mean d = 1.06, 95% CI = .955–1.166, k = 50) and experimental studies with community simulators vs patients (Mean d = .877, 95% CI = .751–1.004, k = 53). Similar to previous findings, symptom-coaching proved more effective than test-coaching in reducing differences between non-clinical simulators and clinical patients. In addition to the previous reviews, the analysis of stimuli material demonstrated the largest effects and resistance to coaching for tasks using numbers and letters & symbols. The analysis of detection strategies across types of contrasts, instruments, and coaching yielded the largest effects for Recognition. Effects were moderate for Magnitude of error, Performance curve, and Recall, lower and more variable for Reaction time, Floor effect, and Consistency, with stand-alone indicators generally proving larger differences than embedded indices. Methodological and practical implications are discussed related to testing validity indicators in research and associating them in assessment.
Similar content being viewed by others
Data Availability
The data that support the findings of this study are openly available in OSF, link for anonymous peer review https://osf.io/rtypw/?view_only=0718a7b2ecc045b9aeabc0cb40688a5c
References
Arnold, G., Boone, K. B., Lu, P., Dean, A., Wen, J., Nitch, S., & McPherson, S. (2005). Sensitivityand Specificity of Finger Tapping Test Scores for the Detection of SuspectEffort. The Clinical Neuropsychologist, 19, 105–120.
Barhon, L. I., Batchelor, J., Meares, S., Chekaluk, E., & Shores, E. A. (2014). A comparison of the degree of effort involved in the TOMM and the ACS word choice test using a dual-task paradigm. Applied Neuropsychology: Adult, 22(2), 1–10. https://doi.org/10.1080/23279095.2013.863775.
Beetar, J., & Williams, M. (1995). Malingering response styles on the memory assessment scales and symptom validity tests. Archives of Clinical Neuropsychology, 10, 57–72. https://doi.org/10.1016/0887-6177(94)E0005-A.
Bender, S. D., & Rogers, R. (2004). Detection of neurocognitive feigning: Development of a multi-strategy assessment. Archives of Clinical Neuropsychology, 19(1), 49–60. https://doi.org/10.1016/S0887-6177(02)00165-8.
Bernard, L., Houston, W., & Natoli, L. (1993). Malingering on neuropsychological tests – Potential objective indicators. Journal of Clinical Psychology, 49(1), 45–53. https://doi.org/10.1002/1097-4679(199301)49:13.0.CO;2-7.
Binder, L., & Willis, S. (1991). Assessment of motivation after financially compensable minor head injury. Psychological Assessment, 3, 175–181. https://doi.org/10.1037/1040-3590.3.2.175.
Bolan, B., Foster, J. K., Schmand, B., & Bolan, S. (2002). A comparison of three tests to detect feigned amnesia: The effects of feedback and the measurement of response latencies. Journal of Clinical and Experimental Neuropsychology, 24(2), 154–167. https://doi.org/10.1076/jcen.24.2.154.1000.
Boone, K. B., Lu, P., Sherman, D., Palmer, B., Black, C., Shameih, E., Warner-Chacon, K., & Sherman, N. G. (2000). Validation of a new technique to detect malingering of cognitive symptoms: The b test. Archives of Clinical Neuropsychology, 15(1), 227–241. https://doi.org/10.1093/arclin/15.3.227.
Brennan, A. M., & Gouvier, W. D. (2006). Are we honestly studying malingering? A profile and comparison of simulated and suspected malingerers. Applied Neuropsychology, 13(1), 1–11. https://doi.org/10.1207/s15324826an1301_1.
Cato, M. A., Brewster, J., Ryan, T., & Giuliano, A. J. (2002). Coaching and the ability to simulate mild traumatic brain injury symptoms. The Clinical Neuropsychologist, 16(4), 524–535. https://doi.org/10.1076/clin.16.4.524.13901.
DiCarlo, M. A., Gfeller, J. D., & Oliveri, M. V. (1998). Effects of coaching on detecting feigned cognitive impairment with the category test. Archives of Clinical Neuropsychology, 13, 23–23.
Durlak, J. (2008). Understanding meta-analysis. In L. G. Grimm & P. R. Yarnold (Eds.), Reading and understanding multivariate statistics (pp. 319–338). American Psychological Association.
Duval, S., & Tweedie, R. (2000). Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics, 56(2), 455–463. https://doi.org/10.1111/j.0006-341x.2000.00455.x.
Goette, W. F., & Goette, H. E. (2018). A meta-analysis of the accuracy of embedded performance validity indicators from the repeatable battery for the assessment of neuropsychological status. The Clinical Neuropsychologist, 33(6), 1044–1068. https://doi.org/10.1080/13854046.2018.1538429.
Green, P., Montijo, J., & Brockhaus, R. (2011). High specificity of the word memory test and medical symptom validity test in groups with severe verbal memory impairment. Applied Neuropsychology, 18(2), 86–94. https://doi.org/10.1080/09084282.2010.523389.
Harrison, C. (2008). Assessing malingered responding: Concurrent validation of a forced-choice test using ink blot stimuli for the identification of malingered responses [Unpublished doctoral dissertation]. University of Mississippi
Heilbronner, R. L., Sweet, J. J., Morgan, J. E., Larrabee, G. J., & Millis, S. R. (2009). American Academy of clinical neuropsychology consensus conference statement on the neuropsychological assessment of effort, response bias, and malingering. The Clinical Neuropsychologist, 23(7), 1093–1129. https://doi.org/10.1080/13854040903155063.
Heinly, M. T., Greve, K. W., & Bianchini, K. J. (2005). WAIS digit-span based indicators of malingered neurocognitive dysfunction: Classification accuracy in traumatic brain injury. Assessment, 12(4), 429–444. https://doi.org/10.1177/1073191105281099.
Hilsabeck, R. C., & Gouvier, D. M. (2005). Detecting simulated memory impairment: Further validation of the word completion memory test (WCMT). Archives of Clinical Neuropsychology, 20(8), 1025–1041. https://doi.org/10.1016/j.acn.2005.05.003.
Hilsabeck, R. C., LeCompte, D. C., Marks, A. R., & Grafman, J. (2001). The word completion memory test (WCMT): A new test to detect malingered memory deficits. Archives of Clinical Neuropsychology, 16(7), 669–678. https://doi.org/10.1093/arclin/16.7.669.
Hiscock, C., Branham, J., & Hiscock, M. (1994). Detection of feigned cognitive impairment: The two-alternative forced-choice method compared with selected conventional tests. Journal of Psychopathology and Behavioral Assessment, 16, 95–110. https://doi.org/10.1007/BF02232721.
Inman, T. H., & Berry, D. T. R. (2002). Cross-validation of indicators of malingering: A comparison of nine neuropsychological tests, four tests of malingering, and behavioral observations. Archives of Clinical Neuropsychology, 17, 1–23. https://doi.org/10.1016/S0887-6177(00)00073-1.
Inman, T., Vickery, C., Berry, D., Lamb, D., Edwards, C., & Smith, G. (1998). Development and initial validation of a new procedure for evaluating adequacy of effort given during neuropsychological testing: The Letter Memory Test. Psychological Assessment, 10, 128–139. https://doi.org/10.1037/1040-3590.10.2.128.
Jasinski, L. J., Berry, D. T., Shandera, A. L., & Clark, J. A. (2011). Use of the Wechsler adult intelligence scale digit span subtest for malingering detection: A meta-analytic review. Journal of Clinical and Experimental Neuropsychology, 33(3), 300–314. https://doi.org/10.1080/13803395.2010.516743.
Jelicic, M., Ceunen, E., Peters, M. J., & Merckelbach, H. (2011). Detecting coached feigning using the test of memory malingering (TOMM) and the structured interview of malingered symptomatology (SIMS). Journal of Clinical Psychology, 67(9), 850–855. https://doi.org/10.1002/jclp.20805.
Jones, A. (2013). Victoria symptom validity test: Cutoff scores for psychometrically defined malingering groups in a military sample. The Clinical Neuropsychologist, 27, 1373–1384. https://doi.org/10.1080/13854046.2013.851740.
Jones, S. M. (2017). Dissimulation strategies on standard neuropsychological tests: A qualitative investigation. Brain Injury, 31(7), 1–11. https://doi.org/10.1080/02699052.2017.1283444.
Kanser, R. J., Rapport, L. J., Bashem, J. R., & Hanks, R. A. (2018). Detecting malingering in traumatic brain injury: Combining response time with performance validity test accuracy. The Clinical Neuropsychologist, 33(1), 1–18. https://doi.org/10.1080/13854046.2018.1440006.
Lace, J. W., Grant, A. F., Ruppert, P., Kaufman, D. A. S., Teague, C. L., Lowell, K., & Gfeller, J. D. (2020). Identifying novel embedded performance validity test formulas within the Repeatable Battery for the Assessment of Neuropsychological Status: A simulation study. Psychological Injury and Law, 13, 303–315. https://doi.org/10.1007/s12207-020-09382-x.
Lay-Khim, G., & Bit-Lian, Y. (2019). Simulated patients’ experience towards simulated patient-based simulation session: A qualitative study. SciMedicine Journal, 1(2), 55–63. https://doi.org/10.28991/SciMedJ-2019-0102-3.
Leighton, A., Weinborn, M., & Maybery, M. (2014). Bridging the gap between neurocognitive processing theory and performance validity assessment among the cognitively impaired: A review and methodological approach. Journal of the International Neuropsychological Society, 20(1), 873–886. https://doi.org/10.1017/S135561771400085X.
Martin, P. K., Schroeder, R. W., & Odland, A. P. (2015). Neuropsychologists‘ validity testing beliefs and practices: A survey of north American professionals. The Clinical Neuropsychologist, 29(6), 741–766. https://doi.org/10.1080/13854046.2015.1087597.
Martin, R., Bolter, J., Todd, M., Gouvier, W., & Niccolls, R. (1993). Effects of sophistication and motivation on the detection of malingered memory performance using a computerized forced-choice task. Journal of Clinical and Experimental Neuropsychology, 15, 867–880. https://doi.org/10.1080/01688639308402604.
Martin, P. K., Schroeder, R. W., Olsen, D. H., Maloy, H., Boettcher, A., Ernst, N., & Okut, H. (2019). A systematic review and meta-analyses of the test of memory malingering in adults: Two decades of deception detection. The Clinical Neuropsychologist, 34(1), 88–119. https://doi.org/10.1080/13854046.2019.1637027.
McHugh, M. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://doi.org/10.11613/BM.2012.031.
Merten, T., Bossink, L., & Schmand, B. (2007). On the limits of effort testing: Symptom validity tests and severity of neurocognitive symptoms in nonlitigant patients. Journal of Clinical & Experimental Neuropsychology, 29(3), 308–318. https://doi.org/10.1080/13803390600693607.
Nitch, S., Boone, K. B., Wen, J., Arnold, G., & Alfano, K. (2006). The Utility of the Rey Word Recognition Test in the Detection of Suspect Effort. The Clinical Neuropsychologist, 20, 873–887.
Pivovarova, E., Rosenfeld, B., Dole, T., Green, D., & Zapf, P. (2009). Are measures of cognitive effort and motivation useful in differentiating feigned from genuine psychiatric symptoms? The International Journal of Forensic Mental Health, 8, 271–278. https://doi.org/10.1080/14999011003635514.
Powell, M. R., Gfeller, J. D., Hendriks, B. L., & Sharland, M. (2004). Detecting symptom- and test-coached simulators with the Test of Memory Malingering. Archives of Clinical Neuropsychology, 19(5), 693–702. https://doi.org/10.1016/j.acn.2004.04.001.
Prigatano, G., & Amin, K. (1993). Digit Memory Test: Unequivocal cerebral dysfunction and suspected malingering. Journal of Clinical and Experimental Neuropsychology, 15, 537–546. https://doi.org/10.1080/01688639308402577.
Rees, L. M., Tombaugh, T. N., Gansler, D. A., & Moczynski, N. P. (1998). Five validation experiments of the Test of Memory Malingering (TOMM). Psychological Assessment, 10(1), 10–20. https://doi.org/10.1037/1040-3590.10.1.10.
Rickards, T. A., Cranston, C. C., Touradij, P., & Bechtold, K. T. (2017). Embedded performance validity testing in neuropsychological assessment: Potential clinical tools. Applied Neuropsychology: Adult, 31(1), 1–12. https://doi.org/10.1080/23279095.2017.1278602.
Rogers, R. (2008). Clinical assessment of malingering and deception. Guilford Press.
Rogers, R., Harrell, E. H., & Liff, C. D. (1993). Feigning neuropsychological impairment: A critical review of methodological and clinical considerations. Clinical Psychology Review, 13(3), 255–274. https://doi.org/10.1016/0272-7358(93)90023-F.
Rüsseler, J., Brett, A., Klaue, U., Seiler, M., & Münte, T. (2008). The effect of coaching on the simulated malingering of memory impairment. BMC Neurology, 37(8), 1–14. https://doi.org/10.1186/1471-2377-8-37.
Schroeder, R. W., Twumasi-Ankrah, P., Baade, L. E., & Marshall, P. S. (2012). Reliable digit span: A systematic review and cross-validation study. Assessment, 19(1), 21–30. https://doi.org/10.1177/1073191111428764.
Schwartz, E. S., Erdodi, L., Rodriguez, N., Ghosh, J. J., Curtain, J. R., Flashman, L. A., & Roth, R. M. (2016). CVLT-II Forced Choice Recognition Trial as an Embedded Validity Indicator: A Systematic Review of the Evidence. Journal of the International Neuropsychological Society, 22, 851–858.
Shura, R. D., Brearly, R. W., Rowland, J. A., Martindale, S. L., Miskey, H. M., & Duff, K. (2018). RBANS validity indices: A systematic review and meta-analysis. Neuropsychology Review, 28(3), 269–284. https://doi.org/10.1007/s11065-018-9377-5.
Silk-Eglit, G. M., Lynch, J. K., & Mccaffrey, R. J. (2016). Validation of Victoria symptom validity test cutoff scores among mild traumatic brain injury litigants using a known-groups design. Archives of Clinical Neuropsychology, 31(3), 231–245. https://doi.org/10.1093/arclin/acv108.
Slick, D. J., Sherman, E. M., & Iverson, J. (1999). Diagnostic criteria for malingered neurocognitive dysfunction. Proposed standards for clinical practice and research. The Clinical Neuropsychologist, 13(4), 545–561. https://doi.org/10.1076/1385-4046(199911)13:04;1-Y;FT545.
Sollman, M. J., & Berry, D. T. (2011). Detection of inadequate effort on neuropsychological testing: A meta-analytic update and extension. Archives of Clinical Neuropsychology, 26(8), 774–789. https://doi.org/10.1093/arclin/acr066.
Strauss, E., Slick, D. J., Levy-Bencheton, J., Hunter, M., MacDonald, S. W. S., & Hultsch, D. F. (2002). Intraindividual variability as an indicator of malingering in head injury. Archives of Clinical Neuropsychology, 17, 423–444. https://doi.org/10.1093/arclin/17.5.423.
Strutt, A. M., Scott, B. M., Lozano, V. J., Tieu, P. G., & Peery, S. (2012). Assessing suboptimal performance with the Test of Memory Malingering in Spanish speaking patients with TBI. Brain Injury, 26(6), 853–863. https://doi.org/10.3109/02699052.2012.655366.
Suhr, J. A. (2002). Malingering, coaching, and the serial position effect. Archives of Clinical Neuropsychology, 17(1), 69–77. https://doi.org/10.1093/arclin/17.1.69.
Sullivan, L. E. (2000). Malingering of head injury on neuropsychological instruments: A meta-analytic review [unpublished doctoral dissertation]. Simon Fraser University.
Tan, J. E., Slick, D. J., Strauss, E., & Hultsch, D. F. (2002). How’d they do it? Malingering strategies on symptom validity tests. The Clinical Neuropsychologist, 16(4), 495–505. https://doi.org/10.1076/clin.16.4.495.13909.
Tombaugh, T. N. (1997). The Test of Memory Malingering (TOMM): Normative data from cognitively intact and cognitively impaired individuals. Psychological Assessment, 9, 260–268.
Tukey, J. W. (1977). Exploratory data analysis. Addison-Wesley Publishing Company.
Vickery, C. D., Berry, D. T., Inman, T. H., Harris, M. J., & Orey, S. A. (2001). Detection of inadequate effort on neuropsychological testing: A meta-analytic review of selected procedures. Archives of Clinical Neuropsychology, 16(1), 45–73. https://doi.org/10.1093/arclin/16.1.45.
Weinborn, M., Woods, S. P., Nulsen, C., & Leighton, A. (2012). The effects of coaching on the verbal and nonverbal medical symptom validity tests. The Clinical Neuropsychologist, 26(5), 832–849. https://doi.org/10.1080/13854046.2012.686630.
Willison, J., & Tombaugh, T. N. (2006). Detecting simulation of attention deficits using reaction time tests. Archives of Clinical Neuropsychology, 21(1), 41–52. https://doi.org/10.1016/j.acn.2005.07.005.
Funding
No funding was available for this study.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors report no potential conflict of interest.
Ethical Approval
All performed procedures were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.
Consent to Participate
As we used exclusively published data that was publicly available, no explicit informed consent was obtained from individual participants included in the study.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Supplementary Information
ESM 1
(DOCX 113 kb)
Rights and permissions
About this article
Cite this article
Crişan, I., Maricuţoiu, LP. & Sava, FA. Strategies to detect invalid performance in cognitive testing: An updated and extended meta-analysis. Curr Psychol 42, 3236–3257 (2023). https://doi.org/10.1007/s12144-021-01659-x
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s12144-021-01659-x