Skip to main content
Search
Search
    Quick Search anywhere
    Quick search in Citations
Advanced Search
Skip main navigationClose Drawer MenuOpen Drawer Menu
Previous article
Next article
Original Articles and Reviews

Are Difficult-To-Study Populations too Difficult to Study in a Reliable Way?

Lessons Learned From Meta-Analyses in Clinical Neuropsychology

Published Online:December 06, 2019https://doi.org/10.1027/1016-9040/a000384

Abstract

Abstract. Replication studies, pre-registration, and increases in statistical power will likely improve the reliability of scientific evidence. However, these measures face critical limitations in populations that are inherently difficult to study. Members of difficult-to-study populations (e.g., patients, children, non-human animals) are less accessible to researchers, which typically results in small-sample studies that are infeasible to replicate. Nevertheless, meta-analyses on clinical neuropsychological data suggest that difficult-to-study populations can be studied in a reliable way. These analyses often produce unbiased effect-size estimates despite aggregating across severely underpowered original studies. This finding can be attributed to a neuropsychological research culture involving the non-selective reporting of results from standardized and validated test procedures. Consensus guidelines, test manuals, and psychometric evidence constrain the methodological choices made by neuropsychologists, who regularly report the results from neuropsychological test batteries irrespective of their statistical significance or novelty. Comparable shifts toward more standardization and validation, complete result reports, and between-lab collaborations can allow for a meaningful and reliable study of psychological phenomena in other difficult-to-study populations.

References

  • Abrahams, S., Goldstein, L. H., Kew, J. J. M., Brooks, D. J., Lloyd, C. M., Frith, C. D., & Leigh, P. N. (1996). Frontal lobe dysfunction in amyotrophic lateral sclerosis A PET study. Brain, 119, 2105–2120. https://doi.org/10.1093/brain/119.6.2105 First citation in articleCrossref, Google Scholar

  • American Psychological Association. (2010). Clinical Neuropsychology. Retrieved from http://www.apa.org/ed/graduate/specialize/neuro.aspx First citation in articleGoogle Scholar

  • Asendorpf, J. B., Conner, M., De Fruyt, F., De Houwer, J., Denissen, J. J., Fiedler, K., … Perugini, M. (2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27, 108–119. https://doi.org/10.1002/per.1919 First citation in articleCrossref, Google Scholar

  • Baldwin, S. A. (2017). Improving the rigor of psychophysiology research. International Journal of Psychophysiology, 111, 5–16. https://doi.org/10.1016/j.ijpsycho.2016.04.006 First citation in articleCrossref, Google Scholar

  • Baumeister, R. F., & Vohs, K. D. (2016). Misguided effort with elusive implications. Perspectives on Psychological Science, 11, 574–575. https://doi.org/10.1177/1745691616652878 First citation in articleCrossref, Google Scholar

  • Begg, C. B., & Mazumdar, M. (1994). Operating characteristics of a rank correlation test for publication bias. Biometrics, 50, 1088–1101. https://doi.org/10.2307/2533446 First citation in articleCrossref, Google Scholar

  • Berg, E. A. (1948). A simple objective technique for measuring flexibility in thinking. The Journal of General Psychology, 39, 15–22. https://doi.org/10.1080/00221309.1948.9918159 First citation in articleCrossref, Google Scholar

  • Carter, E. C., Kofler, L. M., Forster, D. E., & McCullough, M. E. (2015). A series of meta-analytic tests of the depletion effect: Self-control does not seem to rely on a limited resource. Journal of Experimental Psychology: General, 144, 796–815. https://doi.org/10.1037/xge0000083 First citation in articleCrossref, Google Scholar

  • Carter, E. C., & McCullough, M. E. (2013). Is ego depletion too incredible? Evidence for the overestimation of the depletion effect. Behavioral and Brain Sciences, 36, 683–684. https://doi.org/10.1017/S0140525X13000952 First citation in articleCrossref, Google Scholar

  • Carter, E. C., & McCullough, M. E. (2014). Publication bias and the limited strength model of self-control: Has the evidence for ego depletion been overestimated? Frontiers in Psychology, 5, 823. https://doi.org/10.3389/fpsyg.2014.00823 First citation in articleCrossref, Google Scholar

  • Chiò, A., Logroscino, G., Traynor, B. J., Collins, J., Simeone, J. C., Goldstein, L. A., & White, L. A. (2013). Global epidemiology of amyotrophic lateral sclerosis: A systematic review of the published literature. Neuroepidemiology, 41, 118–130. https://doi.org/10.1159/000351153 First citation in articleCrossref, Google Scholar

  • Crawford, J. R. (2004). Psychometric foundations of neuropsychological assessment. In L. H. GoldsteinJ. E. McNeilEds., Clinical neuropsychology: A practical guide to assessment and management for clinicians (pp. 121–140). Chichester, UK: Wiley. First citation in articleGoogle Scholar

  • Demakis, G. J. (2003). A meta-analytic review of the sensitivity of the Wisconsin Card Sorting Test to frontal and lateralized frontal brain damage. Neuropsychology, 17, 255–264. https://doi.org/10.1037/0894-4105.17.2.255 First citation in articleCrossref, Google Scholar

  • Demakis, G. J. (2006). Meta-analysis in neuropsychology: An introduction. The Clinical Neuropsychologist, 20, 5–9. https://doi.org/10.1080/13854040500203266 First citation in articleCrossref, Google Scholar

  • Engber, D. (2016). Everything is crumbling. Slate. Retrieved from http://www.slate.com/articles/health_and_science/cover_story/2016/03/ego_depletion_an_influential_theory_in_psychology_may_have_just_been_debunked.html First citation in articleGoogle Scholar

  • Eysenck, H. J. (1978). An exercise in mega-silliness. American Psychologist, 33, 517. https://doi.org/10.1037/0003-066X.33.5.517.a First citation in articleCrossref, Google Scholar

  • Field, A. P., & Gillett, R. (2010). How to do a meta‐analysis. British Journal of Mathematical and Statistical Psychology, 63, 665–694. https://doi.org/10.1348/000711010X502733 First citation in articleCrossref, Google Scholar

  • Geevasinga, N., Loy, C. T., Menon, P., de Carvalho, M., Swash, M., Schrooten, M., … Noto, Y. I. (2016). Awaji criteria improves the diagnostic sensitivity in amyotrophic lateral sclerosis: A systematic review using individual patient data. Clinical Neurophysiology, 127, 2684–2691. https://doi.org/10.1016/j.clinph.2016.04.005 First citation in articleCrossref, Google Scholar

  • Gelman, A., & Geurts, H. M. (2017). The statistical crisis in science: How is it relevant to clinical neuropsychology? The Clinical Neuropsychologist, 31, 1000–1014. https://doi.org/10.1080/13854046.2016.1277557 First citation in articleCrossref, Google Scholar

  • Gelman, A., & Loken, E. (2014). The statistical crisis in science. American Scientist, 102, 460–465. https://doi.org/10.1511/2014.111.460 First citation in articleCrossref, Google Scholar

  • Goodale, M. A., Meenan, J. P., Bülthoff, H. H., Nicolle, D. A., Murphy, K. J., & Racicot, C. I. (1994). Separate neural pathways for the visual analysis of object shape in perception and prehension. Current Biology, 4, 604–610. https://doi.org/10.1016/S0960-9822(00)00132-9 First citation in articleCrossref, Google Scholar

  • Grant, D. A., & Berg, E. (1948). A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. Journal of Experimental Psychology, 38, 404–411. https://doi.org/10.1037/h0059831 First citation in articleCrossref, Google Scholar

  • Hagger, M. S., & Chatzisarantis, N. L. (2016). Commentary: Misguided effort with elusive implications, and sifting signal from noise with replication science. Frontiers in Psychology, 7, 621. https://doi.org/10.3389/fpsyg.2016.00621 First citation in articleCrossref, Google Scholar

  • Hagger, M. S., Chatzisarantis, N. L., Alberts, H., Anggono, C. O., Batailler, C., Birt, A. R., … Calvillo, D. P. (2016). A multilab preregistered replication of the ego-depletion effect. Perspectives on Psychological Science, 11, 546–573. https://doi.org/10.1177/1745691616652873 First citation in articleCrossref, Google Scholar

  • Hagger, M. S., Wood, C., Stiff, C., & Chatzisarantis, N. L. (2010). Ego depletion and the strength model of self-control: A meta-analysis. Psychological Bulletin, 136, 495–525. https://doi.org/10.1037/a0019486 First citation in articleCrossref, Google Scholar

  • Hammer, A., Vielhaber, S., Rodriguez-Fornells, A., Mohammadi, B., & Münte, T. F. (2011). A neurophysiological analysis of working memory in amyotrophic lateral sclerosis. Brain Research, 1421, 90–99. https://doi.org/10.1016/j.brainres.2011.09.010 First citation in articleCrossref, Google Scholar

  • Inzlicht, M., Gervais, W., & Berkman, E. (2015). Bias-correction techniques alone cannot determine whether ego depletion is different from zero: Commentary on Carter, Kofler, Forster, & McCullough, 2015. Retrieved from https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2659409 First citation in articleGoogle Scholar

  • Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine, 2, e124. https://doi.org/10.1371/journal.pmed.0020124 First citation in articleCrossref, Google Scholar

  • Keane, M. M., Gabrieli, J. D., Mapstone, H. C., Johnson, K. A., & Corkin, S. (1995). Double dissociation of memory capacities after bilateral occipital-lobe or medial temporal-lobe lesions. Brain, 118, 1129–1148. https://doi.org/10.1093/brain/118.5.1129 First citation in articleCrossref, Google Scholar

  • Kilani, M., Micallef, J., Soubrouillard, C., Rey-Lardiller, D., Demattei, C., Dib, M., … Blin, O. (2004). A longitudinal study of the evolution of cognitive function and affective state in patients with amyotrophic lateral sclerosis. Amyotrophic Lateral Sclerosis, 5, 46–54. https://doi.org/10.1080/14660820310017560 First citation in articleCrossref, Google Scholar

  • Klein, R. A., Ratliff, K. A., Vianello, M., Adams, R. B. Jr., Bahník, Š., Bernstein, M. J., … Cemalcilar, Z. (2014). Investigating variation in replicability. Social Psychology, 45, 142–152. https://doi.org/10.1027/1864-9335/a000178 First citation in articleLink, Google Scholar

  • Kopp, B., Lange, F., & Steinke, A. (2019). The reliability of the Wisconsin Card Sorting Test in clinical practice. Assessment. Advance online publication. https://doi.org/10.1177/1073191119866257 First citation in articleCrossref, Google Scholar

  • Kühberger, A., Fritz, A., & Scherndl, T. (2014). Publication bias in psychology: A diagnosis based on the correlation between effect size and sample size. PLoS One, 9, e105825. https://doi.org/10.1371/journal.pone.0105825 First citation in articleCrossref, Google Scholar

  • Lange, F., Brückner, C., Knebel, A., Seer, C., & Kopp, B. (2018). Executive dysfunction in Parkinson’s disease: A meta-analysis on the Wisconsin Card Sorting Test literature. Neuroscience & Biobehavioral Reviews, 93, 38–56. https://doi.org/10.1016/j.neubiorev.2018.06.014 First citation in articleCrossref, Google Scholar

  • Lange, F., & Eggert, F. (2014). Sweet delusion. Glucose drinks fail to counteract ego depletion. Appetite, 75, 54–63. https://doi.org/10.1016/j.appet.2013.12.020 First citation in articleCrossref, Google Scholar

  • Lange, F., Seer, C., Müller-Vahl, K., & Kopp, B. (2017). Cognitive flexibility and its electrophysiological correlates in Gilles de la Tourette syndrome. Developmental Cognitive Neuroscience, 27, 78–90. https://doi.org/10.1016/j.dcn.2017.08.008 First citation in articleCrossref, Google Scholar

  • Lange, F., Seer, C., Salchow, C., Dengler, R., Dressler, D., & Kopp, B. (2016). Meta-analytical and electrophysiological evidence for executive dysfunction in primary dystonia. Cortex, 82, 133–146. https://doi.org/10.1016/j.cortex.2016.05.018 First citation in articleCrossref, Google Scholar

  • Lange, F., Vogts, M.-B., Seer, C., Fürkötter, S., Abdulla, S., Dengler, R., Kopp, B., & Petri, S. (2016). Impaired set-shifting in amyotrophic lateral sclerosis: An event-related potential study of executive function. Neuropsychology, 30, 120–134. https://doi.org/10.1037/neu0000218 First citation in articleCrossref, Google Scholar

  • LeBel, E. P., & Paunonen, S. V. (2011). Sexy but often unreliable: The impact of unreliability on the replicability of experimental findings with implicit measures. Personality and Social Psychology Bulletin, 37, 570–583. https://doi.org/10.1177/0146167211400619 First citation in articleCrossref, Google Scholar

  • Li, C. S. R. (2004). Do schizophrenia patients make more perseverative than non-perseverative errors on the Wisconsin Card Sorting Test? A meta-analytic study. Psychiatry Research, 129, 179–190. https://doi.org/10.1016/j.psychres.2004.06.016 First citation in articleCrossref, Google Scholar

  • Light, G. A., Swerdlow, N. R., Thomas, M. L., Calkins, M. E., Green, M. F., Greenwood, T. A., … Pela, M. (2015). Validation of mismatch negativity and P3a for use in multi-site studies of schizophrenia: Characterization of demographic, clinical, cognitive, and functional correlates in COGS-2. Schizophrenia Research, 163, 63–72. https://doi.org/10.1016/j.schres.2014.09.042 First citation in articleCrossref, Google Scholar

  • Lurquin, J. H., Michaelson, L. E., Barker, J. E., Gustavson, D. E., Von Bastian, C. C., Carruth, N. P., & Miyake, A. (2016). No evidence of the ego-depletion effect across task characteristics and individual differences: A pre-registered study. PLoS One, 11, e0147770. https://doi.org/10.1371/journal.pone.0147770 First citation in articleCrossref, Google Scholar

  • Lurquin, J. H., & Miyake, A. (2017). Challenges to ego-depletion research go beyond the replication crisis: A need for tackling the conceptual crisis. Frontiers in Psychology, 8, 568. https://doi.org/10.3389/fpsyg.2017.00568 First citation in articleCrossref, Google Scholar

  • MacLean, E. L., Hare, B., Nunn, C. L., Addessi, E., Amici, F., Anderson, R. C., … Boogert, N. J. (2014). The evolution of self-control. Proceedings of the National Academy of Sciences, 111, E2140–E2148. https://doi.org/10.1073/pnas.1323533111 First citation in articleCrossref, Google Scholar

  • MacLeod, C. M. (1991). Half a century of research on the Stroop effect: An integrative review. Psychological Bulletin, 109, 163–203. https://doi.org/10.1037/0033-2909.109.2.163 First citation in articleCrossref, Google Scholar

  • McShane, B. B., Tackett, J. L., Bockenholt, U., & Gelman, A. (2019). Large scale replication projects in contemporary psychological research. The American Statistician, 73, 99–105. https://doi.org/10.1080/00031305.2018.1505655 First citation in articleCrossref, Google Scholar

  • Morey, R. D., Chambers, C. D., Etchells, P. J., Harris, C. R., Hoekstra, R., Lakens, D., … Vanpaemel, W. (2016). The Peer Reviewers’ Openness Initiative: Incentivizing open research practices through peer review. Royal Society Open Science, 3, 150547. https://doi.org/10.1098/rsos.150547 First citation in articleCrossref, Google Scholar

  • Munafò, M. R., Nosek, B. A., Bishop, D. V., Button, K. S., Chambers, C. D., du Sert, N. P., … Ioannidis, J. P. (2017). A manifesto for reproducible science. Nature Human Behaviour, 1, 0021. https://doi.org/10.1038/s41562-016-0021 First citation in articleCrossref, Google Scholar

  • Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, aac4716. https://doi.org/10.1126/science.aac4716 First citation in articleCrossref, Google Scholar

  • Palmieri, A., Abrahams, S., Sorarù, G., Mattiuzzi, L., D’Ascenzo, C., Pegoraro, E., & Angelini, C. (2009). Emotional lability in MND: Relationship to cognition and psychopathology and impact on caregivers. Journal of the Neurological Sciences, 278, 16–20. https://doi.org/10.1016/j.jns.2008.10.025 First citation in articleCrossref, Google Scholar

  • Raaphorst, J., de Visser, M., van Tol, M. J., Linssen, W. H., van der Kooi, A. J., de Haan, R. J., … Schmand, B. (2010). Cognitive dysfunction in lower motor neuron disease: Executive and memory deficits in progressive muscular atrophy. Journal of Neurology, Neurosurgery & Psychiatry, 82, 170–175. https://doi.org/10.1136/jnnp.2009.204446 First citation in articleCrossref, Google Scholar

  • Rosenthal, R. (1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86, 638–641. https://doi.org/10.1037/0033-2909.86.3.638 First citation in articleCrossref, Google Scholar

  • Rosenthal, R., & DiMatteo, M. R. (2001). Meta-analysis: Recent developments in quantitative methods for literature reviews. Annual Review of Psychology, 52, 59–82. https://doi.org/10.1146/annurev.psych.52.1.59 First citation in articleCrossref, Google Scholar

  • Seer, C., Lange, F., Georgiev, D., Jahanshahi, M., & Kopp, B. (2016). Event-related potentials and cognition in Parkinson’s disease: An integrative review. Neuroscience & Biobehavioral Reviews, 71, 691–714. https://doi.org/10.1016/j.neubiorev.2016.08.003 First citation in articleCrossref, Google Scholar

  • Sharpe, D. (1997). Of apples and oranges, file drawers and garbage: Why validity issues in meta-analysis will not go away. Clinical Psychology Review, 17, 881–901. https://doi.org/10.1016/S0272-7358(97)00056-1 First citation in articleCrossref, Google Scholar

  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359–1366. https://doi.org/10.1177/0956797611417632 First citation in articleCrossref, Google Scholar

  • Stamelou, M., Edwards, M. J., Hallett, M., & Bhatia, K. P. (2012). The non-motor syndrome of primary dystonia: Clinical and pathophysiological implications. Brain, 135, 1668–1681. https://doi.org/10.1093/brain/awr224 First citation in articleCrossref, Google Scholar

  • Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11, 702–712. https://doi.org/10.1177/1745691616658637 First citation in articleCrossref, Google Scholar

  • Sterling, T. D. (1959). Publication decisions and their possible effects on inferences drawn from tests of significance – or vice versa. Journal of the American Statistical Association, 54, 30–34. https://doi.org/10.2307/2282137 First citation in articleGoogle Scholar

  • Stevens, J. R. (2017). Replicability and reproducibility in comparative psychology. Frontiers in Psychology, 8, 862. https://doi.org/10.3389/fpsyg.2017.00862 First citation in articleCrossref, Google Scholar

  • Strauss, E., Sherman, E., & Spreen, O. (2006). A compendium of neuropsychological tests (3rd ed.). New York, NY: Oxford University Press. First citation in articleGoogle Scholar

  • Stuss, D. T., & Levine, B. (2002). Adult clinical neuropsychology: Lessons from studies of the frontal lobes. Annual Review of Psychology, 53, 401–433. https://doi.org/10.1146/annurev.psych.53.100901.135220 First citation in articleCrossref, Google Scholar

  • Turetsky, B. I., Dress, E. M., Braff, D. L., Calkins, M. E., Green, M. F., Greenwood, T. A., … Radant, A. D. (2015). The utility of P300 as a schizophrenia endophenotype and predictive biomarker: Clinical and socio-demographic modulators in COGS-2. Schizophrenia Research, 163, 53–62. https://doi.org/10.1016/j.schres.2014.09.024 First citation in articleCrossref, Google Scholar

  • Van Elk, M., Matzke, D., Gronau, Q., Guang, M., Vandekerckhove, J., & Wagenmakers, E. J. (2015). Meta-analyses are no substitute for registered replications: A skeptical perspective on religious priming. Frontiers in Psychology, 6, 1365. https://doi.org/10.3389/fpsyg.2015.01365 First citation in articleCrossref, Google Scholar

  • Vazire, S. (2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13, 411–417. https://doi.org/10.1177/1745691617751884 First citation in articleCrossref, Google Scholar

  • Wagenmakers, E. J., Wetzels, R., Borsboom, D., van der Maas, H. L., & Kievit, R. A. (2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7, 632–638. https://doi.org/10.1177/1745691612463078 First citation in articleCrossref, Google Scholar

  • Zwaan, R. A., Etz, A., Lucas, R. E., & Donnellan, M. B. (2018). Making replication mainstream. Behavioral and Brain Sciences, 41, e120. https://doi.org/10.1017/S0140525X17001972 First citation in articleCrossref, Google Scholar