Elsevier

Intelligence

Volume 80, May–June 2020, 101440
Intelligence

Correspondence
A primer on assessing intelligence in laboratory studies

https://doi.org/10.1016/j.intell.2020.101440Get rights and content

Highlights

  • Examination of individual differences represents a departure from traditional experimental psychology investigations.

  • Numerous statistical pitfalls in assessing abilities in laboratory studies may lead to limited or erroneous conclusions.

  • Attention to a facet framework for intelligence assessment may yield more robust findings from laboratory studies.

  • In general, use of single, homogeneous-content intellectual ability tests is not recommended.

  • Tests should adequately sample the underlying content of the factor of interest.

  • Design and analysis dos and don’ts are provided for experimental studies where correlations with abilities is a goal.

Abstract

This paper is an attempt to provide a brief guide to major conceptual and statistical problems that are unique to the study of individual differences in intelligence and various intellectual abilities, in the context of laboratory experimental studies, and to suggest strategies to successfully navigate these problems. Such studies are generally designed so that the goal is to evaluate the relationships between individual differences in basic task performance or related markers on the one hand, and individual differences in intellectual abilities on the other hand. Issues discussed in this paper include: restriction-of-range in talent, method variance and facet theory; speed vs. power; regression to the mean; extreme-groups designs; difference scores; differences in correlations; significant vs. meaningful correlations; factor- pure tests; and criterion variables. A list of representative “do” and “don't” recommendations is provided to help guide the design and evaluation of laboratory studies.

Section snippets

Background

Ever since the groundbreaking work by Hunt and his colleagues (e.g., Hunt, Frost, and Lunneborg, 1973), numerous researchers have attempted to investigate the relationships between intellectual abilities on the one hand, and constructs from experimental cognitive psychology, on the other hand. However, there is a mismatch between the standard paradigm used in experimental psychology and the procedures that are optimal for investigating individual differences in intellectual abilities. Some of

Reliability and validity

The first difficulty in this area of research is that experimental and correlational orientations create a fundamental conflict. In the context of cognitive research, the experimental approach typically involves using stimuli that are either believed to be substantially “overlearned” by most or all study participants (e.g., letters, numbers, high-frequency words, simple polygons), or entirely novel stimuli (e.g., artificial grammars, random number sequences), such that either way, the

Validity

3. Do consider the validity of selected intelligence tests

In experimental cognitive psychology, there is ongoing debate about what particular laboratory tasks are suitable for assessing a particular construct (e.g., memory search speed, decision time, reaction time or movement time, verbal or spatial working memory). The “validity” of such tasks is established through discussion in the literature, leading to a set of relatively standardized procedures to establish that a particular set of task

Data are theory-laden

5. Do understand that data are theory-laden

Thomas Kuhn (1977), physicist-turned-philosopher of science, proposed the theme of “paradigm” to describe the amalgamation of assumptions and methods used within individual laboratories. Although there was a lively argument in the literature about whether psychology could be considered paradigmatic, it is clear that within the domain of intelligence research, there are multiple paradigms, loosely associated with the Spearman and Cattell-Horn-Carroll

Standardized intelligence tests

6. Don't use intelligence tests in a non-standard manner without norms and validation

The traditional ‘gold standard’ assessments of intelligence (e.g., Stanford-Binet, Wechsler) are considered to be standardized tests, in that all of the examiner's words, apparatus, and procedure, and the scoring rules are fixed. The same is true for college admissions tests, such as the SAT or ACT, or tests administered in the manner dictated by test manuals. However, there are numerous variations that have

Regression-to-the-mean

7. Don't use extreme-group designs

Restriction-of-range in talent

9. Do adjust correlations to account for restriction-of-range in talent

When a researcher chooses a sample that is restricted in range-of-talent – a typical phenomenon when the study participants are students from a moderate to highly selective college/university, correlations are expected to be attenuated than would be observed in the population-at-large. Corrections can be made to the correlations, on the basis of comparing the variances of the sample on the variables of interest to the

Differences between correlations

10. Don't use samples too small to detect differences in correlations

Significant vs. meaningful correlations

11. Don't confuse statistically significant with meaningful magnitude correlations

12. Do pre-specify expected correlations

Over the past few decades, scholarly organizations and journal editors have encouraged researchers to move beyond null hypothesis significance testing, in favor of reporting effect sizes (e.g., Wilkinson and Task Force on Statistical Inference, 1999). This movement is especially important for laboratory studies of experimental tasks and intellectual abilities, because it may

‘Method’ factors

As noted by Campbell and Fiske (1959), “Each test or task employed for measurement purposes is a trait-method unit, a union of a particular trait content with measurement procedures not specific to that content.” (p. 81). In the assessment of intellectual abilities, “method” can refer to a variety of different characteristics of a particular test, such as the kinds of question formats (multiple choice, open-ended, sentence completion), administration procedures (e.g., speed vs. power tests),

Content as overlapping ‘Method’ factor

13. Don't ignore common content as method variance

One of the reasonably well-replicated findings from research conducted under the heading of “elementary information process” correlates of intelligence (e.g., Ackerman, 1986; Kyllonen, 1985), is that common stimulus content (e.g., numbers as stimuli/test items) on both the task and ability assessments will lead to higher correlations than when there are different stimulus contents (e.g., a spatial stimulus task and a verbal ability test). This

Speed vs. power method issues

14. Do take account of speed vs. power of reference tests

As the information processing framework in experimental psychology grew out of information theory (e.g., see Atkinson & Shiffrin, 1967; Attneave, 1959; Shannon and Weaver, 1949), the two major dependent variables for such investigations were the speed and accuracy of responding, often with a major emphasis on speed.1

Approaches to assessing intellectual abilities

The following is a set of recommendations, and their underlying justifications, for assessing intellectual abilities in the context of laboratory experimental studies aimed at determining the relations between tasks and intellectual abilities. Examples are provided to underline these recommendations.

Contrasting a single ability measure with multiple ability measures

21. Don't assume that adequate assessment with multiple measures requires excessive administration time

Ackerman et al. (2000) administered a large set of intellectual ability tests, including the Raven's Advanced Progressive Matrices (APM) test, and a battery of 7 working memory (WM) ability tests to a group of 135 young adults, in order to determine the relationships between intelligence, perceptual speed, and working memory ability. The APM was administered in the manner specified by the

Conclusions

Many of the “don'ts” and too few of the “do's” can be found in the extant literature on the relationship between constructs from experimental cognitive psychology and intellectual ability. Although they are prominent in laboratory experimental studies that attempt to relate task performance to individual differences in intellectual abilities, they are pervasive in the wider literature beyond the study of intelligence. It is important to note that there are few, if any, “perfect” studies in the

References (92)

  • T.A. Salthouse et al.

    Contextual analysis of fluid intelligence

    Intelligence

    (2008)
  • R.J. Sternberg

    Haste makes waste versus a stitch in time? A reply to Vernon, Nador, and Kantor

    Intelligence

    (1986)
  • J.A. Van Dyke et al.

    Low working memory capacity is only spuriously related to poor reading comprehension

    Cognition

    (2014)
  • P.A. Vernon et al.

    Reaction times and speed-of-processing: Their relationship to timed and untimed measures of intelligence

    Intelligence

    (1985)
  • W.A. Wickelgren

    Speed-accuracy tradeoff and information processing dynamics

    Acta Psychologica

    (1977)
  • P.L. Ackerman

    Domain-specific knowledge as the “dark matter” of adult intelligence: gf/gc, personality and interest correlates

    Journal of Gerontology: Psychological Sciences

    (2000)
  • P.L. Ackerman

    Adult intelligence: The construct and the criterion problem

    Perspectives on Psychological Science

    (2017)
  • P.L. Ackerman et al.

    Integrating laboratory and field study for improving selection: Development of a battery for predicting air traffic controller success

    Journal of Applied Psychology

    (1993)
  • P.L. Ackerman et al.

    Individual differences in working memory within a nomological network of cognitive and perceptual speed abilities

    Journal of Experimental Psychology: General

    (2002)
  • P.L. Ackerman et al.

    Further explorations of perceptual speed abilities, in the context of assessment methods, cognitive abilities and individual differences during skill acquisition

    Journal of Experimental Psychology: Applied

    (2007)
  • A. Anastasi et al.

    Psychological testing

    (1997)
  • W. Arthur et al.

    Development of a short form for the Raven advanced progressive matrices test

    Educational and Psychological Measurement

    (1994)
  • W. Arthur et al.

    College-sample psychometric and normative data on a short form of the Raven advanced progressive matrices test

    Journal of Psychoeducational Assessment

    (1999)
  • R.C. Atkinson et al.

    Human memory: A proposed system and its control processes

  • F. Attneave

    Applications of information theory to psychology: A summary of basic concepts, methods, and results

    (1959)
  • W.B. Biker et al.

    Development of abbreviated nine-item forms of the Raven’s standard progressive matrices test

    Assessment

    (2012)
  • A. Binet et al.

    New methods for the diagnosis of the intellectual level of subnormals. L’Année Psychologique, 11, 191-244

  • J.O. Boone et al.

    The development of the ATC selection battery: A new procedure to make maximum use of available information when correcting correlations for restriction in range due to selection (FAA-AM-78-36)

    (1978)
  • H.R. Burke

    Raven’s progressive matrices: A review and critical evaluation

    The Journal of Genetic Psychology

    (1958)
  • D.T. Campbell et al.

    Convergent and discriminant validation by the multitrait-multimethod matrix

    Psychological Bulletin

    (1959)
  • J.B. Carroll

    The measurement of intelligence

  • R.B. Cattell

    Abilities: Their structure, growth and action. [revised and reprinted as Intelligence: Its structure, growth, and action]

    (1971-1987)
  • F.I.M. Craik et al.

    Alpha span: A measure of working memory

    Canadian Journal of Experimental Psychology

    (2018)
  • L.J. Cronbach

    Coefficient alpha and the internal structure of tests

    Psychometrika

    (1951)
  • L.J. Cronbach

    The two disciplines of scientific psychology

    American Psychologist

    (1957)
  • L.J. Cronbach

    Essentials of psychological testing

    (1960)
  • L.J. Cronbach et al.

    How we should measure “change” -- or should we?

    Psychological Bulletin

    (1970)
  • H. Ebbinghaus

    Über eine neue Methode zur Prüfung geistiger Fähigkeiten und ihre Anwendung bei Schulkindern. [On a new method for testing mental abilities and its use with school children.]

    Zeitschrift für Psychologie und Psysiologie der Sinnesorgane

    (1896–97)
  • E.E. Ghiselli et al.

    Measurement theory for the behavioral sciences

    (1981)
  • F.L. Goodenough et al.

    The relative potency of the nursery school and the statistical lab in boosting the IQ

    Journal of Educational Psychology

    (1940)
  • S.B. Green et al.

    Psychonomic Bulletin & Review

    (2016)
  • H. Gulliksen

    Theory of mental tests

    (1950)
  • L. Guttman

    An outline of some new methodology for social research

    The Public Opinion Quarterly

    (1954-1955)
  • G.R. Hancock et al.

    The reliability paradox in assessing structural relations within covariance structure models

    Educational and Psychological Measurement

    (2011)
  • C. Hedge et al.

    The reliability paradox: Why robust cognitive tasks do not produce reliable individual differences

    Behavioral Research

    (2018)
  • W.E. Hick

    On the rate of gain of information

    Quarterly Journal of Experimental Psychology

    (1952)
  • Cited by (0)

    View full text