1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Organizational Psychology and Organizational Behavior
  4. Volume 10, 2023
  5. Article

Annual Review of Organizational Psychology and Organizational Behavior

Volume 10, 2023

Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research

Abstract

Some assert that noise (i.e., unwanted variance) is the most neglected yet most important source of error in judgment. We suggest that this problem was discovered nearly 100 years ago in the area of personnel selection and that a century of selection research has shown that noise can be demonstrably reduced by structuring the process (i.e., decomposing the component parts, agreeing on standards, and applying those standards consistently) and by aggregating judgments independently. Algorithms can aid significantly in this process but are often confused with methods that, in their current form, can substantially increase noise in judgment (e.g., artificial intelligence and machine learning).

Keyword(s): forecasting, judgment and decision making, noise, personnel selection, strategic decision making, unwanted variance
Loading

Article metrics loading...

/content/journals/10.1146/annurev-orgpsych-120920-050708
2023-01-23
2024-04-19
Loading full text...

Full text loading...

/deliver/fulltext/orgpsych/10/1/annurev-orgpsych-120920-050708.html?itemId=/content/journals/10.1146/annurev-orgpsych-120920-050708&mimeType=html&fmt=ahah

Literature Cited

  1. Arkes HR, González-Vallejo C, Bonham AJ, Kung YH, Bailey N 2010. Assessing the merits and faults of holistic and disaggregated judgments. J. Behav. Decis. Mak. 23:3250–70
     [Google Scholar]
  2. Arkes HR, Shaffer VA, Dawes RM. 2006. Comparing holistic and disaggregated ratings in the evaluation of scientific presentations. J. Behav. Decis. Mak. 19:5429–39
     [Google Scholar]
  3. Arkes HR, Shaffer VA, Medow MA. 2007. Patients derogate physicians who use a computer-assisted diagnostic aid. Med. Decis. Mak. 27:2189–202
     [Google Scholar]
  4. Armstrong JS. 2001. Combining forecasts. Principles of Forecasting: A Handbook for Researchers and Practitioners JS Armstrong 417–39. Norwell, MA: Kluwer Acad. Publ.
     [Google Scholar]
  5. Arthur W Jr., Day EA, McNelly TL, Edens PS. 2003. A meta-analysis of the criterion-related validity of assessment center dimensions. Pers. Psychol. 56:1125–53
     [Google Scholar]
  6. Barrick MR, Mount MK. 1993. Autonomy as a moderator of the relationships between the big five personality dimensions and job performance. J. Appl. Psychol. 78:111–18
     [Google Scholar]
  7. Barrick MR, Mount MK 2009. Select on conscientiousness and emotional stability. Handbook of Principles of Organizational Behavior EA Locke 19–40. West Sussex, UK: Wiley. , 2nd ed..
     [Google Scholar]
  8. Bosco FA, Aguinis H, Singh K, Field JG, Pierce CA. 2015. Correlational effect size benchmarks. J. Appl. Psychol. 100:2431–49
     [Google Scholar]
  9. Caliskan A, Bryson JJ, Narayanan A. 2017. Semantics derived automatically from language corpora contain humanlike biases. Science 356:183–86
     [Google Scholar]
  10. Chapman DS, Zweig DI. 2005. Developing a nomological network for interview structure: antecedents and consequences of the structured selection interview. Pers. Psychol. 58:3673–702
     [Google Scholar]
  11. Childers M, Highhouse S, Brooks ME. 2022. Apples, oranges, and ironing boards: comparative effect sizes influence lay impressions of test validity. Int. J. Sel. Assess. 30:2230–35
     [Google Scholar]
  12. Conway JM, Jako RA, Goodman DF. 1995. A meta-analysis of interrater and internal consistency reliability of selection interviews. J. Appl. Psychol. 80:5565–79
     [Google Scholar]
  13. Cortina JM, Goldstein NB, Payne SC, Davison HK, Gilliland SW. 2000. The incremental validity of interview scores over and above cognitive ability and conscientiousness scores. Pers. Psychol. 53:2325–51
     [Google Scholar]
  14. Cronbach LJ, Gleser GC, Nanda H, Rajaratnam N. 1972. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. New York: Wiley
  15. Dana J, Dawes R, Peterson N. 2013. Belief in the unstructured interview: the persistence of an illusion. Judgm. Decis. Mak. 8:5512–20
     [Google Scholar]
  16. Dane E, Pratt MG. 2007. Exploring intuition and its role in managerial decision making. Acad. Manag. Rev. 32:133–54
     [Google Scholar]
  17. Dawes RM, Corrigan B. 1974. Linear models in decision making. Psychol. Bull. 81:295–106
     [Google Scholar]
  18. Dawes RM, Faust D, Meehl PE. 1989. Clinical versus actuarial judgment. Science 243:48991668–74
     [Google Scholar]
  19. Dezecache G, Dockendorff M, Ferreiro DN, Deroy O, Bahrami B. 2022. Democratic forecast: Small groups predict the future better than individuals and crowds. J. Exp. Psychol. Appl. 28:3525–37
     [Google Scholar]
  20. Diab DL, Pui SY, Yankelevich M, Highhouse S. 2011. Lay perceptions of selection decision aids in US and non-US samples. Int. J. Sel. Assess. 19:2209–16
     [Google Scholar]
  21. Dietvorst BJ, Simmons JP, Massey C. 2018. Overcoming algorithm aversion: People will use imperfect algorithms if they can (even slightly) modify them. Manag. Sci. 64:31155–70
     [Google Scholar]
  22. Dilchert S, Ones DS. 2009. Assessment center dimensions: individual differences correlates and meta-analytic incremental validity. Int. J. Sel. Assess. 17:3254–70
     [Google Scholar]
  23. Doherty ME, Stewart TR, Holzworth RJ. 2021.. “ Noise” and social judgment theory: a commentary on Kahneman, Sibony, and Sunstein. Brunswik Soc. Newsl. 36:56–66
     [Google Scholar]
  24. Dougherty TW, Turban DB, Callender JC. 1994. Confirming first impressions in the employment interview: a field study of interviewer behavior. J. Appl. Psychol. 79:5659–65
     [Google Scholar]
  25. Einhorn HJ, Hogarth RM. 1978. Confidence in judgment: persistence of the illusion of validity. Psychol. Rev. 85:5395–416
     [Google Scholar]
  26. Eisenkraft N. 2013. Accurate by way of aggregation: Should you trust your intuition-based first impressions?. J. Exp. Soc. Psychol. 49:2277–79
     [Google Scholar]
  27. Eurich TL, Krause DE, Cigularov K, Thornton GC. 2009. Assessment centers: current practices in the United States. J. Bus. Psychol. 24:4387
     [Google Scholar]
  28. Evans JSB, Stanovich KE. 2013. Dual-process theories of higher cognition: advancing the debate. Perspect. Psychol. Sci. 8:3223–41
     [Google Scholar]
  29. Freyd M. 1925. The statistical viewpoint in vocational selection. J. Appl. Psychol. 9:349–56
     [Google Scholar]
  30. Frederick S. 2005. Cognitive reflection and decision making. J. Econ. Perspect. 19:425–42
     [Google Scholar]
  31. Gigerenzer G, Reb J, Luan S. 2022. Smart heuristics for individuals, teams, and organizations. Annu. Rev. Organ. Psychol. Organ. Behav. 9:171–98
     [Google Scholar]
  32. Gladwell M. 2005. Blink: The Power of Thinking Without Thinking New York: Little, Brown Co.
  33. Greguras GJ, Robie C, Schleicher DJ, Goff M III. 2003. A field study of the effects of rating purpose on the quality of multisource ratings. Pers. Psychol. 56:11–21
     [Google Scholar]
  34. Guion RM. 2011. Assessment, Measurement, and Prediction for Personnel Decisions Abington-on-Thames: Routledge
  35. Hammond KR, Hursch CJ, Todd FJ. 1964. Analyzing the components of clinical inference. Psychol. Rev. 71:6438–56
     [Google Scholar]
  36. Highhouse S. 1996. Context-dependent selection: the effects of decoy and phantom job candidates. Organ. Behav. Hum. Decis. Process. 65:168–76
     [Google Scholar]
  37. Highhouse S. 2008. Stubborn reliance on intuition and subjectivity in employee selection. Ind. Organ. Psychol. 1:333–42
     [Google Scholar]
  38. Highhouse S, Broadfoot A, Yugo JE, Devendorf SA. 2009. Examining corporate reputation judgments with generalizability theory. J. Appl. Psychol. 94:3782–89
     [Google Scholar]
  39. Highhouse S, Brooks ME 2017. Straight talk about selecting for upper management. The Oxford Handbook of Talent Management DG Collings, K Mellahi, WF Cascio 268–80. Oxford, UK: Oxford Univ. Press
     [Google Scholar]
  40. Highhouse S, Brooks ME, Nesnidol S, Sim S. 2017. Is a.51 validity coefficient good? Value sensitivity for interview validity. Int. J. Sel. Assess. 25:4383–89
     [Google Scholar]
  41. Highhouse S, Kostek JA. 2013. Holistic assessment for selection and placement. APA Handbook of Testing and Assessment in Psychology Washington, DC: Am. Psychol. Assoc.
     [Google Scholar]
  42. Hollenbeck GP. 2009. Executive selection—What's right…and what's wrong. Ind. Organ. Psychol. 2:130–43
     [Google Scholar]
  43. Huffcutt AI, Arthur W 1994. Hunter and Hunter 1984 revisited: interview validity for entry-level jobs. J. Appl. Psychol. 79:2184–90
     [Google Scholar]
  44. Huffcutt AI, Culbertson SS, Weyhrauch WS. 2014. Moving forward indirectly: reanalyzing the validity of employment interviews with indirect range restriction methodology. Int. J. Sel. Assess. 22:3297–309
     [Google Scholar]
  45. Hunter JE. 1980. Validity Generalization for 12,000 Jobs: An Application of Synthetic Validity and Validity Generalization to the General Aptitude Test Battery (GATB). Washington, DC: U.S. Dep. Labor, Empl. Serv.
  46. Jago AS, Laurin K. 2022. Assumptions about algorithms’ capacity for discrimination. Personal. Soc. Psychol. Bull. 48:4582–95
     [Google Scholar]
  47. Judge TA, Cable DM. 2004. The effect of physical height on workplace success and income: preliminary test of a theoretical model. J. Appl. Psychol. 89:3428–41
     [Google Scholar]
  48. Kahneman D, Sibony O, Sunstein CR. 2021. Noise: A Flaw in Human Judgment. New York: Little, Brown Spark
  49. Kahneman D, Slovic SP, Slovic P, Tversky A, eds. 1982. Judgment Under Uncertainty: Heuristics and Biases Cambridge, UK: Cambridge Univ. Press
  50. Kausel EE, Culbertson SS, Madrid HP. 2016. Overconfidence in personnel selection: when and why unstructured interview information can hurt hiring decisions. Organ. Behav. Hum. Decis. Process. 137:27–44
     [Google Scholar]
  51. Keren G. 2013. A tale of two systems: A scientific advance or a theoretical stone soup? Commentary on Evans & Stanovich 2013. Perspect. Psychol. Sci. 8:3257–62
     [Google Scholar]
  52. Klein GA, Orasanu J, Calderwood R, Zsambok CE, eds. 1993. Decision Making in Action: Models and Methods. Norwood, NJ: Ablex Publ. Corp.
  53. Kosinski M, Wang Y, Lakkaraju H, Leskovec J. 2016. Mining big data to extract patterns and predict real-life outcomes. Psychol. Methods 21:4493–506
     [Google Scholar]
  54. Kraiger K, Teachout MS. 1990. Generalizability theory as construct-related evidence of the validity of job performance ratings. Hum. Perform. 3:119–35
     [Google Scholar]
  55. Krueger JI, Funder DC. 2004. Towards a balanced social psychology: causes, consequences, and cures for the problem-seeking approach to social behavior and cognition. Behav. Brain Sci. 27:3313–27
     [Google Scholar]
  56. Kuncel NR, Highhouse S. 2011. Complex predictions and assessor mystique. Ind. Organ. Psychol. 4:302–306
     [Google Scholar]
  57. Kuncel NR, Klieger DM, Connelly BS, Ones DS. 2013. Mechanical versus clinical data combination in selection and admissions decisions: a meta-analysis. J. Appl. Psychol. 98:1060–72
     [Google Scholar]
  58. Lasky JJ, Hover GL, Smith PA, Bostian DW, Duffendack SC, Nord CL. 1959. Post-hospital adjustment as predicted by psychiatric patients and by their staff. J. Consult. Psychol. 23:3213–18
     [Google Scholar]
  59. Lievens F, De Paepe A. 2004. An empirical investigation of interviewer-related factors that discourage the use of high structure interviews. J. Organ. Behav. 25:29–46
     [Google Scholar]
  60. Lievens F, Highhouse S, De Corte W. 2005. The importance of traits and abilities in supervisors' hirability decisions as a function of method of assessment. J. Occup. Organ. Psychol. 78:3453–70
     [Google Scholar]
  61. Lipshitz R, Klein G, Orasanu J, Salas E. 2001. Taking stock of naturalistic decision making. J. Behav. Decis. Mak. 14:5331–52
     [Google Scholar]
  62. Logg JM. 2019. Using algorithms to understand the biases in your organization. Harvard Business Review Aug. 9. https://hbr.org/2019/08/using-algorithms-to-understand-the-biases-in-your-organization
    [Google Scholar]
  63. Marlowe CM, Schneider SL, Nelson CE. 1996. Gender and attractiveness biases in hiring decisions: Are more experienced managers less biased?. J. Appl. Psychol. 81:111–21
     [Google Scholar]
  64. Meehl PE. 1954. Clinical Versus Statistical Prediction: A Theoretical Analysis and a Review of the Evidence. Minneapolis: Univ. Minn.
  65. Morewedge CK, Kahneman D. 2010. Associative processes in intuitive judgment. Trends Cogn. Sci. 14:10435–40
     [Google Scholar]
  66. Morris SB, Daisley RL, Wheeler M, Boyer P. 2015. A meta-analysis of the relationship between individual assessments and job performance. J. Appl. Psychol. 100:15–20
     [Google Scholar]
  67. Narayanan A. 2019. How to recognize AI snake oil Presented at the Arthur Miller Lecture on Science and Ethics Cambridge, MA: April 22. https://www.cs.princeton.edu/∼arvindn/talks/MIT-STS-AI-snakeoil.pdf
  68. Neumann M, Niessen ASM, Meijer RR. 2021. Implementing evidence-based assessment and selection in organizations: a review and an agenda for future research. Organ. Psychol. Rev. 11:3205–39
     [Google Scholar]
  69. Nisbett RE, Ross L. 1980. Human Inference: Strategies and Shortcomings of Social Judgment. Hoboken, NJ: Prentice Hall
  70. Nolan KP, Carter NT, Dalal DK. 2016. Threat of technological unemployment: Are hiring managers discounted for using standardized employee selection practices?. Pers. Assess. Decis. 2:130–47
     [Google Scholar]
  71. Nolan KP, Highhouse S. 2014. Need for autonomy and resistance to standardized employee selection practices. Hum. Perform. 27:4328–46
     [Google Scholar]
  72. Ones DS, Dilchert S. 2009. How special are executives? How special should executive selection be? Observations and recommendations. Ind. Organ. Psychol. 2:163–70
     [Google Scholar]
  73. Pennycook G, De Neys W, Evans JSB, Stanovich KE, Thompson VA. 2018. The mythical dual-process typology. Trends Cogn. Sci. 22:8667–68
     [Google Scholar]
  74. Prien EP, Schippmann JS, Prien KO. 2003. Individual Assessment: As Practiced in Industry and Consulting. Mahwah, NJ: Lawrence Erlbaum Assoc.
  75. Pulakos ED, Schmitt N, Whitney D, Smith M 1996. Individual differences in interviewer ratings: the impact of standardization, consensus discussion, and sampling error on the validity of a structured interview. Pers. Psychol. 49:185–102
     [Google Scholar]
  76. Raiffa H. 1968. Decision Analysis: Introductory Lectures on Choices Under Uncertainty. Boston: Addison-Wesley
  77. Reagan-Cirincione PA. 1994. Improving the accuracy of group judgment: a process intervention combining group facilitation, social judgment analysis, and information technology. Organ. Behav. Hum. Decis. Process. 58:246–70
     [Google Scholar]
  78. Roose JE, Doherty M. 1976. Judgment theory applied to the selection of life insurance salesmen. Organ. Behav. Hum. Decis. Process. 22:193–15
     [Google Scholar]
  79. Ruscio J. 2003. Holistic judgment in clinical practice. Sci. Rev. Mental Health Pract. 2:38–48
     [Google Scholar]
  80. Sackett PR, Shewach OR, Keiser HN. 2017. Assessment centers versus cognitive ability tests: challenging the conventional wisdom on criterion-related validity. J. Appl. Psychol. 102:101435–47
     [Google Scholar]
  81. Sackett PR, Zhang C, Berry CM, Lievens F. 2022. Revisiting meta-analytic estimates of validity in personnel selection: addressing systematic overcorrection for restriction of range. J. Appl. Psychol. 107:112040–68
     [Google Scholar]
  82. Salas E, Rosen MA, DiazGranados D. 2010. Expertise-based intuition and decision making in organizations. J. Manag. 36:4941–73
     [Google Scholar]
  83. Salsburg D. 2001. The Lady Tasting Tea: How Statistics Revolutionized Science in the Twentieth Century New York: Macmillan
  84. Sarbin TL. 1943. A contribution to the study of actuarial and individual methods of prediction. Am. J. Sociol. 48:598–602
     [Google Scholar]
  85. Silzer R, Jeanneret R. 2011. Individual psychological assessment: a practice and science in search of common ground. Ind. Organ. Psychol. 4:3270–96
     [Google Scholar]
  86. Stewart TR, Roebber PJ, Bosart LF. 1997. The importance of the task in analyzing expert judgment. Organ. Behav. Hum. Decis. Process. 69:3205–19
     [Google Scholar]
  87. Tippins NT, Oswald FL, McPhail SM. 2021. Scientific, legal, and ethical concerns about AI-based personnel selection tools: a call to action. Pers. Assess. Decis. 7:21
     [Google Scholar]
  88. Van der Zee KI, Bakker AB, Bakker P. 2002. Why are structured interviews so rarely used in personnel selection?. J. Appl. Psychol. 87:1176–84
     [Google Scholar]
  89. Viteles MS. 1925. The clinical viewpoint in vocational selection. J. Appl. Psychol. 9:131–38
     [Google Scholar]
  90. Wang LY, Highhouse S, Brooks ME. 2022. Culture versus other sources of variance in risk and benefit perceptions: a comparison of Japan and the United States. J. Behav. Decis. Mak. 35:5e2277
     [Google Scholar]
  91. Weiss DJ, Shanteau J. 2021. The futility of decision making research. Stud. Hist. Philos. Sci. A. 90:10–14
     [Google Scholar]
  92. Wilson TD, Lisle DJ, Schooler JW, Hodges SD, Klaaren KJ, LaFleur SJ. 1993. Introspecting about reasons can reduce post-choice satisfaction. Personal. Soc. Psychol. Bull. 19:3331–39
     [Google Scholar]
  93. Wilson TD, Schooler JW. 1991. Thinking too much: introspection can reduce the quality of preferences and decisions. J. Pers. Soc. Psychol. 60:2181–92
     [Google Scholar]
  94. Yu MC, Kuncel NR. 2022. Testing the value of expert insight: comparing local versus general expert judgment models. Int. J. Sel. Assess. 30:2202–15
     [Google Scholar]
  95. Zhang DC, Highhouse S, Brooks ME, Zhang Y. 2018. Communicating the validity of structured job interviews with graphical visual aids. Int. J. Sel. Assess. 26:2–493–108
     [Google Scholar]
/content/journals/10.1146/annurev-orgpsych-120920-050708
Loading
Improving Workplace Judgments by Reducing Noise: Lessons Learned from a Century of Selection Research
Annual Review of Organizational Psychology and Organizational Behavior 10, 519 (2023); https://doi.org/10.1146/annurev-orgpsych-120920-050708
/content/journals/10.1146/annurev-orgpsych-120920-050708
/content/journals/10.1146/annurev-orgpsych-120920-050708
Loading

Data & Media loading...

  • Article Type: Review Article

Most Cited Most Cited RSS feed

This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error