Abstract

Individuals concerned with subgroup differences on standardized tests suggest replacing these tests with holistic evaluations of unstructured application materials, such as letters of recommendation (LORs), which they posit show less bias. We empirically investigate this proposition that LORs are bias-free, and argue that LORs might actually invite systematic, race and gender subgroup differences in the content and evaluation of LORs. We text analyzed over 37,000 LORs submitted on behalf of over 10,000 graduate school applicants. Results showed that LOR content does differ across applicants. Furthermore, we see some systematic gender, race, and gender-race intersection differences in LOR content. Content of LORs also systematically differed between degree programs (S.T.E.M. vs. non-S.T.E.M.) and degree sought (doctoral vs. masters). Finally, LOR content alone did not predict an appreciable amount of variance in offers of admission (the first barrier to increasing diversity and inclusion in graduate programs). Our results, combined with past research on LOR content bias, highlight concerns that LORs can be biased against marginalized groups. We conclude with suggestions for reducing potential bias in LOR and for increasing diversity in graduate programs.

摘要

关注标准化测试亚组差异的个人建议用对非结构化申请材料的整体评估代替这些测试，例如推荐信 (LOR)，他们认为这些材料的偏见较少。我们从经验上研究了 LOR 是无偏见的这一命题，并认为 LOR 实际上可能会在内容和评估 LOR。我们对代表 10,000 多名研究生院申请者提交的 37,000 多份 LOR 进行了文本分析。结果表明，申请人的 LOR 内容确实不同。此外，我们在 LOR 内容中看到了一些系统性的性别、种族和性别-种族交叉差异。LOR 的内容在学位课程（STEM 与非 STEM）和所寻求的学位（博士与硕士）之间也存在系统性差异。最后，仅 LOR 内容并不能预测录取通知书的显着差异（研究生课程增加多样性和包容性的第一个障碍）。我们的结果，结合过去对 LOR 内容偏见的研究，强调了 LOR 可能会对边缘化群体产生偏见的担忧。最后，我们提出了减少 LOR 潜在偏见和增加研究生课程多样性的建议。