The purpose of this study is to examine the reliability of the skin tone measures in the widely used American National Election Studies data collection (ANES 2016 Time Series). Low reliability in skin tone measurement can lead to false conclusions regarding theoretically important relationships. Consistent with previous reliability analyses based on data from the General Social Survey, we find that different interviewers agree on Black and Latinx respondent skin tone less than 20% of the time and inter-rater reliability coefficients are very low (< .3). We also exploit unique features of the ANES data that allow us to (1) assess intra-rater reliability using Krippendorff’s alpha and (2) compare observer skin tone judgments to respondent self-appraisals. We find that even for cases where the same interviewer judges the same Black or Latinx respondent 2 months later, interviewers agree with their earlier assessment less than 35% of the time—only modestly exceeding expectations based on chance alone. Furthermore, we find weak correlations between how interviewers remember Black and Latinx respondent skin tone and how respondents self-describe. Importantly, our analyses indicate that these data patterns persist regardless of whether or not interviewer race/ethnicity matches that of the respondent. Thus, our results provide little support for the claim that measurement reliability can be significantly improved through a policy of matching respondents to interviewers of the same race and ethnicity. We discuss the implications for future research on skin tone’s relationship with social attitudes and outcomes.

中文翻译：

---

同种族和跨种族肤色判断的可靠性

这项研究的目的是检验广泛使用的美国国家选举研究数据收集（ANES 2016时间序列）中肤色测量的可靠性。肤色测量的低可靠性可能导致有关理论上重要关系的错误结论。与之前基于“一般社会调查”中的数据进行的可靠性分析一致，我们发现不同的访问者都同意Black和Latinx受访者的肤色低于20％的时间，且评分者间的可靠性系数非常低（<.3）。我们还利用了ANES数据的独特功能，这些功能使我们能够（1）使用Krippendorff的alpha评估评估者内部可靠性，以及（2）将观察者的肤色判断与受访者的自我评估进行比较。我们发现，即使对于两个月后同一名访问者判断同一名黑人或拉丁裔受访者的情况，访问者也同意他们的早期评估少于35％的时间，仅凭偶然性就略高于预期。此外，我们发现访调员如何记住Black和Latinx受访者的肤色与受访者的自我描述之间存在弱相关性。重要的是，我们的分析表明，无论访调员的种族/民族是否与受访者的种族/种族相匹配，这些数据模式都将持续存在。因此，我们的结果几乎不支持这样的说法，即通过使受访者与相同种族和族裔的访调员相匹配的政策，可以显着提高测量的可靠性。