Expert-coded datasets provide scholars with otherwise unavailable data on important concepts. However, expert coders vary in their reliability and scale perception, potentially resulting in substantial measurement error. These concerns are acute in expert coding of key concepts for peace research. Here I examine (1) the implications of these concerns for applied statistical analyses, and (2) the degree to which different modeling strategies ameliorate them. Specifically, I simulate expert-coded country-year data with different forms of error and then regress civil conflict onset on these data, using five different modeling strategies. Three of these strategies involve regressing conflict onset on point estimate aggregations of the simulated data: the mean and median over expert codings, and the posterior median from a latent variable model. The remaining two strategies incorporate measurement error from the latent variable model into the regression process by using multiple imputation and a structural equation model. Analyses indicate that expert-coded data are relatively robust: across simulations, almost all modeling strategies yield regression results roughly in line with the assumed true relationship between the expert-coded concept and outcome. However, the introduction of measurement error to expert-coded data generally results in attenuation of the estimated relationship between the concept and conflict onset. The level of attenuation varies across modeling strategies: a structural equation model is the most consistently robust estimation technique, while the median over expert codings and multiple imputation are the least robust.

中文翻译：

---

专家错误有多重要？定量和平研究的意义

专家编码的数据集为学者提供了有关重要概念的其他可用数据。然而，专家编码员的可靠性和规模感知各不相同，可能会导致大量的测量误差。这些问题在和平研究的关键概念的专家编码中尤为突出。在这里，我检查 (1) 这些问题对应用统计分析的影响，以及 (2) 不同建模策略改善它们的程度。具体来说，我用不同形式的错误模拟专家编码的国家年数据，然后使用五种不同的建模策略对这些数据进行内战开始的回归。其中三个策略涉及在模拟数据的点估计聚合上回归冲突开始：专家编码的均值和中值，以及来自潜在变量模型的后验中值。其余两种策略通过使用多重插补和结构方程模型将潜在变量模型的测量误差纳入回归过程。分析表明专家编码数据相对稳健：在模拟中，几乎所有建模策略产生的回归结果大致与专家编码概念和结果之间假设的真实关系一致。然而，将测量误差引入专家编码数据通常会导致概念和冲突开始之间的估计关系减弱。衰减水平因建模策略而异：结构方程模型是最一致的稳健估计技术，而专家编码和多重插补的中值最不稳健。