BACKGROUND Even a well-designed randomized control trial (RCT) study can produce ambiguous results. This article highlights a case in which full sample results from a large-scale RCT in the United Kingdom differ from results for a subsample of survey respondents. OBJECTIVES Our objective is to ascertain the source of the discrepancy in inferences across data sources and, in doing so, to highlight important threats to the reliability of the causal conclusions derived from even the strongest research designs. RESEARCH DESIGN The study analyzes administrative data to shed light on the source of the differences between the estimates. We explore the extent to which heterogeneous treatment impacts and survey nonresponse might explain these differences. We suggest checks which assess the external validity of survey measured impacts, which in turn provides an opportunity to test the effectiveness of different weighting schemes to remove bias. The subjects included 6,787 individuals who participated in a large-scale social policy experiment. RESULTS Our results were not definitive but suggest nonresponse bias is the main source of the inconsistent findings. CONCLUSIONS The results caution against overconfidence in drawing conclusions from RCTs and highlight the need for great care to be taken in data collection and analysis. Particularly, given the modest size of impacts expected in most RCTs, small discrepancies in data sources can alter the results. Survey data remain important as a source of information on outcomes not recorded in administrative data. However, linking survey and administrative data is strongly recommended whenever possible.

中文翻译：

---

使用行政数据探索英国就业保留和晋升示范中调查不答复的影响。


背景技术即使是精心设计的随机对照试验（RCT）研究也可能产生不明确的结果。本文重点介绍了英国大规模随机对照试验的完整样本结果与调查受访者子样本的结果不同的情况。目标 我们的目标是确定跨数​​据源推论差异的根源，并在此过程中强调即使是最强有力的研究设计所得出的因果结论的可靠性所面临的重要威胁。研究设计 该研究分析了行政数据，以揭示估计值之间差异的根源。我们探讨了异质性治疗影响和调查无反应可能在多大程度上解释这些差异。我们建议进行检查，评估调查测量影响的外部有效性，这反过来又提供了一个机会来测试不同加权方案的有效性，以消除偏差。研究对象包括参与大规模社会政策实验的 6,787 人。结果我们的结果并不明确，但表明无反应偏倚是不一致结果的主要原因。结论 结果告诫人们不要对从随机对照试验中得出的结论过于自信，并强调在数据收集和分析时需要非常小心。特别是，考虑到大多数随机对照试验预期的影响规模不大，数据来源的微小差异可能会改变结果。调查数据作为未记录在行政数据中的结果的信息来源仍然很重要。然而，强烈建议尽可能将调查和管理数据联系起来。