Many surveys ask respondents to consult financial records in order to improve data accuracy. However, the assumption that record use reduces measurement error has not been tested with a largescale comparison to administrative data. This paper compares interest, dividend, and rental income in the Survey of Income and Program Participation (SIPP) to administrative IRS 1040 tax data. In a novel estimation strategy, we use various measures of respondent motivation and precision to account for nonrandom selection. Our results show that record use is associated with reducing the discrepancy between survey and administrative data by approximately 21 to 43 percent. In terms of potential costs from encouraging record use, record users spend an extra 3.5 seconds for each asset question, on average, after controlling for their behavior in other parts of the SIPP interview. The extra time per question translates to a 2.2 percent increase in the total duration of the interview. Thus, while record use may be an effective tool for improving data accuracy, it may come at the cost of higher interviewer compensation and increased respondent burden. Financial questions on household surveys are invaluable tools for generating estimates of average income and wealth for the general population. However, many respondents have trouble answering these questions because they are unsure about the exact details of their finances at the time of the interview. Jonathan Eggleston is an economist in the Social, Economic, and Housing Statistics Division at the US Census Bureau, Suitland, MD, USA. Lori Reeder is a survey statistician in the Social, Economic, and Housing Statistics Division at the US Census Bureau, Suitland, MD, USA. The authors would like to thank seminar and conference participants at the US Census Bureau, the University of Virginia, and the 2016 International Conference on Questionnaire Design, Development, Evaluation, and Testing (QDET2) for their advice and comments. The views expressed in this article are those of the authors and not necessarily those of the US Census Bureau. *Address correspondence to Jonathan Eggleston, US Census Bureau, 4600 Silver Hill Road, Washington, DC 20233, USA; email: jonathan.s.eggleston@census.gov. Public Opinion Quarterly doi:10.1093/poq/nfy032 D ow naded rom http/academ ic.p.com /poq/advance-articleoi/10.1093/poq/nfy032/5142555 by gest on 10 Feruary 2019 For example, interviewers for the Survey of Income and Program Participation (SIPP) have left comments in the survey instrument indicating that the respondent’s answer was a guess or an approximation, or that another family member manages their finances. Comments like these suggest that such survey responses may be affected by measurement error, which could lead to biased results in statistical analyses based on these data. To mitigate measurement error, household surveys employ a variety of strategies to help respondents report accurate information. For example, respondents may be more aware of their finances from the past year at the time they are preparing their taxes, so conducting a financial survey around tax season may improve data quality. Another strategy is to encourage respondents to look at financial records (e.g., bank statements or tax returns) during the interview. Many government surveys use this strategy, such as the SIPP, the Consumer Expenditure Survey (CEX), and the American Housing Survey. This paper is the first large-scale comparison of survey to administrative data that evaluates whether financial record use is associated with improved data quality. This paper employs a novel estimation strategy wherein various measures of respondent motivation and precision are considered to account for nonrandom selection. Previous studies on record use have been correlational studies or experiments with smaller sample sizes, resulting in relatively low statistical power. Thus, although nonexperimental, this study overcomes a limitation in the literature on whether record use has a causal impact on data quality. Record Use and Data Quality A common belief in survey methodology is that encouraging respondents to look at financial records will improve data quality (Couper, Ofstedal, and Lee 2013). However, the literature on record use is relatively limited. Most of the work has consisted of nonexperimental technical papers that examine the effects of record use for a single survey. For example, Safir and Goldenberg (2008) found that record use was associated with higher total expenditures in the Consumer Expenditure Survey (CEX), and Edgar and Gonzalez (2009) found that data from record users required less data editing (such as imputing values). Kashihara and Wobus (2006) compared data provided by households to data provided by medical providers in the Medical Expenditure Panel Survey and found that record use by household respondents was associated with a closer match between the household data and medical provider data. Additionally, Eggleston (2015) and Wilson (2017) found that record users reported numbers that had been rounded to a lesser degree than nonrecord users (e.g., record users were more likely to say their interest income was $98.50 rather than $100). In these papers, the correlations between record use and measurement error may not be causal. Respondents who use financial records may generally be Eggleston and Reeder Page 2 of 21 D ow naded rom http/academ ic.p.com /poq/advance-articleoi/10.1093/poq/nfy032/5142555 by gest on 10 Feruary 2019 more aware of their finances or more engaged in the survey response process, and may therefore be more accurate in their survey responses regardless of whether they consult records. Additionally, not all households have records available. Previous studies have found that record availability varies for household expenditures (Geisen et al. 2016), vaccination history (McElligott and Darden 2010), pay slips (Laurie and Moon 2010), and electric bills (O’Brien 2010). Because of this, it is plausible that people who are more aware of their finances in general are more likely to keep financial records such as bank statements and tax documents. To estimate the causal effect, researchers have conducted experiments to determine whether prompting people to consult records is associated both with more people using records and with a reduction in measurement error. Couper, Ofstedal, and Lee (2013) considered the outcomes of two experiments for the Health and Retirement Study (HRS) Internet Survey designed to encourage respondents to use records for asset questions. The authors found that these experiments resulted in an increase in the proportion of respondents who used records but did not significantly increase the precision of the data, as measured by the amount of rounding in the answers and by item nonresponse rates. In another study, Maynes (1968) conducted an experiment on a sample of individuals with savings accounts and loans from the Census Federal Credit Union. In this mail survey, some respondents were randomly assigned to a condition that instructed them to consult records. Maynes found that respondents instructed to consult records reported more accurate account balances, as measured by the difference between survey and credit union data. Nonetheless, the aggregate distribution of reported assets was similar between the two experimental conditions. In other words, while nonrecord users gave less accurate values, these errors, in aggregate, did not decrease the accuracy of the estimated mean and standard deviation. Finally, in an experiment conducted by Murphy et al. (2015), encouraging record use in the Community Advantage Panel Survey was associated with an increase in the proportion of respondents who used records, but had no effect on the amount of rounding or the difference in sample means between experimental conditions. The results from Couper et al. (2013) and Murphy et al. (2015) suggest that record use does not improve data quality. However, there are caveats to these experimental studies. First, these studies do not compare survey responses to administrative data. Because of this, the indicators of data quality employed in these studies, such as rounding, may or may not be associated with actual measurement error. Second, the increase in the proportion of respondents who use records was relatively small (8 and 7 percentage points, respectively, for Couper et al. [2013] and Murphy et al. [2015]), and the sample size was approximately 2,000 for many of the analyzed asset and income items. Thus, although the authors found that experimental manipulations resulted in increased record use, the effect sizes and sample sizes may have been too small to detect significant changes in data quality. Record Use and Measurement Error Page 3 of 21 D ow naded rom http/academ ic.p.com /poq/advance-articleoi/10.1093/poq/nfy032/5142555 by gest on 10 Feruary 2019 Instead of using an experimental manipulation to address whether record use reduces measurement error, this paper employs a novel estimation strategy in which various measures of respondent motivation and precision are used to account for nonrandom selection. The results with this methodology show that record use is associated with a 21 to 43 percent decrease in measurement error. This approach does have limitations compared to an isolated experimental manipulation, since the proxy variables may be imperfect controls for unobserved respondent characteristics. However, in the absence of experimental variation, this approach provides a new way to leverage additional data on respondent accuracy when investigating measurement error in a survey. Additionally, this approach provides the substantial benefit of allowing for the use of an entire sample of a large national survey for examining record use effects as opposed to the small subset that would have been selected for an experimental study.

中文翻译：

---

鼓励金融资产使用记录会提高数据准确性吗？来自行政数据的证据

许多调查要求受访者查阅财务记录以提高数据准确性。然而，记录使用减少测量误差的假设尚未通过与行政数据的大规模比较进行测试。本文将收入和计划参与调查 (SIPP) 中的利息、股息和租金收入与 IRS 1040 行政税收数据进行了比较。在一种新颖的估计策略中，我们使用各种响应动机和精度的度量来解释非随机选择。我们的结果表明，记录使用与将调查和行政数据之间的差异减少约 21% 到 43% 相关。就鼓励记录使用的潜在成本而言，记录用户平均为每个资产问题多花 3.5 秒，在 SIPP 面试的其他部分控制了他们的行为之后。每个问题的额外时间意味着面试总时间增加了 2.2%。因此，虽然记录使用可能是提高数据准确性的有效工具，但代价可能是更高的访调员报酬和增加的受访者负担。住户调查中的财务问题是估算一般人群平均收入和财富的宝贵工具。然而，许多受访者在回答这些问题时遇到困难，因为他们在接受采访时不确定其财务状况的确切细节。乔纳森·埃格尔斯顿 (Jonathan Eggleston) 是美国马里兰州苏特兰市美国人口普查局社会、经济和住房统计部门的经济学家。Lori Reeder 是社会、经济、美国人口普查局住房统计司，马里兰州苏特兰。作者要感谢美国人口普查局、弗吉尼亚大学和 2016 年问卷设计、开发、评估和测试国际会议 (QDET2) 的研讨会和会议参与者的建议和评论。本文中表达的观点是作者的观点，不一定是美国人口普查局的观点。*与美国人口普查局 Jonathan Eggleston 的通信地址，4600 Silver Hill Road, Washington, DC 20233, USA；电子邮件：jonathan.s.eggleston@census.gov。公众意见季刊 doi:10.1093/poq/nfy032 D ow naded rom http/academ ic.p.com /poq/advance-articleoi/10.1093/poq/nfy032/5142555 by gest on 10 Feruary for example, 20142555 收入和计划参与调查 (SIPP) 的访问员在调查工具中留下了评论，表明受访者的答案是猜测或近似值，或者其他家庭成员管理他们的财务。诸如此类的评论表明，此类调查响应可能会受到测量误差的影响，这可能会导致基于这些数据的统计分析结果出现偏差。为了减少测量误差，住户调查采用了多种策略来帮助受访者报告准确的信息。例如，受访者在准备纳税时可能更了解他们过去一年的财务状况，因此在纳税季节进行财务调查可能会提高数据质量。另一种策略是鼓励受访者查看财务记录（例如，面谈期间的银行对账单或纳税申报表）。许多政府调查都使用这种策略，例如 SIPP、消费者支出调查 (CEX) 和美国住房调查。本文是第一次将调查与行政数据进行大规模比较，以评估财务记录的使用是否与数据质量的提高有关。本文采用了一种新颖的估计策略，其中考虑了响应者动机和精确度的各种度量来解释非随机选择。以前关于记录使用的研究是样本量较小的相关研究或实验，导致统计功效相对较低。因此，尽管不是实验性的，但这项研究克服了文献中关于记录使用是否对数据质量有因果影响的限制。记录使用和数据质量 调查方法中的一个普遍信念是，鼓励受访者查看财务记录将提高数据质量（Couper、Ofstedal 和 Lee，2013 年）。然而，关于记录使用的文献相对有限。大多数工作由非实验性技术论文组成，这些论文检查记录使用对单个调查的影响。例如，Safir 和 Goldenberg（2008 年）在消费者支出调查（CEX）中发现记录使用与较高的总支出相关，Edgar 和 Gonzalez（2009 年）发现来自记录用户的数据需要较少的数据编辑（例如估算值）。Kashihara 和 Wobus (2006) 在医疗支出小组调查中将家庭提供的数据与医疗提供者提供的数据进行了比较，发现家庭受访者的记录使用与家庭数据和医疗提供者数据之间的更紧密匹配相关。此外，Eggleston (2015) 和 Wilson (2017) 发现记录用户报告的数字比非记录用户报告的四舍五入程度要小（例如，记录用户更有可能说他们的利息收入是 98.50 美元而不是 100 美元）。在这些论文中，记录使用和测量误差之间的相关性可能不是因果关系。使用财务记录的受访者通常可能是 Eggleston 和 Reeder Page 2 of 21 D ow naded rom http/academ ic.p.com /poq/advance-articleoi/10。1093/poq/nfy032/5142555 by gest 在 2019 年 2 月 10 日更了解他们的财务状况或更多地参与调查响应过程，因此无论他们是否查阅记录，他们的调查响应都可能更准确。此外，并非所有家庭都有可用的记录。先前的研究发现，记录可用性因家庭支出（Geisen 等人，2016 年）、疫苗接种史（McElligott 和 Darden 2010 年）、工资单（Laurie 和 Moon 2010 年）和电费（O'Brien 2010 年）而异。正因为如此，一般来说，对自己的财务状况更加了解的人更有可能保留财务记录，例如银行对账单和税务文件。为了估计因果效应，研究人员进行了实验，以确定提示人们查阅记录是否与更多人使用记录和减少测量误差有关。Couper、Ofstedal 和 Lee（2013 年）考虑了健康与退休研究 (HRS) 互联网调查的两个实验的结果，该调查旨在鼓励受访者将记录用于资产问题。作者发现，这些实验导致使用记录的受访者比例增加，但并没有显着提高数据的精确度，这是通过答案中的舍入量和项目不响应率来衡量的。在另一项研究中，Maynes (1968) 对拥有储蓄账户和来自 Census Federal Credit Union 的贷款的个人样本进行了一项实验。在这次邮件调查中，一些受访者被随机分配到指示他们查阅记录的条件。梅恩斯发现，被指示查阅记录的受访者报告了更准确的账户余额，这是通过调查和信用合作社数据之间的差异衡量的。尽管如此，报告资产的总分布在两个实验条件之间是相似的。换句话说，虽然非记录用户给出的值不太准确，但这些错误总的来说并没有降低估计均值和标准差的准确性。最后，在 Murphy 等人进行的一项实验中。(2015)，在社区优势小组调查中鼓励使用记录与使用记录的受访者比例增加有关，但对舍入量或实验条件之间样本平均值的差异没有影响。Couper 等人的结果。(2013) 和墨菲等人。(2015) 建议记录使用不会提高数据质量。然而，这些实验研究有一些注意事项。首先，这些研究没有将调查响应与行政数据进行比较。因此，这些研究中采用的数据质量指标，例如四舍五入，可能与实际测量误差相关，也可能不相关。其次，使用记录的受访者比例增加相对较小（Couper et al. [2013] 和 Murphy et al. [2015] 分别为 8 和 7 个百分点），样本量约为 2,000许多分析的资产和收入项目。因此，尽管作者发现实验操作导致记录使用增加，但效应量和样本量可能太小而无法检测到数据质量的显着变化。Record Use and Measurement Error Page 3 of 21 Dow naded rom http/academ ic.p.com /poq/advance-articleoi/10.1093/poq/nfy032/5142555 by gest on 10 Feruary 2019 而不是使用实验操作来解决记录使用减少了测量误差，本文采用了一种新颖的估计策略，其中使用各种响应动机和精度的度量来解释非随机选择。这种方法的结果表明，记录使用与测量误差减少 21% 到 43% 相关。与孤立的实验操作相比，这种方法确实有局限性，因为代理变量可能是对未观察到的受访者特征的不完美控制。然而，在没有实验变异的情况下，这种方法提供了一种新方法，可以在调查调查中的测量误差时利用有关受访者准确性的额外数据。此外，这种方法提供了很大的好处，即允许使用大型全国调查的整个样本来检查记录使用效果，而不是为实验研究选择的小子集。