Purpose:

This case study discusses Mathematica’s experience providing large-scale evaluation technical assistance (ETA) to 65 grantees across two cohorts of Teen Pregnancy Prevention (TPP) Program grants. The grantees were required to conduct rigorous evaluations with specific evaluation benchmarks. This case study provides an overview of the TPP grant program, the evaluation requirements, the ETA provider, and other key stakeholders and the ETA provided to the grantees. Finally, it discusses the successes, challenges, and lessons learned from the effort.

Conclusion:

One important lesson learned is that there are two related evaluation features, strong counterfactuals and insufficient target sample sizes, that funders should attend to prior to selecting awardees because they are not easy to change through ETA. In addition, if focused on particular outcomes (for TPP, the goal was to improve sexual behavior outcomes), the funder should prioritize studies with an opportunity to observe differences in these outcomes across conditions; several TPP grantees served young populations, and sexual behavior outcomes were not observed or were rare, limiting the opportunity to observe impacts. Unless funders are attentive to weaning out evaluations with critical limitations during the funding process, requiring grantees to conduct impact evaluations supported by ETA might unintentionally foster internally valid, yet underpowered studies that show nonsignificant program impacts. The TPP funder was able to overcome some of the limitations of the grantee evaluations by funding additional evidence-building activities, including federally led evaluations and a large meta-analysis of the effort, as part of a broader learning agenda.

中文翻译：

---

青少年怀孕预防评估技术援助：案例研究

目的：

本案例研究讨论了 Mathematica 向两个青少年怀孕预防 (TPP) 计划赠款队列中的 65 名受赠者提供大规模评估技术援助 (ETA) 的经验。受助者必须以特定的评估基准进行严格的评估。本案例研究概述了 TPP 赠款计划、评估要求、ETA 提供者和其他主要利益相关者以及提供给受赠者的 ETA。最后，它讨论了成功、挑战和从努力中吸取的教训。

结论：

一个重要的经验教训是，有两个相关的评估特征，强烈的反事实和目标样本量不足，资助者在选择获奖者之前应该注意，因为它们不容易通过 ETA 改变。此外，如果专注于特定结果（对于 TPP，目标是改善性行为结果），资助者应优先研究有机会观察这些结果在不同条件下的差异；一些 TPP 受赠者为年轻人提供服务，性行为结果未观察到或很少见，限制了观察影响的机会。除非资助者在资助过程中注意放弃具有严重限制的评估，否则要求受资助者进行 ETA 支持的影响评估可能会无意中培养内部有效性，然而，表明计划影响不显着的研究结果不足。作为更广泛的学习议程的一部分，TPP 资助者能够通过资助额外的证据构建活动来克服受资助者评估的一些限制，包括联邦政府主导的评估和对这项工作的大型荟萃分析。