1932
0
  1. Home
  2. A-Z Publications
  3. Annual Review of Criminology
  4. Volume 2, 2019
  5. Article

Annual Review of Criminology

Volume 2, 2019

The Real Gold Standard: Measuring Counterfactual Worlds That Matter Most to Social Science and Policy

Abstract

The randomized experiment has achieved the status of the gold standard for estimating causal effects in criminology and the other social sciences. Although causal identification is indeed important and observational data present numerous challenges to causal inference, we argue that conflating causality with the method used to identify it leads to a cognitive narrowing that diverts attention from what ultimately matters most—the difference between counterfactual worlds that emerge as a consequence of their being subjected to different treatment regimes applied to all eligible population members over a sustained period of time. To address this system-level and long-term challenge, we develop an analytic framework for integrating causality and policy inference that accepts the mandate of causal rigor but is conceptually rather than methodologically driven. We then apply our framework to two substantive areas that have generated high-visibility experimental research and that have considerable policy influence: () hot-spots policing and () the use of housing vouchers to reduce concentrated disadvantage and thereby crime. After reviewing the research in these two areas in light of our framework, we propose a research path forward and conclude with implications for the interplay of theory, data, and causal understanding in criminology and other social sciences.

Keyword(s): causality, counterfactuals, experiments, hot-spots policing, housing vouchers, policy
Loading

Article metrics loading...

/content/journals/10.1146/annurev-criminol-011518-024838
2019-01-13
2024-04-19
Loading full text...

Full text loading...

/deliver/fulltext/criminol/2/1/annurev-criminol-011518-024838.html?itemId=/content/journals/10.1146/annurev-criminol-011518-024838&mimeType=html&fmt=ahah

Literature Cited

  1. Angrist JD, Pischke J-S 2010. The credibility revolution in empirical economics: how better research design is taking the con out of econometrics. J. Econ. Perspect. 24:3–30
     [Google Scholar]
  2. Apel RJ, Sweeten G 2010. Propensity score matching in criminology and criminal justice. Handbook of Quantitative Criminology A Piquero, D Weisburd 543–62 New York: Springer
     [Google Scholar]
  3. Bayley D 1994. Police for the Future Oxford, UK: Oxford Univ. Press
  4. Berk R 2005. Randomized experiments as the bronze standard. J. Exp. Crim. 1:417–33
     [Google Scholar]
  5. Blattman C, Green DP, Ortega D, Tobon S 2018. Place based interventions at scale: the direct and spillover effects of policing and city services on crime SSRN Work. Pap. 3050823. https://ssrn.com/abstract=3050823
    [Google Scholar]
  6. Blau PM 1994. Structural Contexts of Opportunities Chicago: Univ. Chicago Press
  7. Blumstein A, Wallman J 2005. The Crime Drop in America New York: Cambridge
  8. Bothwell LE, Greene JA, Podolsky SH, Jones DS 2016. Assessing the gold standard: lessons from the history of RCTs. N. Engl. J. Med. 374:2175–81
     [Google Scholar]
  9. Braga AA, Papachristos AV, Hureau DM 2012. The effects of hot spots policing on crime. Campbell Syst. Rev. 8:1–97
     [Google Scholar]
  10. Braga AA, Papachristos AV, Hureau DM 2014. The effects of hot spots policing on crime: an updated systematic review and meta-analysis. Justice Q 31:633–63
     [Google Scholar]
  11. Branas CC, South E, Kondo MC, Hohl BC, Bourgois P et al. 2018. Citywide cluster randomized trial to restore blighted vacant land and its effects on violence, crime, and fear. PNAS 115:2946–51
     [Google Scholar]
  12. Briggs XDS, Popkin SJ, Goering J 2010. Moving to Opportunity: The Story of an American Experiment to Fight Ghetto Poverty New York: Oxford Univ. Press
     [Google Scholar]
  13. Cartwright N 2007. Are RCTs the gold standard. ? Biosocieties 2:11–20
     [Google Scholar]
  14. Chetty R, Hendren N, Katz LF 2016. The long-term effects of exposure to better neighborhoods: new evidence from the moving to opportunity experiment. Am. Econ. Rev. 106:855–902
     [Google Scholar]
  15. Clampet-Lundquist S, Edin K, Kling JR, Duncan GJ 2011. Moving at-risk teenagers out of high-risk neighborhoods: why girls fare better than boys. Am. J. Sociol. 116:1154–89
     [Google Scholar]
  16. Clampet-Lundquist S, Massey DS 2008. Neighborhood effects on economic self-sufficiency: a reconsideration of the moving to opportunity experiment. Am. J. Sociol. 114:107–43
     [Google Scholar]
  17. Cook PJ, MacDonald J 2011. Public safety through private action: an economic assessment of BIDs. Econ. J. 552:445–62
     [Google Scholar]
  18. DeAngelo G, Hansen B 2008. Life and Death in the Fast Lane: Police Enforcement and Roadway Safety Santa Barbara, CA: Univ. Calif.
  19. Deaton A, Cartwright N 2018. Understanding and misunderstanding randomized controlled trials. Soc. Sci. Med. 210:2–21
     [Google Scholar]
  20. Di Tella R, Schargrodsky E 2004. Do police reduce crime? Estimates using the allocation of police forces after a terrorist attack. Am. Econ. Rev. 94:115–33
     [Google Scholar]
  21. Draca M, Machin S, Witt R 2008. Panic on the streets of London: police, crime, and the July 2005 terror attacks IZA Work. Pap. 3410
    [Google Scholar]
  22. Durlauf S, Fu C, Navarro S 2012. Assumptions matter: model uncertainty and the deterrent effect of capital punishment. Am. Econ. Rev. 102:487–92
     [Google Scholar]
  23. Durlauf S, Fu C, Navarro S 2013. Capital punishment and deterrence: understanding disparate results. J. Quant. Criminol. 29:103–21
     [Google Scholar]
  24. Ellen IG, Lens MC, O'Regan K 2012. American murder mystery revisited: Do housing voucher households cause crime. ? Hous. Policy Debate 22:551–72
     [Google Scholar]
  25. Epple D, Romano R 2010. Peer effects in education: a survey of the theory and evidence. Handbook of Social Economics J Benhabib, M Jackson, A Bisin 1053–164 North Holland, Neth.: Elsevier
     [Google Scholar]
  26. Evans WN, Owens EG 2007. Cops and crime. J. Public Econ. 91:181–201
     [Google Scholar]
  27. Galster GC 1990. White flight from racially integrated neighborhoods in the 1970s: the Cleveland experience. Urban Stud 27:385–99
     [Google Scholar]
  28. Galster GC 2002. An economic efficiency analysis of deconcentrating poverty populations. J. Hous. Econ. 11:303–29
     [Google Scholar]
  29. Galster GC 2003. MTO's impact on sending and receiving neighborhoods. Choosing a Better Life? Evaluating the Moving to Opportunity Social Experiment J Goering, J Feins 365–82 Washington, DC: Urban Inst. Press
     [Google Scholar]
  30. Galster GC 2005. Consequences from the redistribution of urban poverty during the 1990s: a cautionary tale. Econ. Dev. Q. 19:119–25
     [Google Scholar]
  31. Galster GC 2014. Nonlinear and threshold aspects of neighborhood effects. Social Contexts and Social Mechanisms J Friedrichs, A Nonnenmacher 117–33 Wiesbaden, Ger.: Springer
     [Google Scholar]
  32. Galster GC 2019. Making Our Neighborhoods, Making Our Selves Chicago: Univ. Chicago Press In press
    [Google Scholar]
  33. Galster GC, Cutsinger JM, Malega R 2008. The costs of concentrated poverty: neighborhood property markets and the dynamics of decline. Revisiting Rental Housing: Policies, Programs, and Priorities NP Retsinas, ES Belsky 93–143 Washington, DC: Brookings
     [Google Scholar]
  34. Graif C 2015. Delinquency and gender moderation in the moving to opportunity intervention: the role of extended neighborhoods. Criminology 53:366–98
     [Google Scholar]
  35. Groff ER, Johnson SD, Thornton A 2018. State of the art in agent-based modeling of urban crime: an overview. J. Quant. Criminol. https://doi.org/10.1007/s10940-018-9376-y
    [Crossref]
  36. Heaton P 2010. Understanding the effects of antiprofiling policies. J. Law Econ. 53:29–64
     [Google Scholar]
  37. Heckman JJ 2005. The scientific model of causality. Sociol. Methodol. 35:1–97
     [Google Scholar]
  38. Heckman JJ, Vytlacil E 2007. Econometric evaluation of social programs, part I: causal models, structural models and econometric policy evaluation. Handbook of Econometrics JJ Heckman, E Leamer 4779–874 North Holland, Neth.: Elsevier
     [Google Scholar]
  39. Hendey L, Galster GC, Popkin SJ, Hayes C 2016. Housing choice voucher holders and neighborhood crime: a dynamic panel analysis from Chicago. Urban Aff. Rev. 52:471–500
     [Google Scholar]
  40. Hirsch A 1983. Making the Second Ghetto: Race and Housing in Chicago 1940–1960 Chicago: Univ. Chicago Press
     [Google Scholar]
  41. Holland P 1986. Statistics and causal inference. J. Am. Stat. Assoc. 81:945–70
     [Google Scholar]
  42. Hong G 2015. Causality in a Social World: Moderation, Mediation and Spill-Over New York: Wiley
  43. Imbens GW 2010. Better late than nothing: some comments on Deaton 2009 and Heckman and Urzua 2009. J. Econ. Lit. 48:399–423
     [Google Scholar]
  44. Klick J, Tabarrok A 2005. Using terror alert levels to estimate the effect of police on crime. J. Law Econ. 48:267–79
     [Google Scholar]
  45. Kling JR, Ludwig J, Katz LF 2005. Neighborhood effects on crime for female and male youth: evidence from a randomized housing voucher experiment. Q. J. Econ. 120:87–130
     [Google Scholar]
  46. Kondo MC, Andreyeva E, South EC, MacDonald JM, Branas CC 2018. Neighborhood interventions to reduce violence. Annu. Rev. Public Health 39:253–71
     [Google Scholar]
  47. Kyle SH 2018. The persistence of white flight in middle-class suburbia. Soc. Sci. Res. 72:38–52
     [Google Scholar]
  48. Legewie J, Schaeffer M 2016. Contested boundaries: explaining where ethno-racial diversity provokes neighborhood conflict. Am. J. Sociol. 122:125–61
     [Google Scholar]
  49. Leonhardt D, Cox A, Miller CC 2015. Change of address offers a pathway out of poverty. New York Times May 4, p. A1
  50. Levitt SD 1997. Using electoral cycles in police hiring to estimate the effect of police on crime. Am. Econ. Rev. 87:270–90
     [Google Scholar]
  51. Levitt SD 2004. Understanding why crime fell in the 1990s: four factors that explain the decline and six that do not. J. Econ. Perspect. 18:163–90
     [Google Scholar]
  52. Logan JR, Stults BJ 2011. The persistence of segregation in the metropolis: new findings from the 2010 Census Proj. US2010, Brown Univ. Providence, RI:
     [Google Scholar]
  53. Ludwig J, Duncan GJ, Gennetian LA, Katz LF, Kessler RC et al. 2012. Neighborhood effects on the long-term well-being of low-income adults. Science 337:1505–10
     [Google Scholar]
  54. Ludwig J, Hirschfield P, Duncan GJ 2001. Urban poverty and juvenile crime: evidence from a randomized housing-mobility experiment. Q. J. Econ. 116:665–79
     [Google Scholar]
  55. Ludwig J, Liebman JB, Kling JR, Duncan GJ, Katz LF et al. 2008. What can we learn about neighborhood effects from the moving to opportunity experiment? A comment on Clampet-Lundquist and Massey. Am. J. Sociol. 114:144–88
     [Google Scholar]
  56. Manski CF 2013. Public Policy in an Uncertain World Cambridge, MA: Harvard Univ. Press
  57. Massey DS, Denton N 1993. American Apartheid: Segregation and the Making of the Underclass Cambridge, MA: Harvard Univ. Press
  58. Mast BD, Wilson RE 2013. Housing choice vouchers and crime in Charlotte, NC. Hous. Policy Debate 23:559–96
     [Google Scholar]
  59. McCrary J 2002. Using electoral cycles in police hiring to estimate the effect of police on crime: comment. Am. Econ. Rev. 92:1236–43
     [Google Scholar]
  60. Morgan S, Winship C 2014. Counterfactuals and Causal Inference: Methods and Principles for Social Research New York: Cambridge Univ. Press. , 2nd ed..
  61. Mowat R, Subramanian SV, Kawachi I 2018. Randomized controlled trials and evidence-based policy: a multidisciplinary dialogue. Soc. Sci. Med. 210:1
     [Google Scholar]
  62. Nagin DS 2013. Deterrence in the 21st century: a review of the evidence. Crime and Justice: An Annual Review of Research M Tonry 199–262 Chicago: Univ. Chicago Press
     [Google Scholar]
  63. Natl. Acad. Sci. 2018. Proactive Policing: Effects on Crime and Communities Washington, DC: Natl. Acad. Press
  64. O'Brien D 2018. The Urban Commons: How Data and Technology Can Rebuild Our Communities Cambridge, MA: Harvard Univ. Press
  65. O'Brien DT, Sampson RJ, Winship C 2015. Ecometrics in the age of big data: measuring and assessing “broken windows” using large-scale administrative records. Sociol. Methodol. 45:101–47
     [Google Scholar]
  66. Pattillo ME 2007. Black on the Block: The Politics of Race and Class in the City Chicago: Univ. Chicago Press
     [Google Scholar]
  67. Payton CS, Tigh R 2015. Democracy in action?: NIMBY as impediment to equitable affordable housing siting. Hous. Stud. 30:749–69
     [Google Scholar]
  68. Pearl J, Mackenzie D 2018. The Book of Why: The New Science of Cause and Effect New York: Basic Books
  69. Popkin SJ, Rich MJ, Hendey L, Hayes C, Parilla J, Galster GC 2012. Public housing transformation and crime: making the case for responsible relocation. Cityscape J. Policy Dev. Res. 14:137–60
     [Google Scholar]
  70. Poutvaara P, Priks M 2006. Hooliganism in the Shadow of a Terrorist Attack and the Tsunami: Do Police Reduce Group Violence? Helsinki, Finl.: Univ. Helsinki
     [Google Scholar]
  71. Rosin H 2008. American murder mystery. The Atlantic July/August. https://www.theatlantic.com/magazine/archive/2008/07/american-murder-mystery/306872/
  72. Rubin D 1974. Estimating causal effects of treatments in randomized and nonrandomized studies. J. Educ. Psychol. 66:688–701
     [Google Scholar]
  73. Sampson RJ 2008. Moving to inequality: neighborhood effects and experiments meet social structure. Am. J. Sociol. 114:189–231
     [Google Scholar]
  74. Sampson RJ 2010. Gold standard myths: observations on the experimental turn in quantitative criminology. J. Quant. Criminol. 25:489–500
     [Google Scholar]
  75. Sampson RJ 2012.a Great American City: Chicago and the Enduring Neighborhood Effect Chicago: Univ. Chicago Press
     [Google Scholar]
  76. Sampson RJ 2012.b Moving and the neighborhood glass ceiling. Science 337:1464–65
     [Google Scholar]
  77. Sampson RJ, Winship C, Knight C 2013. Translating causal claims: principles and strategies for policy-relevant criminology. Criminol. Public Policy 12:1–30
     [Google Scholar]
  78. Sanbonmatsu L, Ludwig J, Katz LF, Gennetian LA, Duncan GJ et al. 2011. Moving to Opportunity for Fair Housing Demonstration Program: Final Impacts Evaluation Washington, DC: US Dep. Hous. Urban Dev.
     [Google Scholar]
  79. Sciandra M, Sanbonmatsu L, Duncan GJ, Gennetian LA, Katz LF et al. 2013. Long-term effects of the moving to opportunity residential mobility experiment on crime and delinquency. J. Exp. Criminol. 9:451–89
     [Google Scholar]
  80. Sharkey P, Torrats-Espinosa G, Takyara D 2017. Community and the crime decline: the causal effect of local nonprofits on violent crime. Am. Sociol. Rev. 82:1214–40
     [Google Scholar]
  81. Sharkey PT 2018. Uneasy Peace: The Great Crime Decline, the Revival of City Life, and the Next War on Violence New York: Norton
     [Google Scholar]
  82. Sherman L, Gartin P, Buerger M 1989. Hot spots of predatory crime: routine activities and the criminology of place. Criminology 27:27–55
     [Google Scholar]
  83. Sherman L, Weisburd D 1995. General deterrent effects of police patrol in crime “hot spots”: a randomized study. Justice Q 12:625–48
     [Google Scholar]
  84. Sherman LW, Gottfredson DC, MacKenzie DL, Eck J, Reuter P, Bushway SD 1998. Preventing crime: what works, what doesn't, what's promising Res. Brief, Natl. Inst. Justice Washington, DC:
     [Google Scholar]
  85. Shi L 2009. The limits of oversight in policing: evidence from the 2001 Cincinnati riot. J. Public Econ. 93:99–113
     [Google Scholar]
  86. Skogan W 1990. Disorder and Decline: Crime and the Spiral of Decay in American Cities Berkeley: Univ. California Press
  87. Sobel M 2006. What do randomized studies of housing mobility demonstrate? Causal inference in the face of interference. J. Am. Stat. Assoc. 101:1398–407
     [Google Scholar]
  88. Suresh G, Vito GF 2009. Homicide patterns and public housing: the case of Louisville, KY (1989–2007). Homicide Stud 13:411–33
     [Google Scholar]
  89. Suttles GD 1972. The Social Construction of Communities Chicago: Univ. Chicago Press
  90. Trounstine J 2018. Segregation by Design: Local Politics and Inequality in American Cities New York: Cambridge Univ. Press
  91. Weisburd D, Braga AA, Groff ER, Wooditch A 2017. Can hot spots policing reduce crime in urban areas? An agent-based simulation. Criminology 55:137–73
     [Google Scholar]
  92. Weisburd D, Bushway S, Lum C, Yang S-M 2004. Trajectories of crime at places: a longitudinal study of street segments in the city of Seattle. Criminology 42:238–320
     [Google Scholar]
  93. Weisburd D, Green L 1995. Policing drug hot spots: the Jersey City drug market analysis experiment. Justice Q 12:711–35
     [Google Scholar]
  94. Weisburd D, Hinkle JC 2012. The importance of randomized experiments in evaluating crime prevention. The Oxford Handbook of Crime Prevention BC Welsh, DP Farrington 446–65 Oxford, UK: Oxford Univ. Press
     [Google Scholar]
  95. Wilson WJ 1987. The Truly Disadvantaged: The Inner City, the Underclass, and Public Policy Chicago: Univ. Chicago Press
     [Google Scholar]
  96. Wilson WJ 2018. Don't ignore class when addressing racial gaps in intergenerational mobility. Social Mobility Memos Washington, DC: Brookings https://www.brookings.edu/blog/social-mobility-memos/2018/04/12/dont-ignore-class-when-addressing-racial-gaps-in-intergenerational-mobility/
    [Google Scholar]
  97. Zimring FE 2006. The Great American Crime Decline New York: Oxford Univ. Press
  98. Zimring FE 2013. The City That Became Safe: New York's Lessons for Urban Crime and Its Control New York: Oxford Univ. Press
     [Google Scholar]
/content/journals/10.1146/annurev-criminol-011518-024838
Loading
The Real Gold Standard: Measuring Counterfactual Worlds That Matter Most to Social Science and Policy
Annual Review of Criminology 2, 123 (2019); https://doi.org/10.1146/annurev-criminol-011518-024838
/content/journals/10.1146/annurev-criminol-011518-024838
/content/journals/10.1146/annurev-criminol-011518-024838
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error