Skip to main content

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

  • Perspective
  • Published:

Institutionalizing ethics in AI through broader impact requirements

Nature Machine Intelligence volume 3, pages 104–110 (2021)Cite this article

Subjects

Abstract

Turning principles into practice is one of the most pressing challenges of artificial intelligence (AI) governance. In this Perspective, we reflect on a governance initiative by one of the world’s largest AI conferences. In 2020, the Conference on Neural Information Processing Systems (NeurIPS) introduced a requirement for submitting authors to include a statement on the broader societal impacts of their research. Drawing insights from similar governance initiatives, including institutional review boards (IRBs) and impact requirements for funding applications, we investigate the risks, challenges and potential benefits of such an initiative. Among the challenges, we list a lack of recognized best practice and procedural transparency, researcher opportunity costs, institutional and social pressures, cognitive biases and the inherently difficult nature of the task. The potential benefits, on the other hand, include improved anticipation and identification of impacts, better communication with policy and governance experts, and a general strengthening of the norms around responsible research. To maximize the chance of success, we recommend measures to increase transparency, improve guidance, create incentives to engage earnestly with the process, and facilitate public deliberation on the requirement’s merits and future. Perhaps the most important contribution from this analysis are the insights we can gain regarding effective community-based governance and the role and responsibility of the AI research community more broadly.

This is a preview of subscription content, access via your institution

Access options

Buy this article

  • Purchase on Springer Link
  • Instant access to full article PDF
Buy now

Prices may be subject to local taxes which are calculated during checkout

Similar content being viewed by others

References

  1. Winfield, A. F. T. & Jirotka, M. Ethical governance is essential to building trust in robotics and artificial intelligence systems. Phil. Trans. R. Soc. A 376, 20180085 (2018).

    Article  Google Scholar 

  2. Fisher, E., Mahajan, R. L. & Mitcham, C. Midstream modulation of technology: governance from within. Bull. Sci. Technol. Soc. 26, 485–496 (2006).

    Article  Google Scholar 

  3. NeurIPS Call For Papers (2020); https://nips.cc/Conferences/2020/CallForPapers

  4. Johnson, K. NeurIPS requires AI researchers to account for societal impact and financial conflicts of interest. Venturebeat https://venturebeat.com/2020/02/24/neurips-requires-ai-researchers-to-account-for-societal-impact-and-financial-conflicts-of-interest/ (24 February 2020).

  5. Brundage, M. Artificial intelligence and responsible innovation. In Fundamental Issues of Artificial Intelligence 543−554 (Synthese Library, 2016).

  6. Hecht, B. et al. It’s Time To Do Something: Mitigating The Negative Impacts Of Computing Through A Change To The Peer Review Process (ACM Future of Computing Academy, 29 March 2018); https://acm-fca.org/2018/03/29/negativeimpacts/

  7. NeurIPS Getting Started with NeurIPS 2020 (2020); https://medium.com/@NeurIPSConf/getting-started-with-neurips-2020-e350f9b39c28

  8. NeurIPS NeurIPS 2020 FAQ for Authors (2020); https://nips.cc/Conferences/2020/PaperInformation/NeurIPS-FAQ

  9. Lin, H.-T., Balcan, M. F., Hadsell, R. & Ranzato, M. A. What we learned from NeurIPS 2020 reviewing process. Medium https://medium.com/@NeurIPSConf/what-we-learned-from-neurips-2020-reviewing-process-e24549eea38f (2020).

  10. Hamburger, P. The new censorship: institutional review boards. Supreme Court Rev. 2004, 271–354 (2004).

    Article  Google Scholar 

  11. Buchanan, E., Aycock, J., Dexter, S., Dittrich, D. & Hvizdak, E. Computer science security research and human subjects: emerging considerations for research ethics boards. J. Emp. Res. Human Res. Ethics 6, 71–83 (2011).

    Article  Google Scholar 

  12. Amorim, P. F., Sacramento, C., Capra, E. P., Tavares, P. Z. & Ferreira, S. B. L. Submit or not my HCI research project to the ethics committee, that is the question. In Proc. 18th Brazilian Symp. on Human Factors in Computing Systems (IHC ’19) 1−11 (Association for Computing Machinery, 2019).

  13. Abbott, L. & Grady, C. A systematic review of the empirical literature evaluating IRBs: what we know and what we still need to learn. J. Emp. Res. Human Res. Ethics 6, 3–19 (2011).

    Article  Google Scholar 

  14. Hyman, D. A. Institutional review boards: is this the least worst we can do? Northwestern Univ. Law Rev. 101, 749–774 (2007).

    Google Scholar 

  15. Zywicki, T. J. Institutional review boards as academic bureaucracies: an economic and experiential analysis. Northwestern Univ. Law Rev. 101, 861–896 (2007).

    Google Scholar 

  16. Whitney, S. N. et al. Principal investigator views of the IRB system. Int. J. Med. Sci. 5, 68–72 (2008).

    Article  Google Scholar 

  17. Chadwick, G. L. & Dunn, C. Institutional review boards: changing with the times? J. Public Health Manage. Practice 6, 19–27 (2000).

    Article  Google Scholar 

  18. Fost, N. & Levine, R. J. The dysregulation of human subjects research. JAMA 298, 2196 (2007).

    Article  Google Scholar 

  19. Dziak, K. et al. Variations among institutional review board reviews in a multisite health services research study. Health Serv. Res. 40, 279–290 (2005).

    Article  Google Scholar 

  20. Larson, E., Bratts, T., Zwanziger, J. & Stone, P. A survey of IRB process in 68 U.S. hospitals. J. Nurs. Scholarship 36, 260–264 (2004).

    Article  Google Scholar 

  21. Shah, S., Whittle, A., Wilfond, B., Gensler, G. & Wendler, D. How do institutional review boards apply the federal risk and benefit standards for pediatric research? JAMA 291, 476 (2004).

    Article  Google Scholar 

  22. McWilliams, R. Problematic variation in local institutional review of a multicenter genetic epidemiology study. JAMA 290, 360 (2003).

    Article  Google Scholar 

  23. Goldman, J. Inconsistency and institutional review boards. JAMA 248, 197 (1982).

    Article  Google Scholar 

  24. Reeser, J. C., Austin, D. M., Jaros, L. M., Mukesh, B. N. & McCarty, C. A. Investigating perceived institutional review board quality and function using the IRB researcher assessment tool. J. Emp. Res. Human Res. Ethics 3, 25–34 (2008).

    Article  Google Scholar 

  25. Stryjewski, T. P., Kalish, B. T., Silverman, B. & Lehmann, L. S. The impact of institutional review boards (IRBs) on clinical innovation: a survey of investigators and IRB members. J. Emp. Res. Human Res. Ethics 10, 481–487 (2015).

    Article  Google Scholar 

  26. Keith-Spiegel, P., Koocher, G. P. & Tabachnick, B. What scientists want from their research ethics committee. J. Emp. Res. Human Res. Ethics 1, 67–81 (2006).

    Article  Google Scholar 

  27. Saleem, T. & Khalid, U. Institutional review boards—a mixed blessing. Int. Arch. Med. 4, 19 (2011).

    Article  Google Scholar 

  28. ACM SIGMETRICS 2021. Call for Papers (2020). https://www.sigmetrics.org/sigmetrics2021/call_for_papers.html

  29. Narayanan, A. & Zevenbergen, B. No Encore for Encore? Ethical Questions for Web-Based Censorship Measurement SSRN Scholarly Paper ID 2665148 (Social Science Research Network, 2015).

  30. Kenneally, E. & Bailey, M. Cyber-security research ethics dialogue and strategy workshop. ACM SIGCOMM Comput. Commun. Rev. 44, 76–79 (2014).

    Article  Google Scholar 

  31. Burnett, S. & Feamster, N. Encore: lightweight measurement of web censorship with cross-origin requests. In Proc. 2015 ACM Conf. on Special Interest Group on Data Communication (SIGCOMM ’15) 653-667 (Association for Computing Machinery, 2015).

  32. Kramer, A. D. I., Guillory, J. E. & Hancock, J. T. Experimental evidence of massive-scale emotional contagion through social networks. Proc. Natl Acad. Sci. 111, 8788–8790 (2014).

    Article  Google Scholar 

  33. Editorial Expression of Concern: Experimental evidence of massive-scale emotional contagion through social networks. Proc. Natl Acad. Sci. 111, 10779−10779 (2014).

  34. EPSRC Framework for Responsible Innovation https://epsrc.ukri.org/research/framework/ (2020).

  35. NSF Ch. II—Proposal Preparation Instructions. Proposal & Award Policies & Procedures Guide (29 January 2018); https://www.nsf.gov/pubs/policydocs/pappg18_1/pappg_2.jsp#IIC2di

  36. Tretkoff, E. NSF’s ‘broader impacts’ criterion gets mixed reviews. Am. Phys. Soc. News 16, https://www.aps.org/publications/apsnews/200706/nsf.cfm (2007).

  37. Frodeman, R. & Holbrook, J. B. Science’s social effects. Iss. Sci. Technol. 23, 28–30 (2007).

    Google Scholar 

  38. Bozeman, B. & Boardman, C. Broad impacts and narrow perspectives: passing the buck on science and social impacts. Soc. Epist. 23, 183–198 (2009).

    Article  Google Scholar 

  39. Holbrook, J. B. & Frodeman, R. Peer review and the ex ante assessment of societal impacts. Res. Eval. 20, 239–246 (2011).

    Article  Google Scholar 

  40. Bozeman, B. & Youtie, J. Socio-economic impacts and public value of government-funded research: lessons from four US National Science Foundation initiatives. Res. Pol. 46, 1387–1398 (2017).

    Article  Google Scholar 

  41. Owen, R. & Goldberg, N. Responsible innovation: a pilot study with the U.K. Engineering and Physical Sciences Research Council. Risk Anal. 30, 1699–1707 (2010).

    Article  Google Scholar 

  42. EPSRC Anticipate, Reflect, Engage And Act (AREA) https://epsrc.ukri.org/research/framework/area/ (2020).

  43. Owen, R., Macnaghten, P. & Stilgoe, J. Responsible research and innovation: from science in society to science for society, with society. Sci. Public Pol. 39, 751–760 (2012).

    Article  Google Scholar 

  44. Stilgoe, J., Owen, R. & Macnaghten, P. Developing a framework for responsible innovation. Res. Pol. 42, 1568–1580 (2013).

    Article  Google Scholar 

  45. Marchant, G. E., Allenby, B. R. & Herkert, J. R. (eds.) The Growing Gap Between Emerging Technologies and Legal-Ethical Oversight: The Pacing Problem (The International Library of Ethics, Law and Technology, Springer, 2011).

  46. Gray, I. M. & Edwards-Jones, G. A review of the quality of environmental impact assessments in the Scottish forest sector. Forestry Int. J. Forest Res. 72, 1–10 (1999).

    Article  Google Scholar 

  47. Assessing the Social and Environmental Impacts of European Research Tech. Rep. EUR 21702 (European Commission, 2005).

  48. Spaapen, J. & van Drooge, L. Introducing ’productive interactions’ in social impact assessment. Res. Eval. 20, 211–218 (2011).

    Article  Google Scholar 

  49. Pathways to Impact: Impact core to the UK Research and Innovation Application Process (UK Research and Innovation, 2020); https://webarchive.nationalarchives.gov.uk/20200923120040/https://ahrc.ukri.org/newsevents/news/pathways-to-impact-impact-core-to-the-uk-research-and-innovation-application-process/

  50. Bietti, E. From ethics washing to ethics bashing: a view on tech ethics from within moral philosophy. In Proc. 2020 Conf. on Fairness, Accountability, and Transparency 210−219 (Association for Computing Machinery, 2020).

  51. Hagendorff, T. & Meding, K. The big picture: ethical considerations and statistical analysis of industry involvement in machine learning research. Preprint at http://arxiv.org/abs/2006.04541 (2020).

  52. Stanovich, K. E., West, R. F. & Toplak, M. E. Myside bias, rational thinking, and intelligence. Curr. Dir. Psychol. Sci. 22, 259–264 (2013).

    Article  Google Scholar 

  53. Plous, S. The Psychology Of Judgment And Decision Making (McGraw-Hill, 1993).

  54. Curley, S. P., Yates, J. F. & Abrams, R. A. Psychological sources of ambiguity avoidance. Org. Behav. Human Decision Process. 38, 230–256 (1986).

    Article  Google Scholar 

  55. Nickerson, R. S. Confirmation bias: a ubiquitous phenomenon in many guises. Rev. Gen. Psychol. 2, 175–220 (1998).

    Article  Google Scholar 

  56. Ashurst, C. et al. A Guide to Writing the NeurIPS Impact Statement (Centre for the Governance of AI, 13 May 2020); https://medium.com/@GovAI/a-guide-to-writing-the-neurips-impact-statement-4293b723f832

  57. Hecht, B. Suggestions for Writing NeurIPS 2020 Broader Impacts Statements (22 February, 2020); https://medium.com/@BrentH/suggestions-for-writing-neurips-2020-broader-impacts-statements-121da1b765bf

  58. Porter, A. L., Garner, J. & Crowl, T. Research coordination networks: evidence of the relationship between funded interdisciplinary networking and scholarly impact. BioScience 62, 282–288 (2012).

    Article  Google Scholar 

Download references

Acknowledgements

We thank J. Tenenbaum, Y. Gal, T. Shevlane and colleagues at the Centre for the Governance of AI for helpful feedback and comments.

Author information

Authors and Affiliations

  1. Institute for Ethics in AI, University of Oxford, Oxford, UK

    Carina E. A. Prunkl

  2. Future of Humanity Institute, University of Oxford, Oxford, UK

    Carina E. A. Prunkl, Carolyn Ashurst, Markus Anderljung, Jan Leike & Allan Dafoe

  3. Department of Computer Science, University of Oxford, Oxford, UK

    Helena Webb

Authors
  1. Carina E. A. Prunkl
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Carolyn Ashurst
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Markus Anderljung
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Helena Webb
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jan Leike
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Allan Dafoe
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Carina E. A. Prunkl.

Ethics declarations

Competing interests

The authors declare no competing interests.

Additional information

Peer review information Nature Machine Intelligence thanks Gillian Hadfield, Sean Legassick and the other, anonymous, reviewer(s) for their contribution to the peer review of this work.

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Prunkl, C.E.A., Ashurst, C., Anderljung, M. et al. Institutionalizing ethics in AI through broader impact requirements. Nat Mach Intell 3, 104–110 (2021). https://doi.org/10.1038/s42256-021-00298-y

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1038/s42256-021-00298-y

This article is cited by

Search

Advanced search

Quick links

Nature Briefing

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Get the most important science stories of the day, free in your inbox. Sign up for Nature Briefing