Hostname: page-component-8448b6f56d-t5pn6 Total loading time: 0 Render date: 2024-04-18T13:47:04.567Z Has data issue: false hasContentIssue false
  • English
  • Français

Supporting robust, rigorous, and reliable reviewing as the cornerstone of our profession: Introducing a competency framework for peer review

Published online by Cambridge University Press:  04 February 2020

Tine Köhler ,
M. Gloria González-Morales ,
George C. Banks ,
Ernest H. O’Boyle Open the ORCID record for Ernest H. O’Boyle [Opens in a new window] ,
Joseph A. Allen ,
Ruchi Sinha ,
Sang Eun Woo  and
Lisa M. V. Gulick
Tine Köhler*
Affiliation:
Department of Management and Marketing, The University of Melbourne, Parkville, Victoria, Australia
M. Gloria González-Morales
Affiliation:
Department of Psychology, University of Guelph, Guelph, Ontario, Canada
George C. Banks
Affiliation:
University of North Carolina–Charlotte, Charlotte, North Carolina, USA
Ernest H. O’Boyle
Affiliation:
Kelley School of Business, Indiana University, Bloomington, Indiana, USA
Joseph A. Allen
Affiliation:
University of Utah Health, Salt Lake City, Utah, USA
Ruchi Sinha
Affiliation:
UniSA Business School, University of South Australia, Adelaide, South Australia, Australia
Sang Eun Woo
Affiliation:
Purdue University, West Lafayette, Indiana, USA
Lisa M. V. Gulick
Affiliation:
Bendelta, Sydney, New South Wales, Australia
*
*Corresponding author. Email: tkoehler@unimelb.edu.au
Get access Rights & Permissions [Opens in a new window]

Abstract

Peer review is a critical component toward facilitating a robust science in industrial and organizational (I-O) psychology. Peer review exists beyond academic publishing in organizations, university departments, grant agencies, classrooms, and many more work contexts. Reviewers are responsible for judging the quality of research conducted and submitted for evaluation. Furthermore, they are responsible for treating authors and their work with respect, in a supportive and developmental manner. Given its central role in our profession, it is curious that we do not have formalized review guidelines or standards and that most of us never receive formal training in peer reviewing. To support this endeavor, we are proposing a competency framework for peer review. The purpose of the competency framework is to provide a definition of excellent peer reviewing and guidelines to reviewers for which types of behaviors will lead to good peer reviews. By defining these competencies, we create clarity around expectations for peer review, standards for good peer reviews, and opportunities for training the behaviors required to deliver good peer reviews. We further discuss how the competency framework can be used to improve peer reviewing and suggest additional steps forward that involve suggestions for how stakeholders can get involved in fostering high-quality peer reviewing.

Keywords

peer reviewcompetency frameworkdevelopmental feedbackreliable reviewing
Type
Focal Article
Information
Industrial and Organizational Psychology , Volume 13 , Issue 1 , March 2020 , pp. 1 - 27
Copyright
© Society for Industrial and Organizational Psychology, Inc. 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The authors of this article are members of the Rigorous, Robust, and Reliable Reviewing Subcommittee of the Society for Industrial and Organizational Psychology’s (SIOP) Education and Training committee. The subcommittee was charged with creating resources to provide reviewer training and improve reviewing quality in the industrial and organizational (I-O) psychology community. As part of this task, the subcommittee created a competency framework for peer review that we are sharing in this focal article. All subsequent efforts to create and share resources are guided by this competency framework. An earlier version of the competency framework was shared with SIOP’s Executive Board, various SIOP committees, and the SIOP membership at the annual conference in April 2018. Furthermore, this earlier version of the competency framework formed the basis of a reviewer training bootcamp session held in collaboration with the Consortium for the Advancement of Research Methods and Analysis (CARMA) in August 2018. The intention of this focal article is to introduce the competency framework to the larger I-O psychology community to elicit feedback and commentary from its diverse body of members. We hope that the feedback and commentary will help us refine and expand upon the competency framework and suggest various ways in which it can be employed to improve our science and practice. This feedback will be reported back to SIOP’s Executive Board and will hopefully also assist us in creating relevant resources for reviewer training.

References

Aguinis, H., Ramani, R. S., & Alabduljader, N. (2018). What you see is what you get? Enhancing methodological transparency in management research. Academy of Management Annals, 12, 83–110.CrossRefGoogle Scholar
Annesley, T. M. (2012). Seven reasons not to be a peer reviewer—and why these reasons are wrong. Clinical Chemistry, 58(4), 677–679.CrossRefGoogle ScholarPubMed
Antonakis, J. (2017). On doing better science: From thrill of discovery to policy implications. The Leadership Quarterly, 28(1), 5–21.CrossRefGoogle Scholar
Banks, G. C., O’Boyle, E. H., Pollack, J. M., White, C. D., Batchelor, J. H., Whelpley, C. E., … Adkins, C. L. (2016a). Questions about questionable research practices in the field of management: A guest commentary. Journal of Management, 42, 5–20.CrossRefGoogle Scholar
Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., & Rupp, D. E. (2016b). Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31, 323–338.CrossRefGoogle Scholar
Bartlett, D., & Francis-Smythe, J. (2016). Bridging the divide in work and organizational psychology: Evidence from practice. European Journal of Work and Organizational Psychology, 25(5), 615–630.CrossRefGoogle Scholar
Bedeian, A. G. (1996). Improving the journal review process: The question of ghostwriting. American Psychologist, 51, 1189.CrossRefGoogle Scholar
Bedeian, A. G. (2003). The manuscript review process: The proper roles of authors, referees, and editors. Journal of Management Inquiry, 12, 331–338.CrossRefGoogle Scholar
Briner, R. B., & Rousseau, D. M. (2011). Evidence-based I–O psychology: Not there yet. Industrial and Organizational Psychology, 4(1), 3–22.CrossRefGoogle Scholar
Carpenter, M. A. (2009). Editor’s comments: Mentoring colleagues in the craft and spirit of a peer review. Academy of Management Review, 34, 191–195.CrossRefGoogle Scholar
Chambers, C. D., Feredoes, E., Muthukumaraswamy, S. D., & Etchells, P. (2014). Instead of “playing the game” it is time to change the rules: Registered Reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1, 4–17.CrossRefGoogle Scholar
Chan, M. E., & Arvey, R. D. (2012). Meta-analysis and the development of knowledge. Perspectives on Psychological Science, 7, 79–92.CrossRefGoogle Scholar
Cicchetti, D. V. (1980). Reliability of reviews for the American Psychologist: A biostatistical assessment of the data. American Psychologist, 35(3), 300–303.CrossRefGoogle Scholar
Clair, J. A. (2015). Procedural injustice in the system of peer review and scientific misconduct. Academy of Management Learning & Education, 14(2), 159–172.CrossRefGoogle Scholar
Cortina, J. M., Green, J. P., Keeler, K. R., & Vandenberg, R. J. (2017). Degrees of freedom in SEM: Are we testing the models that we claim to test? Organizational Research Methods, 20(3), 350–378.CrossRefGoogle Scholar
DeSimone, J. A., Köhler, T., & Schoen, J. L. (2019). If it were only that easy: The use of meta-analytic research by organizational scholars. Organizational Research Methods, 22(4), 867–891.CrossRefGoogle Scholar
Emerson, G. B., Warme, W. J., Wolf, F. M., Heckman, J. D., Brand, R. A., & Leopold, S. S. (2010). Testing for the presence of positive-outcome bias in peer review: A randomized controlled trial. Archives of Internal Medicine, 170, 1934–1939.CrossRefGoogle ScholarPubMed
Epstein, S. (1995). What can be done to improve the journal review process. American Psychologist, 50(10), 883–885.CrossRefGoogle Scholar
Feldman, D. C. (2005). Writing and reviewing as sadomasochistic rituals. Journal of Management, 31(3), 325–329.CrossRefGoogle Scholar
Findley, M. G., Jensen, N. M., Malesky, E. J., & Pepinsky, T. B. (2016). Can results-free review reduce publication bias? The results and implications of a pilot study. Comparative Political Studies, 49, 1667–1703.CrossRefGoogle Scholar
Grand, J. A., Rogelberg, S. G., Allen, T. D., Landis, R. S., Reynolds, D. H., Scott, J. C., … Truxillo, D. M. (2018a). A systems-based approach to fostering robust science in industrial-organizational psychology. Industrial and Organizational Psychology: Perspectives on Science and Practice, 11(1), 4–42.CrossRefGoogle Scholar
Grand, J. A., Rogelberg, S. G., Banks, G. C., Landis, R. S., & Tonidandel, S. (2018b). From outcome to process focus: Fostering a more robust psychological science through registered reports and results-blind reviewing. Perspectives on Psychological Science, 13, 448–456.CrossRefGoogle ScholarPubMed
Greco, L. M., O’Boyle, E. H., Cockburn, B. S., & Yuan, Z. (2018). Meta-analysis of coefficient alpha: A reliability generalization study. Journal of Management Studies, 55(4), 583–618.CrossRefGoogle Scholar
Grote, G. (2016). There is hope for better science. European Journal of Work and Organizational Psychology, 26, 1–3.CrossRefGoogle Scholar
Hardwicke, T. E., Mathur, M., MacDonald, K., Nilsonne, G., Banks, G. C., Kidwell, M. C., … Tessler, M. H. (2018). Data availability, reusability, and analytic reproducibility: Evaluating the impact of a mandatory open data policy at the journal Cognition. Royal Society Open Science, 5(8). doi:10.1098/rsos.180448.CrossRefGoogle ScholarPubMed
Highfield, R., & Carter, P. (1994). The private lives of Albert Einstein. New York: St. Martins Press.CrossRefGoogle Scholar
Ioannidis, J. P. (2005). Why most published research findings are false. Plos Medicine, 2(8), e124.CrossRefGoogle ScholarPubMed
Kanfer, R., & Ackerman, P. L. (1989). Motivation and cognitive abilities: An integrative/aptitude-treatment interaction approach to skill acquisition. Journal of Applied Psychology, 74(4), 657–690.CrossRefGoogle Scholar
Kanigel, R. (2016). The man who knew infinity: A life of the genius Ramanujan. New York, NY: Washington Square Press.Google Scholar
Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in I-O psychology? Industrial and Organizational Psychology, 6(3), 252–268.CrossRefGoogle Scholar
Kidwell, M. C., Lazarević, L. B., Baranski, E., Hardwicke, T. E., Piechowski, S., Falkenberg, L.-S., … Hess-Holden, C. (2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology, 14(5), e1002456.CrossRefGoogle ScholarPubMed
Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical review, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.CrossRefGoogle Scholar
Köhler, T. (2017). Coercive citations. In Rogelberg, S. (Ed.), The SAGE encyclopedia of industrial and organizational psychology (2nd ed., pp. 185–186). Thousand Oaks, CA: SAGE Publications.Google Scholar
Lee, A. S. (1995). Reviewing a manuscript for publication. Journal of Operations Management, 13(1), 87–92.CrossRefGoogle Scholar
Miller, C. C. (2006). Peer review in the organizational and management sciences: Prevalence and effects of reviewer hostility, bias, and dissensus. Academy of Management Journal, 49(3), 425–431.CrossRefGoogle Scholar
Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., Breckler, S., … Yarkoni, T. (2015). Promoting an open research culture: Author guidelines for journals to promote transparency, openness, and reproducibility. Science, 348, 1422–1425.CrossRefGoogle Scholar
Nosek, B. A., Spies, J. R., & Motyl, M. (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631.CrossRefGoogle ScholarPubMed
O’Boyle, E., Banks, G. C., Carter, K., Walter, S., & Yuan, Z. (2019). A 20-year review of outcome reporting bias in moderated multiple regression. Journal of Business and Psychology, 34(1), 19–37.CrossRefGoogle Scholar
O’Boyle, E. H., Banks, G. C., & Gonzalez-Mulé, E. (2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43(2), 376–399.CrossRefGoogle Scholar
Roth, W. M. (2002). Editorial power/authorial suffering. Research in Science Education, 32(2), 215–240.CrossRefGoogle Scholar
Rotolo, C. T., Church, A. H., Adler, S., Smither, J. W., Colquitt, A. L., Shull, A. C., … Foster, G. (2018). Putting an end to bad talent management: A call to action for the field of industrial and organizational psychology. Industrial and Organizational Psychology, 11(2), 176–219.CrossRefGoogle Scholar
Rupp, D. (2011). Ethical issues faced by editors and reviewers. Management and Organization Review, 7, 481–493.CrossRefGoogle Scholar
Scott, W. A. (1974). Interreferee agreement on some characteristics of manuscripts submitted to the Journal of Personality and Social Psychology. American Psychologist, 29, 698–702.CrossRefGoogle Scholar
Silzer, R. F., & Parson, C. (2012). Industrial-organizational psychology journals and the science–practice gap. The Industrial-Organizational Psychologist, 49(4), 97–117.Google Scholar
Smaldino, P. E. & McElreath, R. (2016). The natural selection of bad science. Royal Society Open Science, 3, 160384.CrossRefGoogle ScholarPubMed
Smithsonian. (2019). The true story of the case Ruth Bader Ginsburg argues in ‘On the basis of sex’. Retrieved from https://www.smithsonianmag.com/history/true-story-case-center-basis-sex-180971110/.Google Scholar
Souder, L. (2011). The ethics of scholarly peer review: A review of the literature. Learned Publishing, 24(1), 55–72.CrossRefGoogle Scholar
Starbuck, W. H. (2003). Turning lemons into lemonade: Where is the value in peer reviews? Journal of Management Inquiry, 12(4), 344–351.CrossRefGoogle Scholar
Starbuck, W. H. (2005). How much better are the most-prestigious journals? The statistics of academic publication. Organization Science, 16(2), 180–200.CrossRefGoogle Scholar
Tay, L., & Diener, E. (2018). Selecting the right journal outlet for your paper. In Sternberg, R. J. (Ed.), Guide to publishing in psychology journals (2nd ed., 181–191). Cambridge, UK: Cambridge University Press.CrossRefGoogle Scholar
Weller, A. C. (2001). Editorial peer review: Its strengths and weaknesses. Medford, NJ: Information Today.Google Scholar
Williams, L. J., O’Boyle, E. H., & Yu, J. (2020). Condition 9 and 10 tests of model confirmation: A review of James, Mulaik, and Brett (1982) and contemporary alternatives. Organizational Research Methods, 23(1), 6–29. doi:10.1177/1094428117736137.CrossRefGoogle Scholar
Woznyj, H. M., Grenier, K., Ross, R., Banks, G. C., & Rogelberg, S. G. (2018). Results-blind review: A masked crusader for science. European Journal of Work and Organizational Psychology, 27(5), 561–576. doi:10.1080/1359432X.2018.1496081.CrossRefGoogle Scholar