Skip to main content
Log in

A framework for assessing the peer review duration of journals: case study in computer science

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

In various fields, scientific article publication is a measure of productivity and in many occasions it is used as a critical factor for evaluating researchers. Therefore, a lot of time is dedicated to writing articles that are then submitted for publication in journals. Nevertheless, the publication process in general and the review process in particular tend to be rather slow. This is the case for instance of computer science (CS) journals. Moreover, the process typically lacks in transparency, where information about the duration of the review process is at best provided in an aggregated manner, if made available at all. In this paper, we develop a framework as a step towards bringing more reliable data with respect to review duration. Based on this framework, we implement a tool—journal response time (JRT), that allows for automatically extracting the review process data and helps researchers to find the average response times of journals, which can be used to study the duration of CS journals’ peer review process. The information is extracted as metadata from the published articles, when available. This study reveals that the response times publicly provided by publishers differ from the actual values obtained by JRT (e.g., for ten selected journals the average duration reported by publishers deviates by more than 500% from the actual average value calculated from the data inside the articles), which we suspect could be from the fact that, when calculating the aggregated values, publishers consider the review time of rejected articles too (including quick desk-rejections that do not require reviewers).

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6

Similar content being viewed by others

Notes

  1. https://www.researchtrends.com/issue-38-september-2014/publish-or-perish-the-rise-of-the-fractional-author.

  2. http://dtim.essi.upc.edu/jrt/index.php.

  3. https://dblp.org.

  4. E-prints posted in http://www.arxiv.org are not peer reviewed.

  5. https://www.scirev.org.

  6. http://dtim.essi.upc.edu/jrt.

  7. http://dtim.essi.upc.edu/jrt/api.

  8. The full list of the journals for each publisher can be found here: http://dtim.essi.upc.edu/jrt/journals.php.

References

  • Alberts, B., Hanson, B., & Kelner, K. L. (2008). Reviewing peer review. Science, 321(5885), 15.

    Article  Google Scholar 

  • Ausloos, M., Nedic, O., & Dekanski, A. (2019). Seasonal entropy, diversity and inequality measures of submitted and accepted papers distributions in peer-reviewed journals. Entropy, 21(6), 564–575.

    Article  Google Scholar 

  • Azar, O.(2003). Rejections and the importance of first response times (or: How many rejections do others receive?). General Economics and Teaching 0309002, University Library of Munich, Germany.

  • Barbero, E. J. (2008). Journal paper requirement for PhD graduation. Journal of Engineering Education, 2(2), 91–102.

    Google Scholar 

  • Bjork, B.-C., & Solomon, D. (2013). The publishing delay in scholarly peer-reviewed journals. Journal of Informetrics, 7(4), 914–923.

    Article  Google Scholar 

  • Buttler, D. (2013). The dark side of publishing. Nature, 495(7442), 433–435.

    Article  Google Scholar 

  • Daraio, C., Lenzerini, M., Leporelli, C., Moed, H. F., Naggar, P., Bonaccorsi, A., et al. (2016). Data integration for research and innovation policy: An ontology-based data management approach. Scientometrics, 106(2), 857–871.

    Article  Google Scholar 

  • Daraio, C., Lenzerini, M., Leporelli, C., Naggar, P., Bonaccorsi, A., & Bartolucci, A. (2016). The advantages of an Ontology-Based Data Management approach: Openness, interoperability and data quality. Scientometrics, 108(1), 441–455.

    Article  Google Scholar 

  • Doi, S. A. R., Salzman-Scott, S. A., & Onitilo, A. A. (2016). Validation of the CoRE questionnaire for a medical journal peer review. Accountability in Research, 23(1), 47–52.

    Article  Google Scholar 

  • Drvenica, I., Bravo, G., Vejmelka, L., Dekanski, A., & Nedic, O. (2018). Peer review of reviewers: The author’s perspective. Publications, 7(1), 1.

    Article  Google Scholar 

  • Ellison, G. (2002). Evolving standards for academic publishing: A q-r theory. Journal of Political Economy, 110(5), 994–1034.

    Article  Google Scholar 

  • Ellison, G. (2007). Is peer review in decline? Economic Inquiry, 49(3), 635–657.

    Article  MathSciNet  Google Scholar 

  • Huisman, J., & Smits, J. (2017). Duration and quality of the peer review process: The author’s perspective. Scientometrics, 113(1), 633–650.

    Article  Google Scholar 

  • Jennings, C. G. (2020). Quality and value: The true purpose of peer review? http://blogs.nature.com/peer-to-peer/2006/06/quality_and_value_the_true_pur.html. Accessed September 20, 2020.

  • Kareiva, P., Marvier, M., West, S., & Hornisher, J. (2002). Slow-moving journals hinder conservation efforts. Nature, 420, 15.

    Article  Google Scholar 

  • Lewin, A. Y. (2014). The peer-review process: The good, the bad, the ugly, and the extraordinary. Management and Organization Review, 10(2), 167–173.

    Article  Google Scholar 

  • Lotriet Cornelius, J. (2012). Reviewing the review process: Identifying sources of delay. The Australasian Medical Journal, 5, 26–29.

    Article  Google Scholar 

  • Mrowinski, M. J., Fronczak, A., Fronczak, P., Nedic, O., & Ausloos, M. (2016). Review time in peer review: Quantitative analysis and modelling of editorial workflows. Scientometrics, 107(1), 271–286.

    Article  Google Scholar 

  • Mrowinski, M. J., Fronczak, A., Fronczak, P., Nedic, O., & Dekanski, A. (2020). The hurdles of academic publishing from the perspective of journal editors: A case study. Scientometrics, 125(1), 115–133.

    Article  Google Scholar 

  • Peacock, S. (2017). The PhD by publication. International Journal of Doctoral Studies, 12, 123–135.

    Article  Google Scholar 

  • Peters, D. P., & Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(2), 187–195.

    Article  Google Scholar 

  • Resnik, D., Gutierrez-Ford, C., & Peddada, S. (2008). Perceptions of ethical problems with scientific journal peer review: An exploratory study. Science and Engineering Ethics, 14(3), 305–310.

    Article  Google Scholar 

  • Robins, L., & Kanowski, P. (2008). PhD by publication: A student’s perspective. Journal of Research Practice, 4(2), 3.

    Google Scholar 

  • Smith, A. J. (1990). The task of the referee. Computer, 23(4), 65–71.

    Article  Google Scholar 

  • Solomon, D. J., & Björk, B.-C. (2012). Publication fees in open access publishing: Sources of funding and factors influencing choice of journal. Journal of the American Society for Information Science and Technology, 63(1), 98–107.

    Article  Google Scholar 

  • Tite, L., & Sara, S. (2007). Why do peer reviewers decline to review? A survey. Journal of Epidemiology and Community Health, 61(1), 9–12.

    Article  Google Scholar 

  • Ware, M., & Mabe, M. (2015). The STM Report: An overview of scientific and scholarly journal publishing. 03 2015.

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Besim Bilalli.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bilalli, B., Munir, R.F. & Abelló, A. A framework for assessing the peer review duration of journals: case study in computer science. Scientometrics 126, 545–563 (2021). https://doi.org/10.1007/s11192-020-03742-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-020-03742-9

Keywords

Navigation