Abstract
Preprints play an increasingly critical role in academic communities. There are many reasons driving researchers to post their manuscripts to preprint servers before formal submission to journals or conferences, but the use of preprints has also sparked considerable controversy, especially surrounding the claim of priority. In this paper, a case study of computer science preprints submitted to arXiv from 2008 to 2017 is conducted to quantify how many preprints have eventually been printed in peer-reviewed venues. Among those published manuscripts, some are published under different titles and without an update to their preprints on arXiv. In the case of these manuscripts, the traditional fuzzy matching method is incapable of mapping the preprint to the final published version. In view of this issue, we introduce a semantics-based mapping method with the employment of Bidirectional Encoder Representations from Transformers (BERT). With this new mapping method and a plurality of data sources, we find that 66% of all sampled preprints are published under unchanged titles and 11% are published under different titles and with other modifications. A further analysis was then performed to investigate why these preprints but not others were accepted for publication. Our comparison reveals that in the field of computer science, published preprints feature adequate revisions, multiple authorship, detailed abstract and introduction, extensive and authoritative references and available source code.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Notes
MDPI is an acronym referring two related organizations, Molecular Diversity Preservation International and Multidisciplinary Digital Publishing Institute.
We originally added abstract pairs (preprint, candidate) to our dataset, but we found that only a little part of data of Crossref had abstracts when the model had actually been used, so the abstract pairs were removed in our model.
Preprints which fall under this condition have statements like “submitted to (a certain journal)” or “submitted to (a certain conference)” included in their metadata. However, up until this paper is written, no corresponding records can be found in any journal or conference proceedings. The results are further confirmed with the method presented in "Methods" section. Therefore, we reach the conclusion that these preprints submitted fail to be accepted.
Full names of categories on arXiv are attached in "Appendix 1" for reference.
Data in "What preprints can be printed" section were tested with the same methods and we came to the same conclusion.
There were 53 preprints without tables among these 100 records and we indeed found that a small part of the automatically parsing values were smaller than the manually calculated ones.
References
Annesley, T., Scott, M., Bastian, H., Fonseca, V., Ioannidis, J. P., Keller, M. A., et al. (2017). Biomedical journals and preprint services: Friends or foes? Clinical Chemistry, 63(2), 453–458. https://doi.org/10.1373/clinchem.2016.268227.
Beltagy, I., Lo, K., & Cohan, A. (2019). SciBERT: A pretrained language model for scientific text. In Proceedings of EMNLP-IJCNLP, association for computational linguistics (pp. 3615–3620). https://doi.org/10.18653/v1/D19-1371.
Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E. J., Berk, R., et al. (2018). Redefine statistical significance. Nature Human Behaviour, 2(1), 6–10. https://doi.org/10.1038/s41562-017-0189-z.
Berg, J. M., Bhalla, N., Bourne, P. E., Chalfie, M., Drubin, D. G., Fraser, J. S., et al. (2016). Preprints for the life sciences. Science, 352(6288), 899–901. https://doi.org/10.1126/science.aaf9133.
Björk, B. C., & Solomon, D. (2013). The publishing delay in scholarly peer-reviewed journals. Journal of Informetrics, 7(4), 914–923. https://doi.org/10.1016/j.joi.2013.09.001.
Brody, T., Harnad, S., & Carr, L. (2006). Earlier web usage statistics as predictors of later citation impact. Journal of the American Society for Information Science and Technology, 57(8), 1060–1072. https://doi.org/10.1002/asi.20373.
Cavacini, A. (2015). What is the best database for computer science journal articles? Scientometrics, 102(3), 2059––2071. https://doi.org/10.1007/s11192-014-1506-1.
D′Agostino, R. B. (1971). An omnibus test of normality for moderate and large size samples. Biometrika, 58(2), 341–348. https://doi.org/10.1093/biomet/58.2.341.
Davis, P. M., & Fromerth, M. J. (2007). Does the arXiv lead to higher citations and reduced publisher downloads for mathematics articles? Scientometrics, 71(2), 20–215. https://doi.org/10.1007/s11192-007-1661-8.
Devlin, J., Chang, M. W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, association for computational linguistics (pp. 4171–4186). https://doi.org/10.18653/v1/N19-1423.
Feldman, S., Lo, K., & Ammar, W. (2018). Citation count analysis for papers with preprints. arXiv preprint arXiv:1805.05238.
Ginsparg, P. (2011). ArXiv at 20. Nature, 476(7359), 145–147. https://doi.org/10.1038/476145a.
Halpern, J. Y. (2000). CoRR: A computing research repository. ACM Journal of Computer Documentation, 24(2), 41–48. https://doi.org/10.1145/337271.337274.
Kling, R. (2005). The internet and unrefereed scholarly publishing. Annual Review of Information Science and Technology, 38(1), 591–631. https://doi.org/10.1002/aris.1440380113.
Kousha, K., & Thelwall, M. (2008). Sources of Google Scholar citations outside the science citation index: A comparison between four science disciplines. Scientometrics, 74(2), 273–294. https://doi.org/10.1007/s11192-008-0217-x.
Lammey, R. (2014). CrossRef developments and initiatives: An update on services for the scholarly publishing community from CrossRef. Science Editing, 1(1), 13–18. https://doi.org/10.6087/kcse.2014.1.13.
Larivière, V., Sugimoto, C. R., Macaluso, B., Milojević, S., Cronin, B., & Thelwall, M. (2014). arXiv e-prints and the journal of record: An analysis of roles and relationships. Journal of the Association for Information Science and Technology, 65(6), 1157–1169. https://doi.org/10.1002/asi.23044.
Ley, M. (2002). The DBLP computer science bibliography: Evolution, research issues, perspectives. In Proceedings of SPIRE (pp. 1–10). https://doi.org/10.1007/3-540-45735-6_1.
Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 18(1), 50–60. https://www.jstor.org/stable/2236101.
Martin-Martin, A., Orduna-Malea, E., Harzing, A. W., & Delgado López-Cózar, E. (2017). Can we use Google Scholar to identify highly-cited documents? Journal of Informetrics, 11(1), 152–163. https://doi.org/10.1016/j.joi.2016.11.008.
Martín-Martín, A., Orduna-Malea, E., Thelwall, M., & Delgado López-Cózar, E. (2018). Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories. Journal of Informetrics, 12(4), 1160–1177. https://doi.org/10.1016/j.joi.2018.09.002.
Pearson, E. S., D′Agostino, R. B., & Bowman, K. O. (1977). Tests for departure from normality: Comparison of powers. Biometrika, 64(2), 231–246. https://doi.org/10.1093/biomet/64.2.231.
R Core Team. (2013). R: A language and environment for statistical computing. R Foundation for Statistical Computing. https://www.r-project.org/.
Rieger, O. Y., Steinhart, G., & Cooper, D. (2016). arXiv@25: Key findings of a user survey. arXiv preprint arXiv:1607.08212.
Sutton, C., & Gong, L. (2017). Popularity of arXiv.org within computer science. arXiv preprint arXiv:1710.05225.
Vale, R. D. (2015). Accelerating scientific publication in biology. Proceedings of the National Academy of Sciences of the United States of America, 112(44), 13439–13446. https://doi.org/10.1073/pnas.1511912112.
Van Noorden, R. (2014). The arXiv preprint server hits 1 million articles. Nature. https://doi.org/10.1038/nature.2014.16643.
Vrettas, G., & Sanderson, M. (2015). Conferences versus journals in computer science. Journal of the Association for Information Science and Technology, 66(12), 2674–2684. https://doi.org/10.1002/asi.23349.
Warner, S. (2001). Open archives initiative protocol development and implementation at arXiv. arXiv preprint arXiv:cs/0101027.
Zeiler, M. D. (2012). ADADELTA: An adaptive learning rate method. arXiv preprint arXiv:1212.5701.
Acknowledgements
We are grateful to Yingmin Wang and Xingchen He for their assistance in processing data for this research. We appreciated Ziyi Chen for her help of statistical analysis. We also thank two anonymous reviewers for their insightful comments. Special and heartfelt gratitude goes to the first author’s wife Fenmei Zhou, for her understanding and love. Her unwavering support and continuous encouragement enable this research to be possible.
Funding
This work is partly sponsored by the State Language Commission of China through the 13th Five-Year Plan project Artificial Intelligence and Language (Grant No. WT135-38).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Appendix 1: Abbreviation—full name of arXiv categories
Appendix 1: Abbreviation—full name of arXiv categories
Abbr. | Full name | Abbr. | Full name |
|---|---|---|---|
CS.AI | Artificial Intelligence | CS.IR | Information Retrieval |
CS.AR | Hardware Architecture | CS.IT | Information Theory |
CS.CC | Computational Complexity | CS.LG | Machine Learning |
CS.CE | Computational Engineering, Finance, and Science | CS.LO | Logic in Computer Science |
CS.CG | Computational Geometry | CS.MA | Multiagent Systems |
CS.CL | Computation and Language | CS.MM | Multimedia |
CS.CR | Cryptography and Security | CS.MS | Mathematical Software |
CS.CV | Computer Vision and Pattern Recognition | CS.NA | Numerical Analysis |
CS.CY | Computers and Society | CS.NE | Neural and Evolutionary Computation |
CS.DB | Databases | CS.NI | Networking and Internet Architecture |
CS.DC | Distributed, Parallel, and Cluster Computing | CS.OH | Other |
CS.DL | Digital Libraries | CS.OS | Operating Systems |
CS.DM | Discrete Mathematics | CS.PF | Performance |
CS.DS | Data Structures and Algorithms | CS.PL | Programming Languages |
CS.ET | Emerging Technologies | CS.RO | Robotics |
CS.FL | Formal Languages and Automata Theory | CS.SC | Symbolic Computation |
CS.GL | General Literature | CS.SD | Sound |
CS.GR | Graphics | CS.SE | Software Engineering |
CS.GT | Computer Science and Game Theory | CS.SI | Social and Information Networks |
CS.HC | Human–Computer Interaction | CS.SY | Systems and Control |
EESS | Electrical Engineering and Systems Science |
Rights and permissions
About this article
Cite this article
Lin, J., Yu, Y., Zhou, Y. et al. How many preprints have actually been printed and why: a case study of computer science preprints on arXiv. Scientometrics 124, 555–574 (2020). https://doi.org/10.1007/s11192-020-03430-8
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-020-03430-8