Skip to main content
SpringerLink
Log in

How accurate are policy document mentions? A first look at the role of altmetrics database

  • Published:
Scientometrics Aims and scope Submit manuscript

Abstract

Policy document mention is considered to indicate the significance and societal impact of scientific product. However, the accuracy of policy document altmetrics data needs to be evaluated to fully understand its strength and limitation. An in-depth coding analysis was conducted on sample policy documents records of Altmetric.com database. The sample consists of 2079 records from all 79 distinct policy document source platforms tracked by the database. Errors about mentioned publications in the policy documents (type A error) are found in 8% of the records, while errors about either the recorded policy documents or the mentioned publications in the altmetrics database (type B error) are found in 70% of the records. In type B error, policy document link error (5% of the records) could be attributable to the policy document website, transcription error (52% of the records) could be attributable to the third-party bibliographic data provider. These two categories of error are relatively minor and may have limited influence on altmetrics research and practices. False positive policy document mention (13% of the records), however, could be attributable to the Altmetric database and may diminish the validity of research based on the policy document altmetrics data. The underlying reasons remain to be further investigated. Considering the high complexity of extracting mentions of publications from various sources and formats of policy documents as well as its short history, Altmetric database has achieved excellent performance.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2

Similar content being viewed by others

Notes

  1. See https://help.altmetric.com/support/solutions/articles/6000060968-what-outputs-and-sources-does-altmetric-track.

  2. See https://help.altmetric.com/support/solutions/articles/6000136884-when-did-altmetric-start-tracking-attention-to-each-attention-source.

References

  • Aduku, K. J., Thelwall, M., & Kousha, K. (2017). Do Mendeley reader counts reflect the scholarly impact of conference papers? An investigation of computer science and engineering. Scientometrics, 112(1), 573–581.

    Article  Google Scholar 

  • Archambault, É., Campbell, D., Gingras, Y., & Larivière, V. (2009). Comparing bibliometric statistics obtained from the Web of Science and Scopus. Journal of the American Society for Information Science and Technology, 60(7), 1320–1326.

    Article  Google Scholar 

  • Bar-Ilan, J. (2014). JASIST@ Mendeley revisited. altmetrics14: Expanding impacts and metrics. In Workshop at web science conference 2014.

  • Bornmann, L., & Haunschild, R. (2015). Which people use which scientific papers? An evaluation of data from F1000 and Mendeley. Journal of Informetrics, 9(3), 477–487.

    Article  Google Scholar 

  • Bornmann, L., Haunschild, R., & Marx, W. (2016). Policy documents as sources for measuring societal impact: How often is climate change research mentioned in policy-related documents? Scientometrics, 109(3), 1477–1495.

    Article  Google Scholar 

  • Buchanan, R. A. (2006). Accuracy of cited references: The role of citation databases. College and Research Libraries, 67(4), 292–303.

    Article  Google Scholar 

  • Calver, M. C., Goldman, B., Hutchings, P. A., & Kingsford, R. T. (2017). Why discrepancies in searching the conservation biology literature matter. Biological Conservation, 213, 19–26.

    Article  Google Scholar 

  • Chamberlain, S. (2013). Consuming article-level metrics: Observations and lessons. Information Standards Quarterly, 25(2), 4–13.

    Article  Google Scholar 

  • Chen, X. J. (2014). Study of quality of references and its auditing methods. Chinese Journal of Scientific and Technical Periodicals, 25(9), 1145–1148. (in Chinese).

    Google Scholar 

  • Donner, P. (2017). Document type assignment accuracy in the journal citation index data of Web of Science. Scientometrics, 113(1), 219–236.

    Article  Google Scholar 

  • Fang, Z., & Costas, R. (2020). Studying the accumulation velocity of altmetric data tracked by Altmetric.com. Scientometrics, 123, 1077–1101. https://doi.org/10.1007/s11192-020-03405-9.

    Article  Google Scholar 

  • Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2014). Scientific journal publishers and omitted citations in bibliometric databases: Any relationship? Journal of Informetrics, 8(3), 751–765.

    Article  Google Scholar 

  • Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016a). The museum of errors/horrors in Scopus. Journal of Informetrics, 10(1), 174–182.

    Article  Google Scholar 

  • Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016b). Empirical analysis and classification of database errors in Scopus and Web of Science. Journal of Informetrics, 10(4), 933–953.

    Article  Google Scholar 

  • Franceschini, F., Maisano, D., & Mastrogiacomo, L. (2016c). Do Scopus and WoS correct old omitted citations? Scientometrics, 107(2), 321–335.

    Article  Google Scholar 

  • Garfield, E. (1974). Errors-theirs, ours and yours. In Essays of an information scientist (Philadelphia: ISI Pr., 1977), 2: 80–81. Originally published in Current Contents (June 19, 1974): 5–6.

  • Harzing, A.-W., & Alakangas, S. (2016). Google Scholar, Scopus and the Web of Science: A longitudinal and cross-disciplinary comparison. Scientometrics, 106(2), 787–804.

    Article  Google Scholar 

  • Haunschild, R., & Bornmann, L. (2017). How many scientific papers are mentioned in policy-related documents? An empirical investigation using Web of Science and Altmetric data. Scientometrics, 110(3), 1209–1216.

    Article  Google Scholar 

  • Haustein, S. (2016). Grand challenges in altmetrics: Heterogeneity, data quality and dependencies. Scientometrics, 108(1), 413–423.

    Article  Google Scholar 

  • Haustein, S., Bowman, T. D. & Costas, R. (2015). When is an article actually published? An analysis of online availability, publication, and indexation dates. Retrieved from https://arxiv.org/ftp/arxiv/papers/1505/1505.00796.pdf.

  • Khazragui, H., & Hudson, J. (2015). Measuring the benefits of university research: Impact and the REF in the UK. Research Evaluation, 24(1), 51–62.

    Article  Google Scholar 

  • Meho, L. I., & Yang, K. (2007). Impact of data sources on citation counts and rankings of LIS faculty: Web of Science versus Scopus and Google Scholar. Journal of the American Society for Information Science and Technology, 58(13), 2105–2125.

    Article  Google Scholar 

  • Meschede, C., & Siebenlist, T. (2018). Cross-metric compatability and inconsistencies of altmetrics. Scientometrics, 115(1), 283–297.

    Article  Google Scholar 

  • Moed, H. F. (2002). The impact-factors debate: The ISI’s uses and limits. Nature, 415(6873), 731.

    Article  Google Scholar 

  • Mohammadi, E., Kwasny, M., & Holmes, K. L. (2018). Academic information on Twitter: A user survey. PLoS ONE, 13(5), e0197265.

    Article  Google Scholar 

  • Newson, R., King, L., Rychetnik, L., et al. (2018). Looking both ways: A review of methods for assessing research impacts on policy and the policy utilisation of research. Health Research Policy and Systems, 16(1), 54.

    Article  Google Scholar 

  • NISO. (2016). Outputs of the NISO alternative assessment project. Retrieved from https://groups.niso.org/apps/group_public/download.php/17091/NISO+RP-25-2016+Outputs+of+the+NISO+Alternative+Assessment+Project.pdf.

  • Ortega, J. L. (2018). Reliability and accuracy of altmetric providers: A comparison among Altmetric.com, PlumX and Crossref Event Data. Scientometrics, 116(3), 2123–2138.

    Article  Google Scholar 

  • Ortega, J. L. (2019). Blogs and news sources coverage in altmetrics data providers: A comparative analysis by country, language, and subject. Scientometrics. https://doi.org/10.1007/s11192-019-03299-2.

    Article  Google Scholar 

  • Peters, I., Jobmann, A., Eppelin, A., et al. (2014). Altmetrics for large, multidisciplinary research groups: A case study of the Leibniz Association. Libraries in the Digital Age (LIDA) Proceedings, 13, 1–9.

    Google Scholar 

  • Prins, A. A., Costas, R., van Leeuwen, T., & Wouters, P. F. (2016). Using Google Scholar in research evaluation of humanities and social science programs: A comparison with Web of Science data. Research Evaluation, 25(3), 264–270.

    Article  Google Scholar 

  • Su, X. N. (2001). Quality control of data in citation indexes. Journal of Library Science in China, 27(2), 76–78. (in Chinese).

    Google Scholar 

  • Tattersall, A., & Carroll, C. (2018). What can altmetric.com tell us about policy citations of research? An analysis of Altmetric.com data for research articles from the University of Sheffield. Frontiers in Research Metrics and Analytics, 2(9), 1–11.

    Google Scholar 

  • Wang, Q., & Waltman, L. (2016). Large-scale analysis of the accuracy of the journal classification systems of Web of Science and Scopus. Journal of Informetrics, 10(2), 347–364.

    Article  Google Scholar 

  • Wooldridge, J., & King, M. B. (2018). Altmetric scores: An early indicator of research impact. Journal of the Association for Information Science and Technology. https://doi.org/10.1002/asi.24122.

    Article  Google Scholar 

  • Yu, H. (2017). Context of altmetrics data matters: An investigation of count type and user category. Scientometrics, 111(1), 267–283.

    Article  Google Scholar 

  • Yu, H., Cao, X., Xiao, T., & Yang, Z. (2019). Accuracy of policy document mentions: The role of altmetrics databases. In Proceedings of ISSI 2019—The 17th international conference on scientometrics and informetrics (pp. 477–488). Italy: Sapienza University.

  • Zahedi, Z., & Costas, R. (2018). General discussion of data quality challenges in social media metrics: Extensive comparison of four major altmetric data aggregators. PLoS ONE, 13(5), e0197326.

    Article  Google Scholar 

  • Zahedi, Z., Fenner, M., & Costas, R. (2014). How consistent are altmetrics providers? Study of 1000 PLOS ONE publications using the PLOS ALM, Mendeley and Altmetric.com APIs. altmetrics 14. In Workshop at the Web Science conference, Bloomington, USA.

  • Zahedi, Z., & Haustein, S. (2018). On the relationships between bibliographic characteristics of scientific documents and citation and Mendeley readership counts: A large-scale analysis of Web of Science publications. Journal of Informetrics, 12(1), 191–202.

    Article  Google Scholar 

  • Zahedi, Z., Haustein, S., & Bowman, T. (2014). Exploring data quality and retrieval strategies for Mendeley reader counts. In SIG/MET workshop, ASIS&T 2014 annual meeting, Seattle. Retrieved from: www.asis.org/SIG/SIGMET/data/uploads/sigmet2014/zahedi.pdf.

  • Zhao, Q. M. (2009). Analysis of errors in refereces of scientific journals and the prevention strategies. Editorial Friends, 6, 47–49. (in Chinese).

    Google Scholar 

Download references

Acknowledgements

The present study is an extended version of an article (Yu et al. 2019) presented at the 17th International Conference on Scientometrics and Informetrics, Rome (Italy), 2–5 September, 2019. The authors would like to thank Longfei Li and Zihan Yin for helping conduct the coding work. The authors would like to thank Altmetric.com company for providing access to the data and Stacy Konkiel for the useful comments. The research is supported by National Natural Science Foundation of China (NO.71804067), Humanity and Social Science Foundation of Ministry of Education of China (18YJC870023), the Fundamental Research Funds for the Central Universities (No. 30920021203).

Author information

Authors and Affiliations

  1. School of Economics and Management, Nanjing University of Science and Technology, Nanjing, China

    Houqiang Yu, Xueting Cao & Zhenyi Yang

  2. Nanjing Library, Nanjing, China

    Tingting Xiao

Authors
  1. Houqiang Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xueting Cao
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tingting Xiao
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhenyi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Houqiang Yu.

Appendix

Appendix

See Tables 6 and 7; Figs. 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19 and 20.

Table 6 Coding table for type A error (errors found in the policy document)
Full size table
Table 7 Coding table for type B error (errors found in altmetrics database)
Full size table
Fig. 3
figure 3

Example of type A error due to the inaccurate author names of mentioned publication. This error is classified to sub-category 1.1. The Altmetric database was queried in June 2018

Full size image
Fig. 4
figure 4

Example of type A error due to the inaccurate title of mentioned publication. This error is classified to sub-category 1.2. The Altmetric database was queried in June 2018

Full size image
Fig. 5
figure 5

Example of type A error due to the inaccurate publication date of mentioned publication. This error is classified to sub-category 1.3. The Altmetric database was queried in June 2018

Full size image
Fig. 6
figure 6

Example of type A error due to the inaccurate title of source journal. This error is classified to sub-category 1.4. The Altmetric database was queried in June 2018

Full size image
Fig. 7
figure 7

Example of type A error due to that policy document link is updated or expired. This error is classified to sub-category 2.1.1. The Altmetric database was queried in June 2018

Full size image
Fig. 8
figure 8

Example of type B error due to that the policy document page cannot be viewed. This error is classified to sub-category 2.1.2. The Altmetric database was queried in June 2018

Full size image
Fig. 9
figure 9

Example of type B error due to that multiple policy document are provided in the page. This error is classified to sub-category 2.1.3. The Altmetric database was queried in June 2018

Full size image
Fig. 10
figure 10

Example of type B error due to that policy document has mentioned itself. This error is classified to sub-category 2.2.1. The Altmetric database was queried in June 2018

Full size image
Fig. 11
figure 11

The first example of type B error that policy document has not mentioned the scientific product. This error is classified to sub-category 2.2.2. The Altmetric database was queried in June 2018

Full size image
Fig. 12
figure 12

The second example of type B error that policy document has not mentioned the scientific product. This error is classified to sub-category 2.2.2. The Altmetric database was queried in June 2018

Full size image
Fig. 13
figure 13

Example of type B error that title of mentioned scientific product is incorrect in the Altmetric database. This error is classified to sub-category 2.3.1. The Altmetric database was queried in June 2018

Full size image
Fig. 14
figure 14

Example of type B error that author of mentioned scientific product is omitted. This error is classified to sub-category 2.3.2.1. The Altmetric database was queried in June 2018

Full size image
Fig. 15
figure 15

Example of type B error that authors of mentioned scientific product are misspelled. This error is classified to sub-category 2.3.2.2. The Altmetric database was queried in June 2018

Full size image
Fig. 16
figure 16

Example of type B error that author information of mentioned scientific product is duplicated. This error is classified to sub-category 2.3.2.3. The Altmetric database was queried in June 2018

Full size image
Fig. 17
figure 17

Example of type B error that journal title of scientific product is incorrect in altmetrics database. This error is classified to sub-category 2.3.3. The Altmetric database was queried in June 2018

Full size image
Fig. 18
figure 18

Example of type B error that publication date of mentioned publication is omitted. This error is classified to sub-category 2.3.4.1. The Altmetric database was queried in June 2018

Full size image
Fig. 19
figure 19

Example of type B error that publication date of mentioned publication recorded in Altmetric database is inconsistent with the original publication. This error is classified to sub-category 2.3.4.2. The Altmetric database was queried in June 2018

Full size image
Fig. 20
figure 20

Example of type B error that title of policy document recorded in Altmetric database is inconsistent with the source policy document. This error is classified to sub-category 2.3.5. The Altmetric database was queried in June 2018

Full size image

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Yu, H., Cao, X., Xiao, T. et al. How accurate are policy document mentions? A first look at the role of altmetrics database. Scientometrics 125, 1517–1540 (2020). https://doi.org/10.1007/s11192-020-03558-7

Download citation

  • Received:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11192-020-03558-7

Keywords

Search

Navigation