Abstract
Finding a set of units that can serve as a reference for growth or improvement in positions within a ranking is not a simple task, since each ranking method can place the same unit in different positions and may even differ in relative position within the ranking. This study proposes a method that applies a combination of network analysis and efficiency methods to global university rankings. Complex networks allow the creation of a graph structure that selects a set of units that change positions in consecutive rankings and also the selection of the set of nodes that are linked with a selected node. For this new set, it is possible to calculate the efficiency level using Data Envelopment Analysis (DEA), from which the benchmarks of the indicators for each of the selected universities can be computed. The purpose of this paper is to develop a methodology to find a set of universities that compete with any university selected from those in the global university rankings, in particular ARWU, THE and QS. Moreover, this work also proposes to estimate the efficiency level of each university that competes with the selected university using the Data Envelopment Analysis methodology in order to establish benchmarks for each of the target Universities. This methodology is replicable for any university in any ranking or set of rankings. Given the high volatility of rankings, this process can serve university policy makers in selecting indicators to focus on for improved results in the short term.
This is a preview of subscription content, log in via an institution to check access.
Source: The authors, 2020. Using R and deaR
Source: Daniel Pimentel Neves, 2020, through the R language using the benchmarking package
Similar content being viewed by others
References
Academic Ranking of World Universities. (2019a). About Academic Ranking of World Universities About ARWU. http://www.shanghairanking.com/aboutarwu.html. Accessed 12 August 2018.
Academic Ranking of World Universities. (2019). Ranking Methodology of Academic Ranking of World Universities—2017. http://www.shanghairanking.com/ARWU-Methodology-2017.html. Accessed 12 August 2018.
Aguillo, I., Bar-Ilan, J., Levene, M., & Ortega, J. (2010). Comparing university rankings. Scientometrics, 85(1), 243–256.
Altmetric Explorer. (2018). Exploring data for all research outputs. https://www.altmetric.com/explorer/highlights. Accessed 12 August 2018.
Barra, C., Lagravinese, R., & Zotti, R. (2018). Does econometric methodology matter to rank universities? An analysis of Italian higher education system. Socio-Economic Planning Sciences, 62, 104–120.
Belloni, J. Â. (2000). Uma metodologia de avaliação da eficiência produtiva de universidades federais brasileiras (Ph.D. Thesis). Universidade Federal de Santa Catarina, Florianópolis.
Boccaletti, V., Latora, Y., Moreno, M., Chavez, M., & Hwang, D. U. (2006). Complex Networks: Structure and dynamics. Physics Reports, 424(2006), 175–308.
Bornmann, L., Mutz, R., & Daniel, H.-D. (2013). Multilevel-statistical reformulation of citation-based university rankings: The Leiden ranking 2011/2012. Journal of the American Society for Information Science and Technology, 64(8), 1649–1658. https://doi.org/10.1002/asi.22857.
Buela-Casal, G., Gutiérrez-Martínez, O., Bermúdez-Sánchez, M. P., & Vadillo-Muñoz, O. (2007). Comparative study of international academic rankings of universities. Scientometrics, 71(3), 349–365. https://doi.org/10.1007/s11192-007-1653-8.
Camargo, B. F., Acero, L. L., & Lacalle, J. G. (2018). Medición de la eficiencia en las bibliotecas de universidades públicas españolas. Revista gesto, 6(1), 89–99.
Casado, F. L. (2007). Análise Envoltória de Dados: conceitos, metodologia e estudo da arte na educação superior. Revista Sociais e Humanas, 20(1), 59–71.
Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision making units. European Journal of Operational Research, 2(6), 429–444. https://doi.org/10.1016/0377-2217(78)90138-8.
Cheng, Y., & Liu, N. C. (2007). Academic ranking of world universities by broad subject fields. Higher education in Europe, 32(1), 17–29.
Chi siamo|Sapienza Università di Roma. (n.d.). https://www.uniroma1.it/it/pagina/chi-siamo. Accessed 03 June 2019
Criado, R., García, E., Pedroche, F., & Romance, M. (2013). A new method for comparing rankings through complex networks: Model and analysis of competitiveness of major European soccer leagues. Chaos An Interdisciplinary Journal of Nonlinear Science, 23(4), 043114.
Dyson, R. G., Allen, R., Camanho, A. S., Podinovski, V. V., Sarrico, C. S., & Shale, E. A. (2001). Pitfalls and protocols in DEA. European Journal of Operational Research, 132(2), 245–259.
ETER—European Tertiary Education Register. (2018). https://www.eter-project.com/#/home. Accessed 17 September 2019.
Fontoura, L. F. (2018). What is a complex network? (CDT-2), (PREPRINT). https://doi.org/10.13140/RG.2.2.10450.04804/1. Accessed June 23, 2020
García-Zorita, C., Rousseau, R., Marugan-Lazaro, S., & Sanz-Casado, E. (2018). Ranking dynamics and volatility. Journal of Informetrics, 12(3), 567–578.
Guironnet, J.-P., & Peypoch, N. (2018). The geographical efficiency of education and research: The ranking of US universities. Socio-Economic Planning Sciences, 62, 44–55.
Higher Education Statistics Agency. (n.d.). HE Student Data|HESA. https://www.hesa.ac.uk/data-and-analysis/students/table-1%3E. Accessed June 03. 2019.
Huang, M.-H. (2011). A comparison of three major academic rankings for world universities: from a research evaluation perspective, 26.
InCites–Clarivate Analytics–Sign In. (n.d.). https://error.incites.clarivate.com/error/Error?DestApp = IC2&Error = IPError&Params = DestApp%3DIC2&RouterURL = https%3A%2F%2Flogin.incites.clarivate.com%2F&Domain = .clarivate.com&Src = IP&Alias = IC2. Accessed 17 January 2020.
Incities. (2019). Analytics Web of Science. https://incites.clarivate.com/#/explore/0/region/. Accessed 20 de March de 2019.
Johnes, J. (2006). Data envelopment analysis and its application to the measurement of efficiency in higher education. Economics of education Review, 25(3), 273–288.
Johnes, J., & Li, Y. U. (2008). Measuring the research performance of Chinese higher education institutions using data envelopment analysis. China Economic Review, 19(4), 679–696. https://doi.org/10.1016/j.chieco.2008.08.004.
Kempkes, Gerhard, & Pohl, Carsten. (2010). The efficiency of German universities–some evidence from nonparametric and parametric methods. Applied Economics, 42(16), 2063–2079.
Kosztyán, Z. T., Banász, Z., Csányi, V. V., et al. (2019). Rankings or leagues or rankings on leagues? Ranking in fair reference groups. Tertiary Education and Management, 25, 289–310. https://doi.org/10.1007/s11233-019-09028-x.
Liu, N. C., & Cheng, Y. (2005). The Academic Ranking of World Universities. Higher Education in Europe, 30(2), 127–136. https://doi.org/10.1080/03797720500260116.
Liu, N. C., Cheng, Y., & Liu, L. (2005). Academic ranking of world universities using scientometrics-A comment to the “Fatal Attraction”. Scientometrics, 64(1), 101–109. https://doi.org/10.1007/s11192-005-0241-z.
Lo, W. Y. W. (2014). University rankings: Implications for higher education in Taiwan. Singapore: Springer Science & Business Media.
Marcovitch, J., Goldemberg, J., Axel-Berg, J., Righetti, S., dos Santos, S. M., & de Oliveira, L. N., et al. (2018). Repensar a universidade: desempenho acadêmico e comparações internacionais. Universidade de São Paulo. Escola de Comunicações e Artes. https://doi.org/10.11606/9788571661868
Marginson, S. (2007). Global university rankings: Implications in general and for Australia. Journal of Higher Education Policy and Management, 29(2), 131–142.
Musselin, C. (2018). New forms of competition in higher education1. Socio-Economic Review, 16(3), 657–683. https://doi.org/10.1093/ser/mwy033.
Newman, M. E. J. (2003). The structure and function of complex networks. Siam Review., 45(2), 167–256.
US News. (2019aa). Nanjing University. https://www.usnews.com/education/best-global-universities/nanjing-university-503849. Accessed 03 June 2019
US News. (2019b). Zhejiang University. https://www.usnews.com/education/best-global-universities/zhejiang-university-50477. Accessed 03 June 2019.
Pastor, J. T., Ruiz, J. L., & Sirvent, I. (2002). A statistical test for nested radial DEA models. Operations Research, 50(4), 728–735.
Pedroche Sánchez, F. (2010). Competitivity groups on social network sites. Mathematical and Computer Modelling, 52(7), 1052–1057. https://doi.org/10.1016/j.mcm.2010.02.031.
Privy Council Officce. (n.d.). Higher Education. https://www.privy-council.org.uk/output/Page27.asp. Accessed 12 August 2018
Quacquarelli Symonds. (2019). QS World University Rankings Report. https://www.topuniversities.com/student-info/qs-guides/qs-world-university-rankings-2019-supplement/thank-you?field_guide_category_tid=3600. Accessed 12 August 2018
IREG Observatory on Academic Ranking and Excellence. (n.d.). https://ireg-observatory.org/en/. Accessed 17 August 2020.
IREG Observatory on Academic Ranking and Excellence (2015). Pautas para grupos de interés de rankings académicos. http://ireg-observatory.org/en/pdfy/IREG-Guidelines_Spanish.pdf. Accessed 12 August 2018.
Raponi, V., Martella, F., & Maruotti, A. (2016). A biclustering approach to university performances: an Italian case study. Journal of Applied Statistics, 43(1), 31–45.
Réka Albert and Albert-László Barabási. (2002). Statistical mechanics of complex networks. Reviews of Modern Physics, 74, 47.
Rhaiem, M. (2017). Measurement and determinants of academic research efficiency: a systematic review of the evidence. Scientometrics, 110(2), 581–615. https://doi.org/10.1007/s11192-016-2173-1.
Safón, V. (2013). What do global university rankings really measure? The search for the X factor and the X entity. Scientometrics, 97(2), 223–244. https://doi.org/10.1007/s11192-013-0986-8.
Sapienza Universita Di Roma. (2017). Bilancio unico di ateneo d’esercizio 2017. https://www.uniroma1.it/sites/default/files/field_file_allegati/1_bilancio_unico_di_ateneo_di_esercizio_2017.pdf. Accessed 03 June 2019
Sapienza Universita Di Roma. Chi siamo. Disponível em: https://www.uniroma1.it/it/pagina/chi-siamo. Accessed 03 June 2019
Sassaki, A., Luque, C. A., Goldemberg, J., Oliveira, L. N. de, Ranieri, N., Toneto Jr., R., et al. (2017). Universidade em movimento: memória de uma crise. (J. Marcovitch, Ed.). Com-Arte Editora Laboratório do Curso de Editoração ECA-USP; Fapesp. https://doi.org/10.11606/9788571661325
Sherman, H. D., & Zhu, J. (2006). Service productivity management: Improving service performance using data envelopment analysis (DEA). Springer, New York.. https://doi.org/10.1007/0-387-33231-6.
Sungkyunkwan University. (n.d.). About SKKU. https://www.skku.edu/eng/. Accessed 03 June 2019
Tel Aviv University. (2018). Facts and Figures. https://international.tau.ac.il/Facts_and_Figures. Accessed 03 Junes 2019
Texas A&M Universtity. (2018). Annual Financial Report. https://apps.system.tamus.edu/afrdisclaimer/disclaimer.aspx?report=SOBAFY2018TAMUSCOMBO. Accessed 03 June 2019
Texas A&M Universtity. (2019). Enrollment Profile. https://dars.tamu.edu/Student/Enrollment-Profile. Accessed 03 June 2019.
Texas A&M Universtity. (n.d.). Faculty Salary Study Nine Month Salaries AAUDE Data. https://dars.tamu.edu/Faculty/files/aaude-faculty-salary-fall-2017_Final.aspx. Accessed 03 June 2019
The University of Adelaide. (2018). 2018 Pocket Statistics. https://www.adelaide.edu.au/planning/statistics/pocket-stats/2018pocket_stats.pdf. Accessed 03 June 2019
The University of Iowa. (2018). Financial Report. July 1, 2017, to June 30, 2018. https://afr.fo.uiowa.edu/sites/afr.fo.uiowa.edu/files/Financial_Report_2018_for_web.pdf. Accessed 03 June 2019
The University of Iowa. (n.d.). About Iowa. https://uiowa.edu/about. Accessed 03 June 2019
Toloo, M., & Tichý, T. (2015). Two alternative approaches for selecting performance measures in data envelopment analysis. Measurement, 65, 29–40. https://doi.org/10.1016/j.measurement.2014.12.043.
Tuesta, E. F., García-Zorita, C., Romera Ayllon, R., & Sanz-Casado, E. (2019). Does a country/region’s economic status affect its universities’ presence in international rankings? Journal of Data and Information Science, 4(2), 56–78. https://doi.org/10.2478/jdis-2019-0009.
Universidade de São Paulo. (n.d.). Anuário Estatístico da USP. https://uspdigital.usp.br/anuario/AnuarioControle. Accessed 03 June 2019
Universidade de São Paulo. (n.d.). Apoio ao pesquisador. http://www.sibi.usp.br/apoio-pesquisador/indicadores-pesquisa/incites. Accessed 20 May 2019
Universitat Innsbruck. (2018). At a Glance. https://www.uibk.ac.at/universitaet/profil/dokumente/uni-facts-and-figures-2018.pdf. Accessed 03 June 2019
Université Catholique de Louvain. (2015). Rapport annuel. https://cdn.uclouvain.be/public/Exports%20reddot/ac-arec/images/Rapport_annuel_2013_2014.pdf. Accessed 03 June 2020
Université de Strasburg. (2019). Chiffres clés. https://www.unistra.fr/index.php?id=26613#c120909. Accessed 03 June 2019
University of Barcelona. (2019). Report on the academic year 2017-2018. http://www.ub.edu/gtr/documents/memories/mem1718_en.pdf#pagemode=bookmarks. Accessed 03 June 2019
University of Calgary (2018). Annual Report for the year ended March 21, 2017. https://www.ucalgary.ca/sites/default/files/ucgy/groups/Marketing/2016-17%20Annual%20Report.pdf. Accessed 03 June 2019.
University of Gothenburg (2018). Facts and Figures. https://www.gu.se/english/about_the_university/Introduction/facts-figures. Accessed 03 June 2019.
University of Science and Technology of China. (2018). USTC Introducion. https://en.ustc.edu.cn/2011/0113/c5394a50095/page.htm. Accessed 03 June 2019
University of Science and Technology of China (n.d.). Departmental Budget 2018. http://xxgk.ustc.edu.cn/_upload/article/files/12/e5/5c88edf348349c0942174399b0e4/f5e9acfd-67c9-44cd-9644-e3a75fd594c7.pdf. Accessed 03 June 2019
van Raan, A. F. J. (2005). Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics, 62(1), 133–143. https://doi.org/10.1007/s11192-005-0008-6.
Vincke, P. (2009). University Rankings. In C. Dehon, D. Jacobs, & C. Vermandele (Eds.), Ranking Universities (pp. 11–26). Bruxelas: Editions de L’Universite de Bruxelles.
Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E. C., Tijssen, R. J., van Eck, N. J., et al. (2012). The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation. Journal of the American Society for Information Science and Technology, 63(12), 2419–2432.
Wilensky, U., & Rand, W. (2015). An introduction to agent-based modeling: modeling natural, social, and engineered complex systems with NetLogo. Cambridge: MIT Press Ltd.
World Intellectual Property Organization. (n.d.) Patentscope, the user’s guide. https://www.wipo.int/edocs/pubdocs/en/patents/434/wipo_pub_l434_08.pdf. Accessed 12 August 2018
World University Rankings 2018 methodology. (2017, August 29). Times Higher Education (THE). https://www.timeshighereducation.com/world-university-rankings/methodology-world-university-rankings-2018. Accessed 12 August 2020
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Tuesta, E.F., Bolaños-Pizarro, M., Neves, D.P. et al. Complex networks for benchmarking in global universities rankings. Scientometrics 125, 405–425 (2020). https://doi.org/10.1007/s11192-020-03637-9
Received:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11192-020-03637-9