Global and national university rankings: A multivariate analysis

Introduction. In the context of globalisation and internationalisation of higher education, university rankings are becoming an important tool for assessing the quality of education received by students at various higher education institutions around the world. These processes actualise the issues of possibilities for practical use of methodologies for calculation of global and national university rankings.
The aim of the study was to develop and apply a methodological approach to multivariate the analysis of Higher Education Institutions (HEIs) classification procedures, to construct and analyse aggregated indicators for global and national rating systems of higher education organisations, and to assess the relationship between them.
Methodology and research methods. The current paper presents a system analysis of databases of rating systems and an aggregation of independent evaluations of global and national rankings of HEIs using the methodology of league table analysis based on mathematical apparatus of the voting theory. The dependence of global and national university rankings indicators was investigated using correlation, cluster, factor, regression (linear and polynomial) and dispersion methods of analysis.
Results and scientific novelty. A comprehensive comparative analysis of ranking systems and their results was carried out. The authors solved the problem of aggregating multiple heterogeneous studies of global and national ranking systems with their qualitative and quantitative variety of criteria, indicators and methods of assessment. The correlation between the indicators of aggregated global and national rankings was revealed; the regression dependence of the integral national ranking with the results of the leading global rating systems was determined.
Practical significance. The developed methodical approach is a convenient and effective mechanism for comprehensive monitoring of the members of educational process.

1. Soh K. What the overall doesn’t tell about world university rankings: Examples from ARWU, QSWUR, and THEWUR in 2013. Journal of Higher Education Policy and Management. 2015; 37 (3): 295–307. DOI: 10.1080/1360080X.2015.1035523

2. Marconi G., Ritzen J. Determinants of international university rankings scores. Applied Economics. 2015; 47 (7): 6211–6227. DOI: 10.1080/00036846.2015.1068921

3. Huang Z., Qiu R. A quantitative and model-driven approach to assessing higher education in the United States of America. Quality in Higher Education. 2016; 22 (1): 78–95. DOI: 10.1080/13538322.2016.1147215

4. Peters M. A. Global university rankings: Metrics, performance, governance. Educational Philosophy and Theory. 2019; 51 (1): 5–13. DOI: 10.1080/00131857.2017.1381472

5. Motova G., Navodnov V. Twenty years of accreditation in Russian higher education: Lessons learnt. Higher Education Evaluation and Development. 2020; 14 (1): 33–51. DOI: 10.1108/HEED-05-2019-0023

6. Salimova T. A., Biryukova L. I., Shilkina A. T., Khakhaleva E. V. Towards a methodology of sustainable competitiveness of organization. International Journal of Civil Engineering and Technology [Internet]. 2018 [cited 2021 Mar 22]; 9 (11): 161–172. Available from: https://www.elibrary.ru/item.asp?id=36518513&

7. Hossain M. N., Zabed Ahmed S. M. Use of scholarly communication and citation-based metrics as a basis for university ranking in developing country perspective. Global Knowledge, Memory and Communication. 2019; 69 (6/7): 461–482. DOI: 10.1108/GKMC-09-2019-0112

8. Berbegal-Mirabent J., Ribeiro-Soriano D. E. Behind league tables and ranking systems: A critical perspective of how university quality is measured. Journal of Service Theory and Practice. 2013; 25 (3): 242–266. DOI: 10.1108/JSTP-04-2013-0059

9. Kusumastuti D., Idrus N. Nurturing quality of higher education through national ranking: A potential empowerment model for developing countries. Quality in Higher Education. 2017; 23 (3): 230–248. DOI: 10.1080/13538322.2017.1407400

10. Jin J. C., Kim D. A. Higher education in South Korea: recent changes in school competitiveness and research productivity. Journal of Asian Public Policy. 2019; 14 (7144): 1–23. DOI: 10.1080/17516234.2019.1622181

11. Donetskaia S. S. The Berlin principles in Russian university rankings. Problems of Economic Transition. 2017; 59 (7–9): 614–626. DOI: 10.1080/10611991.2017.1394756

12. Köse M. F., Korkmaz M. Why are some universities better? An evaluation in terms of organizational culture and academic performance. Higher Education Research & Development. 2019; 38 (6): 1213–1226. DOI: 10.1080/07294360.2019.1634679

13. Hosier M., Hoolash K. A. The effect of methodological variations on university rankings and associated decision-making and policy. Studies in Higher Education. 2019; 44 (1): 201–214. DOI: 10.1080/03075079.2017.1356282

14. Rybinski K. Are rankings and accreditation related? Examining the dynamics of higher education in Poland. Quality Assurance in Education. 2020; 28 (3): 193–204. DOI: 10.1108/QAE-03-2020-0032

15. Doğan G., Al U. Is it possible to rank universities using fewer indicators? A study on five international university rankings. Aslib Journal of Information Management. 2019; 71 (1): 18–37. DOI: 10.1108/AJIM-05-2018-0118

16. Hauptman Komotar M. Discourses on quality and quality assurance in higher education from the perspective of global university rankings. Quality Assurance in Education. 2020; 28 (1): 78–88. DOI: 10.1108/QAE-05-2019-0055

17. Gao X., Zheng Y. ‘Heavy mountains’ for Chinese humanities and social science academics in the quest for world-class universities. Compare: Journal of Comparative and International Education. 2018; 50 (1): 1–19. DOI: 10.1080/03057925.2018.1538770

18. Shattock M. The ‘world class’ university and international ranking systems: what are the policy implications for governments and institutions? Policy Reviews in Higher Education. 2017; 1 (1): 4–21. DOI: 10.1080/23322969.2016.1236669

19. Jabnoun N. The influence of wealth, transparency, and democracy on the number of top ranked universities. Quality Assurance in Education. 2015; 23 (2): 108–122. DOI: 10.1108/QAE-07-2013-0033

20. Huang M. Exploring the h-index at the institutional level: A practical application in world university rankings. Online Information Review. 2012; 36 (4): 534–547. DOI: 10.1108/14684521211254059

21. Van Raan A. F. J. Fatal attraction: Conceptual and methodological problems in the ranking of universities by bibliometric methods. Scientometrics. 2005; 62 (1): 133–143. DOI: 10.1007/s11192-005-0008-6

22. Ioannidis J. P. A., et al. International ranking systems for universities and institutions: A critical appraisal. BMC Medicine [Internet]. 2007 [cited 2021 Mar 22]. Available from: http://www.biomedcentral.com/1741-7015/5/30

23. Holmes R. The THES university rankings: Are they really world class. Asian Journal of University Education [Internet]. 2006 [cited 2021 Mar 22]; 1 (1): 1–14. Available from: https://www.researchgate.net/publication/301781569_The_THES_University_Rankings_Are_They_Really_World_Class

24. Waltman L., et al. The Leiden Ranking 2011/2012: Data collection, indicators, and interpretation [Internet]. 2012 [cited 2021 Mar 22]. Available from: http://arxiv.org/abs/1202.3941

25. Billaut J.-C., Bouyssou D., Vincke P. Should you believe in the Shanghai ranking? Scientometrics [Internet]. 2010; 84 (1): 237–263. DOI: 10.1007/s11192-009-0115-x

26. Van Raan A. F. J. Challenges in ranking of universities. In: Invited Paper for the First International Conference on World Class Universities [Internet]. Jaio Tong University. Shanghai; 2005 June [cited 2021 Mar 22]. p. 19–50. Available from: https://www.researchgate.net/publication/250794968_Challenges_in_Ranking_of_Universities

27. Zitt M., Filliatreau G. Big is (made) beautiful: Some comments about the Shanghai ranking of world-class universities. In: Sadlack J., Liu N. C. (eds.). The w orld-class university and ranking: Aiming beyond status [Internet]. Shanghai, China: Cluj University Press; 2007 [cited 2021 Mar 22]. p. 17–24. Available from: https://www.researchgate.net/publication/252727031_Big_is_made_Beautiful_Some_comments_about_the_Shanghai_ranking_of_world-class_universities

28. Moed H. F. Bibliometric rankings of world universities. In: CWTS Report 2006-01 [Internet]. 2006 [cited 2021 Mar 22]. Available from: http://www.cwts.nl/hm/bibl_rnk_wrld_univ_full.pdf

29. Soh K. The seven deadly sins of world university ranking: A summary from several papers. Journal of Higher Education Policy and Management. 2017; 39 (1): 104–115. DOI: 10.1080/1360080X.2016.1254431

30. Soh K. Rectifying an honest error in world university rankings: A solution to the problem of indicator weight discrepancies. Journal of Higher Education Policy and Management. 2013; 35 (6): 574–585. DOI: 10.1080/1360080X.2013.844670

31. Sheeja N. K., Susan Mathew K., Surendran Cherukodan. Impact of scholarly output on university ranking. Global Knowledge, Memory and Communication. 2018; 67 (3): 154–165. DOI: 10.1108/GKMC-11-2017-0087

32. McCoy C. G., Nelson M. L., Weigle M. C. Mining the Web to approximate university rankings. Information Discovery and Delivery. 2018; 46 (3): 173–183. DOI: 10.1108/IDD-05-2018-0014

33. Qureshi M. S., Daud A., Hayat M. K., Afzal M. T. OpenRank – a novel approach to rank universities using objective and publicly verifiable data sources. Library Hi Tech. 2021. DOI: 10.1108/LHT-07-2019-0131

34. Brusca I., Cohen S., Manes-Rossi F., Nicolò G. Intellectual capital disclosure and academic rankings in European universities: Do they go hand in hand? Meditari Accountancy Research. 2019; 28 (1): 51–71. DOI: 10.1108/MEDAR-01-2019-0432

35. Jin J. C., Kim D. Higher education in South Korea: Recent changes in school competitiveness and research productivity. Journal of Asian Public Policy. 2019; 14 (3): 291–313. DOI: 10.1080/17516234.2019.1622181

36. Tutterow C., Evans J. A. Reconciling the small effect of rankings on university performance with the transformational cost of conformity. The University Under Pressure. 2016; 46: 265–301. DOI: 10.1108/S0733-558X20160000046009

37. Salimova T., Guskova N., Neretina E. Education for sustainable development in Russia: problems and challenges. International Journal of Innovation and Sustainable Development. 2015; 9 (3/4): 246–261. DOI: 10.1504/IJISD.2015.071855

38. Marconi G., Ritzen J. Determinants of international university rankings scores. Applied Economics. 2015. 47 (57): 6211–6227. DOI: 10.1080/00036846.2015.1068921

39. Zhu Y. Social media engagement and Chinese international student recruitment: understanding how UK HEIs use Weibo and WeChat. Journal of Marketing for Higher Education. 2019; 29 (2): 173–190. DOI: 10.1080/08841241.2019.1633003

40. Çakır M. P., Acartürk C., Alkan S., Akbulut U. Multi-authoring and its impact on university rankings: A case study of CERN effect on Turkish universities. Studies in Higher Education. 2019; 44 (6): 1052–1068. DOI: 10.1080/03075079.2017.1414780

41. Paruolo P., Saltelli A., Saisana M. Ratings and rankings: Voodoo or science? Journal of the Royal Statistical Society. Series A (Statistics in Society) [Internet]. 2011 [cited 2021 Mar 22]; 176 (3): 609–634. Available from: https://www.researchgate.net/publication/51887919_Ratings_and_rankings_Voodoo_or_Science

42. Angelis L., Bassiliades N., Manolopoulos Y. On the necessity of multiple university rankings. Journal of Scientometrics and Information Management. 2019; 13 (1): 11–36. DOI: 10.1080/09737766.2018.1550043

43. Hosier M., Hoolash B. K. The effect of methodological variations on university rankings and associated decision-making and policy. Studies in Higher Education. 2019; 44 (1): 201–214. DOI: 10.1080/03075079.2017.1356282

44. Hallinger P. Riding the tiger of world university rankings in East Asia: where are we heading? International Journal of Educational Management. 2014; 28 (2): 230–245. DOI: 10.1108/IJEM-11-2012-0126

45. Tofallis С. A different approach to university rankings. Higher Education. 2012; 63 (1): 1–18. DOI: 10.1007/s10734-011-9417-z

46. Nethal K. J., Harrison J. World university ranking systems: An alternative approach using partial least squares path modelling. Journal of Higher Education Policy and Management. 2014; 36 (5): 471–482. DOI: 10.1080/1360080X.2014.936090

47. Luque-Martínez T., Faraoni N. Meta-ranking to position world universities. Studies in Higher Education. 2020; 45 (4): 819–833. DOI: 10.1080/03075079.2018.1564260

48. Erganova N. E., Kolyasnikova L. V. Rating creation for professional educational organizations based on the item response theory. Obrazovanie i nauka = The Education and Science Journal. 2016; 4: 46–64. DOI: 10.17853/1994-5639-2016-4-46-64 (In Russ.)

49. Yurin A. Y. Group decision-making methods for adapting solutions derived from case-based reasoning. Scientific and Technical Information Processing. 2015; (42): 375–381. DOI: 10.3103/S014768821505010X

50. Navodnov V. G., Motova G. N., Ryzhakova O. Е. Comparison of international rankings and results of Russian monitoring of HEIs effectiveness by the methodology of league analysis. Voprosy obrazovaniya = Educational Studies. Moscow [Internet]. 2019 [cited 2021 Mar 22]; 3: 130–151. Available from: https://vo.hse.ru/data/2019/09/16/1541237741/05%20 Navodnov.pdf (In Russ.)

51. Fursov V. V. Аbout the possibility for medical universities to enter world university rankings. Obrazovanie i nauka = The Education and Science Journal. 2017 [cited 2021 Mar 22]; 19 (5): 138–152. Available from: https://doi.org/10.17853/1994-5639-2017-5-138-152 (In Russ.)