Intelligent systems for assessing competency development in students and graduates of engineering specialisations: expectations of educators, students, and employers

Introduction. Procedures for assessing the quality of graduate training are relevant to both the higher education system and the labour market. Key stakeholders in the educational process seek an independent, objective, and comprehensive assessment of competencies. Aim. The present research aims to establish the expectations of the key participants in the training process within higher education programmes – namely employers, educators, and students – regarding the functionality of in telligent systems for assessing the development of professional and supraprofessional competencies in students and graduates specialising in engineering. Methodology and research methods. Data collection took place in Omsk in 2024 and comprised a questionnaire survey (May to October) and interviews (September to November). The total number of respondents was 41 employers, 44 university lecturers, and 215 students. The interviews were attended by 19 employers and 23 lecturers. Kendall’s τ-b correlation coefficient was used to measure the relationship between the functionality of intelligence systems as reported by employers, lecturers, and students. The theoretical developments of behavioural and functional methodological approaches to studying competencies informed the design of the questionnaire and interview guide. Results. The results indicated that employers, teachers, and students largely agree that the intelligent system should be capable of assessing the developmental levels of both professional and supraprofessional competencies in students and graduates specialising in engineering. However, the ability to assess professional competencies was mentioned more frequently than that of supraprofessional competencies. The participants involved in educational relationships, alongside instrumental procedures for competency assessment, expect intelligent systems to provide technologies that enable the transfer of knowledge, skills, and abilities relevant to the real economy into the educational sphere; the design of practice-oriented educational programmes; and the facilitation of career guidance and effective employment. Scientific novelty. The authors have developed a multi-stakeholder approach to studying the potential of an intelligent system for assessing the competencies of university students and graduates, thereby extending current research practices in this area. The results obtained suggest that intelligent competency assessment systems can serve as a tool to reduce transaction costs within the higher education and labour market systems. Practical significance. The obtained results can be used to design intelligent systems for assessing the competency of students and graduates from universities, encompassing both engineering and educational programmes, as well as other broad groups of training areas and specialities.

1. Gilmanov T.A., Naumov P.Y., Dyachkov A.A. Criteria for assessing the level of formation of professional competence in strategic analysis skills. Nauchnoe mnenie = Scientific Opinion. 2019;3:83–87. (In Russ.) doi:10.25807/PBH.22224378.2019.3.83.87

2. Khorosheva E.R., Zabavnov V.A. Assessment of the level of formation of professional competences of students according to the results of independent assessment of qualifications. Vek kachestva = Century of Quality. 2022;1:253–265. (In Russ.)

3. Chudinsky R.M., Maleva A.A., Duzhakova M.V., Malev V.V., Dubov V.M. Realisation in educational organisations of higher pedagogical education of monitoring and evaluation of students’ professional competences. Mezhdunarodnyj nauchno-issledovatel’skij zhurnal = International Research Journal. 2023;11(137). (In Russ.) doi:10.23670/IRJ.2023.137.68 4

4. . Eliseev I.N., Germanova O.E., Eliseev I.I. Formalisation of students’ competence assessment as a means of providing intellectual support for managerial decision-making. Informatizaciya obrazovaniya i nauki = Informatisation of Education and Science. 2016;2(30):126–135. (In Russ.)

5. Kazarinova N.L. Intellectual analysis in the assessment of the formation of professional competences. Aktual’nye nauchnye issledovaniya v sovremennom mire = Actual Scientific Research in the Modern World. 2021;2-3(70):50–57. (In Russ.)

6. Beresneva Y.V., Britvina V.V. Methodology of development of intellectual automated system for assessing the level of competence formation. Teoriya i praktika proektnogo obrazovaniya = Theory and Practice of Project Education. 2019;3(11):45–50. (In Russ.)

7. Karamzina A.G. Intellectual data analysis for personnel selection by professional competences for the position of system analyst. Sovremennye naukoemkie tekhnologii = Modern Science-Intensive Technologies. 2020;12:56–62. (In Russ.) doi:10.17513/snt.38411

8. Ketoyeva N.L., Znamenskaya M.A., Pavlov E.M., Proshkin N.E. Modelling and program development for the intellectual system of support of personnel management decision making in electric power industry. Inzhenernyj vestnik Dona = Engineering Vestnik Don. 2024;5(113):698–712. (In Russ.)

9. Huang X., Yang F., Zheng J., Feng C., Zhang L. Personalized human resource management via HR analytics and artificial intelligence: theory and implications. Asia Pacific Management Review. 2023;28(4):598–610. doi:10.1016/j.apmrv.2023.04.004

10. Khaperskaya A.V., Minin M.G. Tools of simulation modelling and process management for assessment of professional competences in independent learning. Sovremennye problemy nauki i obrazovaniya = Modern Problems of Science and Education. 2025;1:45. (In Russ.) doi:10.17513/spno.33882

11. Becker J., Becker A., Sałabun W. Construction and use of the ANP decision model taking into account the experts’ competence. Procedia Computer Science. 2017;112:2269–2279. doi:10.1016/j.procs.2017.08.145

12. Meijerink J., Boons M., Keegan A., Marler J. Algorithmic human resource management: synthesizing developments and cross-disciplinary insights on digital HRM. International Journal of Human Resource Management. 2021;32(12):2545–2562. doi:10.1080/09585192.2021.1925326

13. Chowdhury S., Joel-Edgar S., Dey Pr.K., Bhattacharya S., Kharlamov A. Embedding transparency in artificial intelligence machine learning models: managerial implications on predicting and explaining employee turnover. International Journal of Human Resource Management. 2022;34(14):1– 32. doi:10.1080/09585192.2022.2066981

14. Osipenko E.A., Zheng J. Innovative digital tools for assessing the competences of students of the Faculty of Physical Education. Uchenye zapiski universiteta imeni P. F. Lesgafta = Scientific Notes of P. F. Lesgaft University. 2025;1(239):104–110. (In Russ.) doi:10.5930/1994-4683-2025-104-110

15. Bohlouli M., Mittas N., Kakarontzas G., Theodosiou Th., Angelis L., Fathi M. Competence assessment as an expert system for human resource management: a mathematical approach. Expert Systems with Applications. 2017;70:83–102. doi:10.1016/j.eswa.2016.10.046

16. Gorbunovs A., Kapenieks A., Kudina I. Competence development in a combined assessment and collaborative e-portfolio information system. Procedia Computer Science. 2013;26:79–100. doi:10.1016/j.procs.2013.12.009

17. Hamilton R.H., Sodeman W.A. The questions we ask: opportunities and challenges for using big data analytics to strategically manage human capital resources. Business Horizons. 2020;63:85–95. doi:10.1016/j.bushor.2019.10.001

18. Hunkenschroer A.L., Luetge Ch. Ethics of AI-enabled recruiting and selection: a review and research agenda. Journal of Business Ethics. 2022;178(3):977–1007. doi:10.1007/s10551-022-05049-6

19. Qamar Y., Samad T. Human resource analytics: a review and bibliometric analysis. Personnel Review. 2021;51(1):251–283. doi:10.1108/PR-04-2020-0247

20. Esch P.V., Black J.St. Factors that influence new generation candidates to engage with and complete digital, AI-enabled recruiting. Business Horizons. 2019;62(6):729–739. doi:10.1016/j.bushor.2019.07.004

21. Sutadji E., Susilo H., Wibawa A.P., Jabari N.A.M., Rohmad S.N. Authentic assessment implementation in natural and social science. Education Sciences. 2021;11(9):534. doi:10.3390/educsci11090534

22. Sutadji E., Susilo H., Wibawa A.P., Jabari N.A.M., Rohmad S.N. Adaptation strategy of authentic assessment in online learning during the covid-19 pandemic. Journal of Physics: Conference Series. 2021;1810(1). doi:10.1088/1742-6596/1810/1/012059

23. Shikwaya R.O., Amadhila E.P. Authentic assessment’s integrity for online learning in higher education. Global Scientific Journals. 2023;11(3):181–187.

24. Sokhanvar Z., Salehi K., Sokhanva F. Advantages of authentic assessment for improving the learning experience and employability skills of higher education students: a systematic literature review. Studies in Educational Evaluation. 2021;70:101030. doi:10.1016/j.stueduc.2021.101030

25. Heil J., Ifenthaler D. Online assessment in higher education: a systematic review. Online Learning. 2024;27(1):187–218. doi:10.24059/olj.v27i1.3398

26. Mohamed R., Lebar O. Authentic assessment in assessing higher order thinking skills. International Journal of Academic Research in Business and Social Sciences. 2017;7(2):466–476. doi:10.6007/IJARBSS/v7-i2/2021

27. Zhang Q., Zhang X., Liu J. A Holistic review of authentic assessment in mathematics education. In: Zhang Q., Zhang X., Liu J., eds. Authentic Assessment and Evaluation Approaches and Practices in a Digital Era. Cham: Brill Publisher; 2021:95–115. doi:10.1163/9789004501577_005

28. Raynault A., Heilporn G., Mascarenhas A., Denis C. Teaching experiences of E-authentic assessment: lessons learned in higher education. Journal of Teaching and Learning with Technology. 2022;11:3–17. doi:10.14434/jotlt.v11i1.34594

29. Gusyatnikov V.N., Sokolova T.N., Bezrukov A.I., Kayukova I.V. Using artificial intelligence methods to assess competences during testing. Informatika i obrazovanie = Informatics and Education. 2023;38(6):75–85. (In Russ.) doi:10.32517/0234-0453-2023-38-6-75-85

30. Nehaev I.N., Zhuikov I.V., Bastrakov R.V. Architecture of intellectual system for testing the levels of subject competences formation. Kibernetika i programmirovanie = Cybernetics and Programming. 2017;4:41–65. (In Russ.)

31. Pesha A.V., Shavrovskaya M.N., Nikolaeva M.A., Shramko N.V. Existing possibilities of digital assessment of university students’ competences. Vestnik Tyumenskogo gosudarstvennogo universiteta. Social’no-ekonomicheskie i pravovye issledovaniya = Bulletin of Tyumen State University. Socio-Economic and Legal Studies. 2021;3(27):6–19. (In Russ.) doi:10.21684/2411-7897-2021-7-3-6-19

32. Shavrovskaya M.N. Analiz cifrovyh vozmozhnostej ocenki sformirovannosti nadprofessional’nyh kompetencij studentov vuzov = Analyses of digital possibilities for assessing the formation of supraprofessional competences of university students. In: Pesha A.V., Shavrovskaya M.N., Nikolaeva M.A. et al., eds. Razvitie i ocenka nadprofessional’nyh kompetencij studentov universitetov: teoretiko-metodologicheskie osnovy = Development and Assessment of Supraprofessional Competences of University Students: Theoretical and Methodological Foundations. Kazan: Limited Liability Company “Buk”; 2020:188–193. (In Russ.)