Мультимодальная учебная аналитика: библиометрический и онтологический анализ

Введение. Мультимодальная учебная аналитика (MMLA) – новое направление исследований в образовании, интерес к которому растет во всем мире. Актуальность такой аналитики заключается в возможности более комплексного и точного понимания процессов обучения за счет интеграции различных типов данных, таких как цифровые, физические, физиологические, психологические, психометрические и экологические (данные окружающей среды).

Цель – выйти за рамки описательного анализа текущих практик и перейти к структурному пониманию сущностей и взаимосвязей, формирующих исследовательское поле; уточнить границы MMLA как научного направления с целью выявления скрытых областей потенциального применения. Особое внимание уделяется коллаборативной аналитике, перспективному направлению изучения данных о совместной деятельности.

Методология, методы и методики. В качестве основного метода применяется библиометрический анализ. Для онтологического осмысления поля использован метод веерных матриц.

Результаты и научная новизна. Анализ полученных библиометрических данных позволил проследить основные вехи развития мультимодальной учебной аналитики с момента ее появления до настоящего времени. Определены основные исследовательские группы, содержание их исследований и используемые источники данных. Выделены основные исследовательские темы и проанализирована их динамика. Обнаружен сдвиг исследовательского интереса: от анализа индивидуальных траекторий к анализу групповой динамики в контексте совместного обучения. Изучены возможности применения MMLA для анализа коллективных форм учебной деятельности, таких как совместное решение задач, групповая работа или проектное обучение. Онтологическое осмысление поля MMLA позволило выделить существующие пространства и подходы и предположить те, которые могут появиться в будущем.

Практическая значимость. Результаты могут быть использованы для проектирования учебных сред, в том числе ориентированных на формирование навыков коммуникации, сотрудничества и работы в команде, а также междисциплинарных контекстах.

1. Giannakos M., Spikol D., Di Mitri D., Sharma K., Ochoa X., Hammad R. Introduction to multimodal learning analytics. In: The Multimodal Learning Analytics Handbook. Cham: Springer International Publishing; 2022:3–28. doi:10.1007/978-3-030-65604-1_1

2. Worsley M., Ochoa X. Towards Collaboration Literacy Development through Multimodal Learning Analytics. Accessed June 05, 2025. https://tiilt.northwestern.edu/assets/papers/towards_collaboration_literacy_2020.pdf

3. Di Mitri D., Schneider J., Specht M., Drachsler H. From signals to knowledge: a conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning. 2018;34:338–349. doi:10.1111/jcal.12288

4. Fei N., Lu Z., Gao Y., Yang G., Huo Y., Wen J., et al. Towards artificial general intelligence via a multimodal foundation model. Nature Communications. 2022;13(1):3094. doi:10.1038/s41467-022-30761-2

5. Reddy A.S. Multimodal gen AI: integrating text, image, and video analysis for comprehensive claims assessment. ESP International Journal of Advancements in Computational Technology. 2024;2(2):133– 141. doi:10.56472/25838628/IJACT-V2I2P117

6. Febriantoro W., Gauthier A., Cukurova M. The promise of physiological data in collaborative learning: a systematic literature review. In: European Conference on Technology Enhanced Learning. Cham: Springer Nature Switzerland; 2023:75–88. doi:10.1007/978-3-031-42682-7_6

7. Chejara P., Prieto L.P., Dimitriadis Y., Rodríguez-Triana M.J., Ruiz-Calleja A., Kasepalu R., et al. The impact of attribute noise on the automated estimation of collaboration quality using multimodal learning analytics in authentic classrooms. Journal of Learning Analytics. 2024;11(2):73–90. doi:10.18608/jla.2024.8253

8. Giannakos M., Cukurova M. The role of learning theory in multimodal learning analytics. British Journal of Educational Technology. 2023;54(5):1246–1267. doi:10.1111/bjet.13320

9. Mangaroska K., Sharma K., Giannakos M., Trætteberg H., Dillenbourg P. Gaze-driven design insights to amplify debugging skills: a learner-centred analysis approach. Journal of Learning Analytics. 2018;5(3):98–119. doi:10.18608/jla.2018.53.7

10. Larmuseau C., Cornelis J., Lancieri L., Desmet P., Depaepe F. Multimodal learning analytics to investigate cognitive load during online problem solving. British Journal of Educational Technology. 2020;51(5):1548–1562. doi:10.1111/bjet.12958

11. Lee-Cultura S., Sharma K., Giannakos M. Children’s play and problem-solving in motion-based learning technologies using a multi-modal mixed methods approach. International Journal of Child-Computer Interaction. 2022;31:100355. doi:10.1016/j.ijcci.2021.100355

12. Andrade A., Danish J.A., Maltese A.V. A measurement model of gestures in an embodied learning environment: accounting for temporal dependencies. Journal of learning Analytics. 2017;4(3):18–46. doi:10.18608/jla.2017.43.3

13. Kosmas P., Ioannou A., Retalis S. Moving bodies to moving minds: a study of the use of motion-based games in special education. TechTrends. 2018;62(6):594–601. doi:10.1007/s11528-018-0294-5

14. Oviatt S., Lin J., Sriramulu A. I know what you know: what hand movements reveal about domain expertise. ACM Transactions on Interactive Intelligent Systems. 2021;11(1):1–26. doi:10.1145/3423049

15. Pardos Z.A., Rosenbaum L.F., Abrahamson D. Characterizing learner behavior from touchscreen data. International Journal of Child-Computer Interaction. 2022;31:100357. doi:10.1016/j.ijcci.2021.100357

16. Amos B., Ludwiczuk B., Satyanarayanan M. Openface: a general-purpose face recognition library with mobile applications. CMU School of Computer Science. 2016;6(2):20. Accessed April 23, 2025. https://www.academia.edu/85057925/OpenFace_A_general_purpose_face_recognition_library_with_mobile_applications

17. Prieto L.P., Sharma K., Kidzinski Ł., Rodríguez-Triana M.J., Dillenbourg P. Multimodal teaching analytics: automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning. 2018;34:193–203. doi:10.1111/jcal.12232

18. Martinez-Maldonado R., Kay J., Buckingham Shum S., Yacef K. Collocated collaboration analytics: principles and dilemmas for mining multimodal interaction data. Human-Computer Interaction. 2019;34(1):1–50. doi:10.1080/07370024.2017.1338956

19. Yan L., Martinez-Maldonado R., Zhao L., Deppeler J., Corrigan D., Gašević D. How do teachers use open learning spaces? Mapping from teachers’ socio-spatial data to spatial pedagogy. In: LAK22: 12th International Learning Analytics and Knowledge Conference. New York, NY, USA: Association for Computing Machinery; 2022:87–97. doi:10.1145/3506860.3506872

20. Cukurova M., Zhou Q., Spikol D., Landolfi L. Modelling collaborative problem-solving competence with transparent learning analytics: Is video data enough? Proceedings of the Tenth International Conference on Learning Analytics & Knowledge. New York, NY, USA: Association for Computing Machinery; 2020:270–275. doi:10.1145/3375462.3375484

21. Jacobson M.J., Levin J.A., Kapur M. Education as a complex system: conceptual and methodological implications. Educational Researcher. 2019;48(2):112–119. doi:10.3102/0013189X19826958

22. Blikstein P., Abrahamson D., Wilensky U. The classroom as a complex adaptive system: an agentbased framework to investigate students’ emergent collective behaviors. In: Kanselaar G., Jonker V., Kirschner P.A., Prins F.J., eds. International Perspectives in the Learning Sciences: Cre8ing a learning world. Proceedings of the Eighth International Conference for the Learning Sciences – ICLS 2008, Vol. 3. Utrecht, The Netherlands: International Society of the Learning Sciences; 2008:12–13. doi:10.22318/icls2008.3.12

23. Knight B. The classroom as a complex adaptive system (CAS): credible framing, useful metaphor or mis-designation? International Journal of Complexity in Education. 2022;3(1). doi:10.26262/ijce.v3i1.9457

24. Mu S., Cui M., Huang X. Multimodal data fusion in learning analytics: a systematic review. Sensors. 2020;20(23):6856. doi:10.3390/s20236856

25. Shankar S.K., Prieto L.P., Rodríguez-Triana M.J., Ruiz-Calleja A. A review of multimodal learning analytics architectures. In: IEEE 18th International Conference on Advanced Learning Technologies (ICALT). Mumbai, India; 2018:212–214. doi:10.1109/ICALT.2018.00057

26. Worsley M., Martinez-Maldonado R. Multimodal learning analytics’ past, present, and potential futures. CrossMMLA@ LAK. 2018;2.

27. Кутузов А.И., Богданова А.В. Инструменты оценки групповой и командной работы студентов в высшем образовании: систематический обзор литературы. Высшее образование в России. 2025;34(4):118–143. doi:10.31992/0869-3617-2025-34-4-118-143

28. Donthu N., Kumar S., Mukherjee D., Pandey N., Lim W. M.How to conduct a bibliometric analysis: an overview and guidelines. Journal of Business Research. 2021;133:285–296. doi:10.1016/j.jbusres.2021.04.070

29. Jeong H., Néda Z., Barabási A.L. Measuring preferential attachment in evolving networks. Europhysics Letters. 2003;61(4):567. doi:10.1209/epl/i2003-00166-9

30. Кордонский С. Веерные матрицы как инструмент построения онтологий. Вашингтон: Издательство «Юго-Восток»; 2011. 66 с. Режим доступа: https://relteam.ru/projgit/fractal-matrix.pdf (дата обращения: 23.04.2025).

31. Чебанов С.В. Когнитивная графика как способ изображения идей. МЕТОД: Московский ежегодник трудов из обществоведческих дисциплин. 2020;10:309–376. doi:10.31249/metod/2020.10.16

32. Патаракин Е.Д. Игровое поле вычислительной дидактики. Современная «цифровая» дидактика. В книге : Современная «цифровая» дидактика. / Отв. ред.: Б. Б. Ярмахов. М.: ООО «Грин Принт», 2022. С. 35–70. Режим доступа: https://publications.hse.ru/pubs/share/direct/854781314.pdf (дата обращения: 05.06.2025).

33. Effendi D.N., Anggraini W., Jatmiko A., Rahmayanti H., Ichsan I.Z., Rahman M.M. Bibliometric analysis of scientific literacy using VOS viewer: analysis of science education. Journal of Physics: Conference Series. 2021;1796(1):012096. doi:10.1088/1742-6596/1796/1/012096

34. Малахов В.А. Библиометрический анализ как метод науковедческих исследований: возможности и ограничения. Науковедческие исследования. 2022;(1):212–227. doi:10.31249/scis/2022.01.10

35. Рыжкова Е. [Рец.] Практики анализа качественных данных в социальных науках. Отв. ред. Е. В. Полухина. М.: Издательский Дом ВШЭ, 2023. Пути России. 2024;2(1):267–271. Режим доступа: https://publications.hse.ru/pubs/share/direct/810942174.pdf (дата обращения: 23.04.2025).

36. Blikstein P. Using learning analytics to assess students’ behavior in open-ended programming tasks. In: Proceedings of the 1st International Conference on Learning Analytics and Knowledge. New York: The Association for Computing Machinery; 2011:110–116. doi:10.1145/2090116.2090132

37. Worsley M., Blikstein P. What’s an expert? Using learning analytics to identify emergent markers of expertise through automated speech, sentiment and sketch analysis. In: EDM 2011 – Proceedings of the 4th International Conference on Educational Data Mining. Eindhoven, The Netherlands; 2011:234–239. Accessed April 23, 2025. https://www.academia.edu/23019716/Whats_an_Expert_Using_learning_analytics_to_identify_emergent_markers_of_expertise_through_automated_speech_sentiment_and_sketch_analysis

38. Worsley M. Multimodal learning analytics: enabling the future of learning through multimodal data analysis and interfaces. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI ‘12). New York, NY, USA: Association for Computing Machinery; 2012:353–356. doi:10.1145/2388676.2388755

39. Blikstein P. Multimodal learning analytics. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. New York, NY, USA: Association for Computing Machinery; 2013:102–106. doi:10.1145/2460296.2460316

40. Worsley M., Blikstein P. Towards the development of multimodal action based assessment. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. New York, NY, USA: Association for Computing Machinery; 2013:94–101. doi:10.1145/2460296.2460315

41. Schneider B., Pea R. The effect of mutual gaze perception on students’ verbal coordination. In: Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014). 2014:138–144. Accessed June 05, 2025. http://life-slc.org/docs/LSLC_rp_A206-Schneider-Pea_EDM-2014-Full.pdf

42. Worsley M., Scherer S., Morency L.-P., Blikstein P. Exploring behavior representation for learning analytics. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI ‘15). New York, NY, USA: Association for Computing Machinery; 2015:251–258. doi:10.1145/2818346.2820737

43. Worsley M., Martinez-Maldonado R., D’Angelo C. A new era in multimodal learning analytics: twelve core commitments to ground and grow MMLA. Journal of Learning Analytics. 2021;8(3)10–27. doi:10.18608/jla.2021.7361

44. Alwahaby H., Cukurova M., Papamitsiou Z., Giannakos M. The evidence of impact and ethical considerations of multimodal learning analytics: a systematic literature review. In: Giannakos M., Spikol D., Di Mitri D., Sharma K., Ochoa X., Hammad R., eds. The Multimodal Learning Analytics Handbook. Cham: Springer; 2022:289–325. doi:10.1007/978-3-031-08076-0_12

45. Worsley M., Abrahamson D., Blikstein P., Grover S., Schneider B., Tissenbaum M. Workshop: situating multimodal learning analytics. In: Looi C.-K., Polman J. L., Cress U., Reimann P., eds. “Transforming Learning, Empowering Learners”, Proceedings of the International Conference of the Learning Sciences (ICLS 2016). Vol. 3. Singapore: International Society of the Learning Sciences; 2016:1346– 1349. Accessed April 23, 2025. https://www.academia.edu/83862566/Context_aware_Multimodal_Learning_Analytics_Taxonomy

46. Rodríguez-Triana M.J., Prieto L., Martínez-Monés A., Asensio-Pérez J., Dimitriadis Y. The teacher in the loop: customizing multimodal learning analytics for blended learning. In: Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18). New York, NY, USA: Association for Computing Machinery; 2018:417–426. doi:10.1145/3170358.3170364

47. Di Mitri D., Schneider J., Klemke R., Specht M., Drachsler H. Read between the lines: an annotation tool for multimodal data for learning. In: Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK ’19) New York, NY, USA: Association for Computing Machinery; 2019:51–60. doi:10.1145/3303772.3303776

48. Martinez-Maldonado R., Echeverria V., Fernandez G., Buckingham Shum S. From data to insights: a layered storytelling approach for multimodal learning analytics. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ‘20). New York, NY, USA: Association for Computing Machinery; 2020:1–15. doi:10.1145/3313831.3376148

49. Mitri D., Schneider J., Trebing K., Sopka S., Specht M., Drachsler H. Real-time multimodal feedback with the CPR tutor. In: Artificial Intelligence in Education: 21st International Conference, AIED 2020. Proceedings. Part I; July 6–10, 2020; Ifrane, Morocco. Berlin, Heidelberg: Springer-Verlag; 2020:141–152. doi:10.1007/978-3-030-52237-7_12

50. Echeverria V., Martinez-Maldonado R., Buckingham Shum S. Towards collaboration translucence: giving meaning to multimodal group data. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ‘19). New York, NY, USA: Association for Computing Machinery; 2019:1–16. doi:10.1145/3290605.3300269

51. Cukurova M., Giannakos M., Martinez-Maldonado R. The promise and challenges of multimodal learning analytics. British Journal of Educational Technology. 2020;51(5):1441–1449. doi:10.1111/bjet.13015

52. Dindar M., Järvelä S., Haataja E. What does physiological synchrony reveal about metacognitive experiences and group performance? British Journal of Educational Technology. 2020;51(5):1577–1596. doi:10.1111/bjet.12981

53. Dindar M., Malmber J., Jaervelae S., Haataja E., Kirschner P.A., et al. Matching self-reports with electrodermal activity data: investigating temporal changes in self-regulated learning. Education and Information Technologies. 2020;25:1785–1802. doi:10.1007/s10639-019-10059-5

54. Ouhaichi H., Spikol D., Vogel B. Rethinking MMLA: design considerations for multimodal learning analytics systems. In: Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S ‘23). New York, NY, USA: Association for Computing Machinery; 2023:354–359. doi:10.1145/1234567890

55. Yan L., Martinez-Maldonado R., Cordoba B.G., Deppeler J., Corrigan D., Nieto G.F., Gašević D. Footprints at school: modelling in-class social dynamics from students’ physical positioning traces. In: LAK21: 11th International Learning Analytics and Knowledge Conference. New York, NY, USA: Association for Computing Machinery; 2021:43–54. doi:10.1145/3448139.3448144

56. Martinez-Maldonado R., Schulte J., Echeverria V., Gopalan Y., Buckingham Shum S. Where is the teacher? Digital analytics for classroom proxemics. Journal of Computer Assisted Learning. 2020;36:741–762. doi:10.1111/jcal.12444

57. Martínez-Maldonado R., Yan L., Deppeler J., Phillips M., Gašević D. Classroom analytics: telling stories about learning spaces using sensor data. In: Gil E., Mor Y., Dimitriadis Y., Köppe C., eds. Hybrid Learning Spaces, Understanding Teaching-Learning Practice. Cham: Springer International Publishing; 2022:185–203. doi:10.1007/978-3-030-88520-5_11

58. Emerson A., Henderson N., Min W., Rowe J., Minogue J., Lester J. Multimodal trajectory analysis of visitor engagement with interactive science museum exhibits. In: Artificial Intelligence in Education: 22nd International Conference, AIED 2021. Proceedings, Part II; June 14–18, 2021; Utrecht, the Netherlands. Berlin, Heidelberg: Springer-Verlag; 2021:151–155. doi:10.1007/978-3-030-78270-2_27

59. Emerson A., Min W., Rowe J., Azevedo R., Lester J. Multimodal predictive student modeling with multi-task transfer learning. In: LAK23: 13th International Learning Analytics and Knowledge Conference. New York, NY, USA: Association for Computing Machinery; 2023:333–344. doi:10.1145/3576050.3576101

60. Lester J. AI and the future of education. In: Proceedings of the 29th ACM International Conference on Multimedia, MM ‘21. Association for Computing Machinery. New York, NY, USA: Association for Computing Machinery; 2021:3642. doi:10.1145/3474085.3475715

61. Schwendimann B.A., Rodríguez-Triana M.J., Vozniuk A. L., Prieto P., Shirvani Boroujeni M., Holzer A., et al. Perceiving learning at a glance: a systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies. 2017;10:30–41. doi:10.1109/TLT.2016.2599522

62. Blikstein P., Worsley M. Multimodal learning analytics and education data mining: using computational technologies to measure complex learning tasks. Journal of Learning Analytics. 2016;3:220– 238. doi:10.18608/jla.2016.32.11

63. Worsley M. Framing the future of multimodal learning analytics. In: Giannakos M., Spikol D., Di Mitri D., Sharma K., Ochoa X., Hammad R., eds. The Multimodal Learning Analytics Handbook. Cham: Springer; 2022:359–369. doi:10.1007/978-3-031-08076-0_14

64. Ochoa X. Multimodal learning analytics: Rationale, process, examples, and direction. In: Lang C., Siemens G., Wise A., Gašević D., Merceron A., eds. Handbook of Learning Analytics. Vancouver, BC: SoLAR; 2022:54–65. doi:10.18608/hla22.006

65. Ochoa X., Lang A.C., Siemens G. Multimodal learning analytics. In: Lang C., Siemens G., Wise A., Gašević D., eds. Handbook of Learning Analytics. Vancouver, BC: SoLAR; 2017;1:129–141. doi:10.18608/hla17

66. Zhou Q., Suraworachet W., Cukurova M. Detecting non-verbal speech and gaze behaviours with multimodal data and computer vision to interpret effective collaborative learning interactions. Education and Information Technologies. 2023;29:1071–1098. doi:10.1007/s10639-023-12315-1

67. Di Mitri D., Scheffel M., Drachsler H., Börner D., Ternier S., Specht M. Learning pulse: a machine learning approach for predicting performance in self-regulated learning using multimodal data. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, LAK ‘17. New York, NY, USA: Association for Computing Machinery; 2017:188–197. doi:10.1145/3027385.3027447

68. Sharma K., Giannakos M. Multimodal data capabilities for learning: what can multimodal data tell us about learning? British Journal of Educational Technology. 2020;51:1450–1484. doi:10.1111/bjet.12993

69. Sharma K., Papamitsiou Z., Giannakos M. Building pipelines for educational data using AI and multimodal analytics: a “grey-box” approach. British Journal of Educational Technology. 2019;50:3004– 3031. doi:10.1111/bjet.12854

70. Fernandez-Nieto G.M., Echeverria V., Buckingham Shum S., Mangaroska K., Kitto K., Palominos E., Martinez-Maldonado R. Storytelling with learner data: guiding student reflection on multimodal team data. IEEE Transactions on Learning Technologies. 2021;14(5):695–708. doi:10.1109/TLT.2021.3131842.3

71. Ochoa X., Wise A.F. Supporting the shift to digital with student-centered learning analytics. Educational Technology Research and Development. 2021;69(1):357–361. doi:10.1007/s11423-020-09882-2

72. Parandekar S., Patarakin E., Yayla G. A modern aspect of instrumental literacy: coding. In: Dobryakova M., Froumin I., Barannikov K., Moss G., Remorenko I., Hautamäki J., eds. Key Competences and New Literacies: From Slogans to School Reality. UNIPA Springer Series. Cham: Springer; 2023:367– 390. doi:10.1007/978-3-031-23281-7_13