Multimodal learning analytics: a bibliometric and ontological analysis

Introduction. Multimodal Learning Analytics (MMLA) is an emerging research domain in education that has garnered global attention. Its significance lies in its potential to offer a more comprehensive and accurate understanding of learning processes by integrating diverse data types, including digital, physical, physiological, psychological, psychometric, and environmental data.

Aim. This research aims to move beyond a descriptive analysis of current practices to develop a structural understanding of the entities and relationships that constitute the research field. This involves refining the boundaries of MMLA as a scientific discipline to identify unexplored areas with potential applications. Particular attention is given to collaborative analytics, a promising area focused on studying data related to joint activities.

Methodology and research methods. Bibliometric analysis was employed as the primary method. Additionally, fractal matrix table analysis was used to gain an ontological understanding of the field.

Results and scientific novelty. The bibliometric analysis enabled the tracing of major developmental milestones in MMLA from its inception to the present day. Key research groups were identified, along with their thematic focuses and preferred data sources. Dominant research themes were extracted, and their evolution over time was analysed. Shifts in research interests revealed a transition from analysing individual learning trajectories to studying group dynamics within collaborative learning contexts. The study also examined the application of MMLA to various forms of collective learning activities, such as collaborative problem solving, group-based tasks, and project-based learning. Ontological modelling of the field facilitated the identification of existing conceptual frameworks and methodological approaches, as well as the projection of emerging directions.

Practical significance. The research findings can be used to design learning environments that foster communication, collaboration, and teamwork skills, including those in interdisciplinary educational contexts.

1. Giannakos M., Spikol D., Di Mitri D., Sharma K., Ochoa X., Hammad R. Introduction to multimodal learning analytics. In: The Multimodal Learning Analytics Handbook. Cham: Springer International Publishing; 2022:3–28. doi:10.1007/978-3-030-65604-1_1

2. Worsley M., Ochoa X. Towards Collaboration Literacy Development through Multimodal Learning Analytics. Accessed June 05, 2025. https://tiilt.northwestern.edu/assets/papers/towards_collaboration_literacy_2020.pdf

3. Di Mitri D., Schneider J., Specht M., Drachsler H. From signals to knowledge: a conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning. 2018;34:338–349. doi:10.1111/jcal.12288

4. Fei N., Lu Z., Gao Y., Yang G., Huo Y., Wen J., et al. Towards artificial general intelligence via a multimodal foundation model. Nature Communications. 2022;13(1):3094. doi:10.1038/s41467-022-30761-2

5. Reddy A.S. Multimodal gen AI: integrating text, image, and video analysis for comprehensive claims assessment. ESP International Journal of Advancements in Computational Technology. 2024;2(2):133– 141. doi:10.56472/25838628/IJACT-V2I2P117

6. Febriantoro W., Gauthier A., Cukurova M. The promise of physiological data in collaborative learning: a systematic literature review. In: European Conference on Technology Enhanced Learning. Cham: Springer Nature Switzerland; 2023:75–88. doi:10.1007/978-3-031-42682-7_6

7. Chejara P., Prieto L.P., Dimitriadis Y., Rodríguez-Triana M.J., Ruiz-Calleja A., Kasepalu R., et al. The impact of attribute noise on the automated estimation of collaboration quality using multimodal learning analytics in authentic classrooms. Journal of Learning Analytics. 2024;11(2):73–90. doi:10.18608/jla.2024.8253

8. Giannakos M., Cukurova M. The role of learning theory in multimodal learning analytics. British Journal of Educational Technology. 2023;54(5):1246–1267. doi:10.1111/bjet.13320

9. Mangaroska K., Sharma K., Giannakos M., Trætteberg H., Dillenbourg P. Gaze-driven design insights to amplify debugging skills: a learner-centred analysis approach. Journal of Learning Analytics. 2018;5(3):98–119. doi:10.18608/jla.2018.53.7

10. Larmuseau C., Cornelis J., Lancieri L., Desmet P., Depaepe F. Multimodal learning analytics to investigate cognitive load during online problem solving. British Journal of Educational Technology. 2020;51(5):1548–1562. doi:10.1111/bjet.12958

11. Lee-Cultura S., Sharma K., Giannakos M. Children’s play and problem-solving in motion-based learning technologies using a multi-modal mixed methods approach. International Journal of Child-Computer Interaction. 2022;31:100355. doi:10.1016/j.ijcci.2021.100355

12. Andrade A., Danish J.A., Maltese A.V. A measurement model of gestures in an embodied learning environment: accounting for temporal dependencies. Journal of learning Analytics. 2017;4(3):18–46. doi:10.18608/jla.2017.43.3

13. Kosmas P., Ioannou A., Retalis S. Moving bodies to moving minds: a study of the use of motion-based games in special education. TechTrends. 2018;62(6):594–601. doi:10.1007/s11528-018-0294-5

14. Oviatt S., Lin J., Sriramulu A. I know what you know: what hand movements reveal about domain expertise. ACM Transactions on Interactive Intelligent Systems. 2021;11(1):1–26. doi:10.1145/3423049

15. Pardos Z.A., Rosenbaum L.F., Abrahamson D. Characterizing learner behavior from touchscreen data. International Journal of Child-Computer Interaction. 2022;31:100357. doi:10.1016/j.ijcci.2021.100357

16. Amos B., Ludwiczuk B., Satyanarayanan M. Openface: a general-purpose face recognition library with mobile applications. CMU School of Computer Science. 2016;6(2):20. Accessed April 23, 2025. https://www.academia.edu/85057925/OpenFace_A_general_purpose_face_recognition_library_with_mobile_applications

17. Prieto L.P., Sharma K., Kidzinski Ł., Rodríguez-Triana M.J., Dillenbourg P. Multimodal teaching analytics: automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning. 2018;34:193–203. doi:10.1111/jcal.12232

18. Martinez-Maldonado R., Kay J., Buckingham Shum S., Yacef K. Collocated collaboration analytics: principles and dilemmas for mining multimodal interaction data. Human-Computer Interaction. 2019;34(1):1–50. doi:10.1080/07370024.2017.1338956

19. Yan L., Martinez-Maldonado R., Zhao L., Deppeler J., Corrigan D., Gašević D. How do teachers use open learning spaces? Mapping from teachers’ socio-spatial data to spatial pedagogy. In: LAK22: 12th International Learning Analytics and Knowledge Conference. New York, NY, USA: Association for Computing Machinery; 2022:87–97. doi:10.1145/3506860.3506872

20. Cukurova M., Zhou Q., Spikol D., Landolfi L. Modelling collaborative problem-solving competence with transparent learning analytics: Is video data enough? Proceedings of the Tenth International Conference on Learning Analytics & Knowledge. New York, NY, USA: Association for Computing Machinery; 2020:270–275. doi:10.1145/3375462.3375484

21. Jacobson M.J., Levin J.A., Kapur M. Education as a complex system: conceptual and methodological implications. Educational Researcher. 2019;48(2):112–119. doi:10.3102/0013189X19826958

22. Blikstein P., Abrahamson D., Wilensky U. The classroom as a complex adaptive system: an agentbased framework to investigate students’ emergent collective behaviors. In: Kanselaar G., Jonker V., Kirschner P.A., Prins F.J., eds. International Perspectives in the Learning Sciences: Cre8ing a learning world. Proceedings of the Eighth International Conference for the Learning Sciences – ICLS 2008, Vol. 3. Utrecht, The Netherlands: International Society of the Learning Sciences; 2008:12–13. doi:10.22318/icls2008.3.12

23. Knight B. The classroom as a complex adaptive system (CAS): credible framing, useful metaphor or mis-designation? International Journal of Complexity in Education. 2022;3(1). doi:10.26262/ijce.v3i1.9457

24. Mu S., Cui M., Huang X. Multimodal data fusion in learning analytics: a systematic review. Sensors. 2020;20(23):6856. doi:10.3390/s20236856

25. Shankar S.K., Prieto L.P., Rodríguez-Triana M.J., Ruiz-Calleja A. A review of multimodal learning analytics architectures. In: IEEE 18th International Conference on Advanced Learning Technologies (ICALT). Mumbai, India; 2018:212–214. doi:10.1109/ICALT.2018.00057

26. Worsley M., Martinez-Maldonado R. Multimodal learning analytics’ past, present, and potential futures. CrossMMLA@ LAK. 2018;2.

27. Kutuzov A.I., Bogdanova A.V. Assessment tools for students’ group and teamwork in higher education: a systematic literature review. Vysshee obrazovanie v Rossii = Higher Education in Russia. 2025; 34(4):118–143. (In Russ.) doi:10.31992/0869-3617-2025-34-4-118-143

28. Donthu N., Kumar S., Mukherjee D., Pandey N., Lim W. M.How to conduct a bibliometric analysis: an overview and guidelines. Journal of Business Research. 2021;133:285–296. doi:10.1016/j.jbusres.2021.04.070

29. Jeong H., Néda Z., Barabási A.L. Measuring preferential attachment in evolving networks. Europhysics Letters. 2003;61(4):567. doi:10.1209/epl/i2003-00166-9

30. Kordonsky S. Veernye matricy kak instrument postroenija ontologij = Fractal Matrix Tables as a Tool for Ontologies’ Creation. Washington: South Eastern Publishing House; 2011. 66 p. (In Russ.) Accessed April 23, 2025. https://relteam.ru/projgit/fractal-matrix.pdf

31. Chebanov S.V. Cognitive graphics as a way of representing ideas. METOD: Moskovskij ezhegodnik trudov iz obshhestvovedcheskih discipline = METHOD: Moscow Annual of Works from Social Science Disciplines. 2020;10:309–376. (In Russ.) doi:10.31249/metod/2020.10.16

32. Patarakin E.D. Igrovoe pole vychislitel’noj didaktiki = The playing field of computational didactics. In: Yarmakhov B.B., ed. Sovremennaja “cifrovaja” didaktika = Modern “Digital” Didactics. Moscow: Publishing House Green Print LLC; 2022:35–70. (In Russ.) Accessed June 05, 2025. https://publications.hse.ru/pubs/share/direct/854781314.pdf

33. Effendi D.N., Anggraini W., Jatmiko A., Rahmayanti H., Ichsan I.Z., Rahman M.M. Bibliometric analysis of scientific literacy using VOS viewer: analysis of science education. Journal of Physics: Conference Series. 2021;1796(1):012096. doi:10.1088/1742-6596/1796/1/012096

34. Malakhov V.A. Bibliometric analysis as a method of scientific research: opportunities and limitations. Naukovedcheskie issledovanija = Science Studies. 2022;(1):212–227. (In Russ.) doi:10.31249/scis/2022.01.10

35. Ryzhkova E. Praktiki analiza kachestvennyh dannyh v social’nyh naukah. Otv. red. E. V. Poluhina. M.: Izdatel’skij Dom VSHE; 2023 = Practices of qualitative data analysis in social sciences. Ex. ed. E.V. Polukhina. Moscow: HSE Publishing House, 2023. Puti Rossii = The Journal of Russian Social Research and Ethnography. 2024;2(1):267–271. (In Russ.) Accessed April 23, 2025. https://publications.hse.ru/pubs/share/direct/810942174.pdf

36. Blikstein P. Using learning analytics to assess students’ behavior in open-ended programming tasks. In: Proceedings of the 1st International Conference on Learning Analytics and Knowledge. New York: The Association for Computing Machinery; 2011:110–116. doi:10.1145/2090116.2090132

37. Worsley M., Blikstein P. What’s an expert? Using learning analytics to identify emergent markers of expertise through automated speech, sentiment and sketch analysis. In: EDM 2011 – Proceedings of the 4th International Conference on Educational Data Mining. Eindhoven, The Netherlands; 2011:234–239. Accessed April 23, 2025. https://www.academia.edu/23019716/Whats_an_Expert_Using_learning_analytics_to_identify_emergent_markers_of_expertise_through_automated_speech_sentiment_and_sketch_analysis

38. Worsley M. Multimodal learning analytics: enabling the future of learning through multimodal data analysis and interfaces. In: Proceedings of the 14th ACM International Conference on Multimodal Interaction (ICMI ‘12). New York, NY, USA: Association for Computing Machinery; 2012:353–356. doi:10.1145/2388676.2388755

39. Blikstein P. Multimodal learning analytics. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. New York, NY, USA: Association for Computing Machinery; 2013:102–106. doi:10.1145/2460296.2460316

40. Worsley M., Blikstein P. Towards the development of multimodal action based assessment. In: Proceedings of the Third International Conference on Learning Analytics and Knowledge. New York, NY, USA: Association for Computing Machinery; 2013:94–101. doi:10.1145/2460296.2460315

41. Schneider B., Pea R. The effect of mutual gaze perception on students’ verbal coordination. In: Proceedings of the 7th International Conference on Educational Data Mining (EDM 2014). 2014:138–144. Accessed June 05, 2025. http://life-slc.org/docs/LSLC_rp_A206-Schneider-Pea_EDM-2014-Full.pdf

42. Worsley M., Scherer S., Morency L.-P., Blikstein P. Exploring behavior representation for learning analytics. In: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction (ICMI ‘15). New York, NY, USA: Association for Computing Machinery; 2015:251–258. doi:10.1145/2818346.2820737

43. Worsley M., Martinez-Maldonado R., D’Angelo C. A new era in multimodal learning analytics: twelve core commitments to ground and grow MMLA. Journal of Learning Analytics. 2021;8(3)10–27. doi:10.18608/jla.2021.7361

44. Alwahaby H., Cukurova M., Papamitsiou Z., Giannakos M. The evidence of impact and ethical considerations of multimodal learning analytics: a systematic literature review. In: Giannakos M., Spikol D., Di Mitri D., Sharma K., Ochoa X., Hammad R., eds. The Multimodal Learning Analytics Handbook. Cham: Springer; 2022:289–325. doi:10.1007/978-3-031-08076-0_12

45. Worsley M., Abrahamson D., Blikstein P., Grover S., Schneider B., Tissenbaum M. Workshop: situating multimodal learning analytics. In: Looi C.-K., Polman J. L., Cress U., Reimann P., eds. “Transforming Learning, Empowering Learners”, Proceedings of the International Conference of the Learning Sciences (ICLS 2016). Vol. 3. Singapore: International Society of the Learning Sciences; 2016:1346–1349. Accessed April 23, 2025. https://www.academia.edu/83862566/Context_aware_Multimodal_Learning_Analytics_Taxonomy

46. Rodríguez-Triana M.J., Prieto L., Martínez-Monés A., Asensio-Pérez J., Dimitriadis Y. The teacher in the loop: customizing multimodal learning analytics for blended learning. In: Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18). New York, NY, USA: Association for Computing Machinery; 2018:417–426. doi:10.1145/3170358.3170364

47. Di Mitri D., Schneider J., Klemke R., Specht M., Drachsler H. Read between the lines: an annotation tool for multimodal data for learning. In: Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK ’19) New York, NY, USA: Association for Computing Machinery; 2019:51–60. doi:10.1145/3303772.3303776

48. Martinez-Maldonado R., Echeverria V., Fernandez G., Buckingham Shum S. From data to insights: a layered storytelling approach for multimodal learning analytics. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI ‘20). New York, NY, USA: Association for Computing Machinery; 2020:1–15. doi:10.1145/3313831.3376148

49. Mitri D., Schneider J., Trebing K., Sopka S., Specht M., Drachsler H. Real-time multimodal feedback with the CPR tutor. In: Artificial Intelligence in Education: 21st International Conference, AIED 2020. Proceedings. Part I; July 6–10, 2020; Ifrane, Morocco. Berlin, Heidelberg: Springer-Verlag; 2020:141–152. doi:10.1007/978-3-030-52237-7_12

50. Echeverria V., Martinez-Maldonado R., Buckingham Shum S. Towards collaboration translucence: giving meaning to multimodal group data. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ‘19). New York, NY, USA: Association for Computing Machinery; 2019:1–16. doi:10.1145/3290605.3300269

51. Cukurova M., Giannakos M., Martinez-Maldonado R. The promise and challenges of multimodal learning analytics. British Journal of Educational Technology. 2020;51(5):1441–1449. doi:10.1111/bjet.13015

52. Dindar M., Järvelä S., Haataja E. What does physiological synchrony reveal about metacognitive experiences and group performance? British Journal of Educational Technology. 2020;51(5):1577–1596. doi:10.1111/bjet.12981

53. Dindar M., Malmber J., Jaervelae S., Haataja E., Kirschner P.A., et al. Matching self-reports with electrodermal activity data: investigating temporal changes in self-regulated learning. Education and Information Technologies. 2020;25:1785–1802. doi:10.1007/s10639-019-10059-5

54. Ouhaichi H., Spikol D., Vogel B. Rethinking MMLA: design considerations for multimodal learning analytics systems. In: Proceedings of the Tenth ACM Conference on Learning @ Scale (L@S ‘23). New York, NY, USA: Association for Computing Machinery; 2023:354–359. doi:10.1145/1234567890

55. Yan L., Martinez-Maldonado R., Cordoba B.G., Deppeler J., Corrigan D., Nieto G.F., Gašević D. Footprints at school: modelling in-class social dynamics from students’ physical positioning traces. In: LAK21: 11th International Learning Analytics and Knowledge Conference. New York, NY, USA: Association for Computing Machinery; 2021:43–54. doi:10.1145/3448139.3448144

56. Martinez-Maldonado R., Schulte J., Echeverria V., Gopalan Y., Buckingham Shum S. Where is the teacher? Digital analytics for classroom proxemics. Journal of Computer Assisted Learning. 2020;36:741–762. doi:10.1111/jcal.12444

57. Martínez-Maldonado R., Yan L., Deppeler J., Phillips M., Gašević D. Classroom analytics: telling stories about learning spaces using sensor data. In: Gil E., Mor Y., Dimitriadis Y., Köppe C., eds. Hybrid Learning Spaces, Understanding Teaching-Learning Practice. Cham: Springer International Publishing; 2022:185–203. doi:10.1007/978-3-030-88520-5_11

58. Emerson A., Henderson N., Min W., Rowe J., Minogue J., Lester J. Multimodal trajectory analysis of visitor engagement with interactive science museum exhibits. In: Artificial Intelligence in Education: 22nd International Conference, AIED 2021. Proceedings, Part II; June 14–18, 2021; Utrecht, the Netherlands. Berlin, Heidelberg: Springer-Verlag; 2021:151–155. doi:10.1007/978-3-030-78270-2_27

59. Emerson A., Min W., Rowe J., Azevedo R., Lester J. Multimodal predictive student modeling with multi-task transfer learning. In: LAK23: 13th International Learning Analytics and Knowledge Conference. New York, NY, USA: Association for Computing Machinery; 2023:333–344. doi:10.1145/3576050.3576101

60. Lester J. AI and the future of education. In: Proceedings of the 29th ACM International Conference on Multimedia, MM ‘21. Association for Computing Machinery. New York, NY, USA: Association for Computing Machinery; 2021:3642. doi:10.1145/3474085.3475715

61. Schwendimann B.A., Rodríguez-Triana M.J., Vozniuk A. L., Prieto P., Shirvani Boroujeni M., Holzer A., et al. Perceiving learning at a glance: a systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies. 2017;10:30–41. doi:10.1109/TLT.2016.2599522

62. Blikstein P., Worsley M. Multimodal learning analytics and education data mining: using computational technologies to measure complex learning tasks. Journal of Learning Analytics. 2016;3:220– 238. doi:10.18608/jla.2016.32.11

63. Worsley M. Framing the future of multimodal learning analytics. In: Giannakos M., Spikol D., Di Mitri D., Sharma K., Ochoa X., Hammad R., eds. The Multimodal Learning Analytics Handbook. Cham: Springer; 2022:359–369. doi:10.1007/978-3-031-08076-0_14

64. Ochoa X. Multimodal learning analytics: Rationale, process, examples, and direction. In: Lang C., Siemens G., Wise A., Gašević D., Merceron A., eds. Handbook of Learning Analytics. Vancouver, BC: SoLAR; 2022:54–65. doi:10.18608/hla22.006

65. Ochoa X., Lang A.C., Siemens G. Multimodal learning analytics. In: Lang C., Siemens G., Wise A., Gašević D., eds. Handbook of Learning Analytics. Vancouver, BC: SoLAR; 2017;1:129–141. doi:10.18608/hla17

66. Zhou Q., Suraworachet W., Cukurova M. Detecting non-verbal speech and gaze behaviours with multimodal data and computer vision to interpret effective collaborative learning interactions. Education and Information Technologies. 2023;29:1071–1098. doi:10.1007/s10639-023-12315-1

67. Di Mitri D., Scheffel M., Drachsler H., Börner D., Ternier S., Specht M. Learning pulse: a machine learning approach for predicting performance in self-regulated learning using multimodal data. In: Proceedings of the Seventh International Learning Analytics & Knowledge Conference, LAK ‘17. New York, NY, USA: Association for Computing Machinery; 2017:188–197. doi:10.1145/3027385.3027447

68. Sharma K., Giannakos M. Multimodal data capabilities for learning: what can multimodal data tell us about learning? British Journal of Educational Technology. 2020;51:1450–1484. doi:10.1111/bjet.12993

69. Sharma K., Papamitsiou Z., Giannakos M. Building pipelines for educational data using AI and multimodal analytics: a “grey-box” approach. British Journal of Educational Technology. 2019;50:3004– 3031. doi:10.1111/bjet.12854

70. Fernandez-Nieto G.M., Echeverria V., Buckingham Shum S., Mangaroska K., Kitto K., Palominos E., Martinez-Maldonado R. Storytelling with learner data: guiding student reflection on multimodal team data. IEEE Transactions on Learning Technologies. 2021;14(5):695–708. doi:10.1109/TLT.2021.3131842.3

71. Ochoa X., Wise A.F. Supporting the shift to digital with student-centered learning analytics. Educational Technology Research and Development. 2021;69(1):357–361. doi:10.1007/s11423-020-09882-2

72. Parandekar S., Patarakin E., Yayla G. A modern aspect of instrumental literacy: coding. In: Dobryakova M., Froumin I., Barannikov K., Moss G., Remorenko I., Hautamäki J., eds. Key Competences and New Literacies: From Slogans to School Reality. UNIPA Springer Series. Cham: Springer; 2023:367– 390. doi:10.1007/978-3-031-23281-7_13