Problems of analysis of the relationship between the learning context and TIMSS testing results

Introduction. For more than two decades, the Organisation for Economic Cooperation and Development has been organising a number of comparative studies of the quality of education in different countries. One of them is the study of mathematics and science education TIMSS (Trends in International Mathematics and Science Study), which is conducted jointly with the International Association for the Evaluation of Educational Achievements (IEA). The last, seventh, cycle of the study was conducted in 2019. TIMSS statistics is regularly posted in the public domain on the IEA website so that specialists can independently conduct research in any aspect of their interest. One of the areas of analysis in this case is traditionally the search for the causes of certain test results, which are determined by the peculiarities of the organisation of the educational process and the context of learning in different countries. At the same time, a study of professional literature showed that among the factors of the social and school context, the analysis of which is provided by the research tools, only an extremely limited range of them turned out to be statistically directly related to TIMSS scores. Specialists systematically encounter inexplicable absence or low correlation values of TIMSS test scores and context indicators. The authors think that the main reason for such difficulties is inattention to the peculiarities of the indicators used in the calculation of measures of statistical relationships. Aim. The present research aims to identify a statistical relationship between test results and indicators of the context of schoolchildren’s learning, as well as the influence of the TIMSS information collection and processing system on the productivity of analysing the research results.

Methodology and research methodology. The methodological basis of the work is a systematic approach, which is based on the consideration of the results of the international TIMSS study as a whole: i.e. a complex of interrelated elements (organisations, tools, assessment indicators, scoring systems). The work was carried out on the basis of applied research procedures (observation, description, comparison, measurement, etc.), within which general scientific (comparative analysis, systematisation, generalisation) and statistical research methods (statistical and correlation analysis, etc.) were also used. The source of information was the International Database of Electronic Testing TIMSS-2019, hosted in the IEA repository. The TIMSS datasets were analysed using the IEA International Database (IDB) parser plug-in for SPSS (version 4.0).

Results. For most indicators of the social and student context of learning, the authors found the absence or low value of statistical relationships with TIMSS scores. The number of books at home and parents’ education turned out to be statistically related to TIMSS scores concerning the indicators of social well-being and home learning conditions envisaged by the organisers. The indicators of learning conditions at school included the frequency of independent work in class; motivational factors included plans to continue education and self-evaluation of students’ math proficiency. Evidently, even these relationships turned out to be weak. It was revealed that the difficulties in detecting a correlation between TIMSS scores and learning conditions are caused by the very nature of the analysed variables: 1. the approximate nature of individual student assessments used in TIMSS; 2. low differentiation of students according to a number of indicators of the learning context; 3. insufficient reliability of information obtained from sociological surveys of schoolchildren.

Practical significance. The authors believe that in order to improve the quality of analytical work on relevant topics, it is necessary to pay close attention to the essence behind the variables used in statistical calculations. In turn, the TIMSS organisers need to continue improving the measurement procedures and research tools by introducing additional success criteria that reflect the individual and comparable results of students in the current TIMSS cycle, as well as indicators of the reliability of contextual information obtained by sociological means.

1. Yastrebov G. A., Pinskaya M. A., Kosaretsky S. G. using contextual data for education quality assessment: The experience of tools development and testing. Voprosy obrazovaniya = Educational Studies Moscow. 2014; 4: 58–95. DOI: 10.17323/1814-9545-2014-4-58-95 (In Russ.)

2. Lessani A., Yunus A., Tarmiz R., Mahmud R. Why Singaporean 8-th grade students gain highest mathematics ranking in TIMSS (1999–2011). International Education Studies [Internet] 2014 [cited 2022 May 15]; 11 (7): 173–181. Available from: https://web.archive.org/web/20170814020957id_/http://files.eric.ed.gov/fulltext/EJ1071031.pdf

3. Tajudin N., Chinnappan M. The link between higher order thinking skills, representation and concepts in enhancing TIMSS tasks. International Journal of Instruction [Internet]. 2016 [cited 2022 May 15]; 2 (9): 199–214. Available from: https://www.e-iji.net/dosyalar/iji_2016_2_14.pdf

4. Lin F. The data makes the difference: How Chinese Taipei used TIMSS data to reform mathematics education. IEA Compass: Briefs in Education series [Internet]. № 2. Amsterdam: International Association for the Evaluation of Educational Achievement (IEA); 2018 [cited 2022 May 15]. 7 p. Available from: https://web.archive.org/web/20170814020957id_/http://files.eric.ed.gov/fulltext/ED588381.pdf

5. Kovaleva G. S., Krasnyanskaya K. A., Pentin A. Yu., Sadovshchikova O. I. What new results were obtained in the framework of the international study TIMSS‑2019. Otechestvennaya i zarubezhnaya pedagogika = Domestic and Foreign Pedagogy [Internet]. 2021 [cited 2022 Aug 17]; 5 (79): 98–123. Available from: https://www.elibrary.ru/download/elibrary_47228467_81809060.pdf (In Russ.)

6. Neidorf T., Arora A., Erberber E., Tsokodayi Y., Mai T. Student misconceptions and errors in physics and mathematics exploring data from TIMSS and TIMSS advanced [Internet]. IEA Research for Education; 2020 [cited 2022 May 15]. 173 p. Available from: https://web.archive.org/web/20200326175114id_/https://link.springer.com/content/pdf/10.1007/978-3-030-30188-0.pdf

7. Tallberg C., Axelsson M. Exploring coherence between Swedish grades and TIMSS 2015. IEA Compass: Briefs in Education series [Internet]. Amsterdam: International Association for the Evaluation of Educational Achievement (IEA); 2021 [cited 2022 May 15]. 14 p. Available from: https://files.eric.ed.gov/fulltext/ED615156.pdf

8. Naidoo Ja., Sibanda D. Examining science performance of South African grade 9 learners in TIMSS 2015 through a gender lens. South African Journal of Education [Internet]. 2020 [cited 2022 May 15]; 12 (40): S1–S10. Available from: https://pdfs.semanticscholar.org/b9ef/3a07ffa77c049c8ffe24dcfb85114cb66cc6.pdf

9. Arends F., Visser M. The contribution of South African teachers to students’ sense of belonging and mathematics achievement: Students’ perspective from the 2015 trends in international mathematics and science study. South African Journal of Childhood Education [Internet]. 2019 [cited 2022 May 15]; 1 (9). Available from: https://www.researchgate.net/profile/Mariette_Visser/publication/335609948_The_contribution_of_South_African_teachers_to_students_sense_of_belonging_and_mathematics_achievement_Students_perspective_from_the_2015_Trends_in_International_Mathematics_and_Science_Study/links/5d81dc2a458515fca1712977/The-contribution-of-South-African-teachers-to-students-sense-of-belonging-and-mathematics-achievement-Students-perspective-from-the-2015-Trends-in-International-Mathematics-and-Science-Study.pdf

10. Winnaar L., Arends F., Beku U. Reducing bullying in schools by focusing on school climate and school socio-economic status. South African Journal of Education [Internet]. 2018 [cited 2022 May 15]; 10 (38): S1–S10. Available from: https://www.timss-sa.org/wp-content/uploads/2021/07/Reducing-bullying-in-schools_-Winnaar-et-al_2018.pdf

11. Albayrakoglu O., Yildirim S. School characteristics mediating the relationship between school socioeconomic status and mathematics achievement. International Journal of Assessment Tools in Education [Internet]. 2022 [cited 2022 May 15]; 1 (9): 98–117. Available from: https://dergipark.org.tr/en/download/article-file/1308733

12. Mirazchiyski P. Response to «An Examination of Plausible Score Correlation from the Trend in Mathematics and Science Study». Athens Journal of Education [Internet]. 2019 [cited 2022 May 15]; 6 (2): 145–158. Available from: https://www.athensjournals.gr/education/2019-6-2-4-Mirazchiyski.pdf.

13. Sanchez J. Ponce M. Physics-mathematics associations: Evidence from TIMSS student achievements. Science Education International [Internet]. 2020 [cited 2022 May 15]; 9 (31): 229–236. Available from: https://www.researchgate.net/profile/Joje-Mar-Sanchez/publication/344032111_Physics-Mathematics_Associations_Evidence_from_TIMSS_Student_Achievements/links/5f4ec37c299bf13a3196f335/Physics-Mathematics-Associations-Evidence-from-TIMSS-Student-Achievements.pdf

14. Michaelides M., Brown G., Eklöf H., Papanastasiou E. Motivational profiles in TIMSS mathematics: Exploring student clusters across countries and time [Internet]. International Association for the Evaluation of Educational Achievement IEA Research for Education; 2017 [cited 2022 May 15]; 149 p. Available from: https://library.oapen.org/bitstream/handle/20.500.12657/22955/1007205.pdf?sequence=1

15. Lee J., Chen M. Cross country predictive validities of non cognitive variables for mathematics achievement: Evidence based on TIMSS 2015. EURASIA Journal of Mathematics, Science and Technology Education [Internet]. 2019 [cited 2022 May 15]; 15 (8). Available from: https://www.ejmste.com/download/cross-country-predictive-validities-of-non-cognitive-variables-for-mathematics-achievement-evidence-7685.pdf

16. Geesa R., Izci B., Song H., Chen S. Exploring factors of home resources and attitudes towards mathematics in mathematics achievement in South Korea, Turkey, and the United States. EURASIA Journal of Mathematics, Science and Technology Education [Internet]. 2019 [cited 2022 May 15]; 15 (9): 1–18. Available from: https://www.ejmste.com/download/exploring-factors-of-home-resources-and-attitudes-towards-mathematics-in-mathematics-achievement-in-7708.pdf

17. Koyuncu I. TIMSS international benchmarks of eighth graders in mathematics: A correspondence analysis study. International Electronic Journal of Elementary Education [Internet]. 2021 [cited 2022 May 15]; 14 (2): 179–194. Available from: https://www.iejee.com/index.php/IEJEE/article/view/1701/563

18. Awang H., Hashim F., Salleh A., Tan L. The influence of mathematics score and student factors on science achievement using TIMSS data. EURASIA Journal of Mathematics, Science and Technology Education [Internet]. 2021 [cited 2022 May 15]; 17 (6): em1976. Available from: https://web.archive.org/web/20210528063623id_/https://www.ejmste.com/download/the-influence-of-mathematics-score-and-student-factors-on-science-achievement-using-timss-data-10931.pdf

19. Mikk J. Students’ homework and TIMSS 2003 mathematics results. In: 7th International Conference Teaching Mathematics: Retrospective and Perspectives [Internet]; 2006 May 12–13; Tartu, Estonia. Tartu, Estonia; 2006 [cited 2022 May 15]; 8 p. Available from: https://web.archive.org/web/20200711224453id_/https://files.eric.ed.gov/fulltext/ED491866.pdf

20. Khavenson T., Kersha Yu. Comparative analysis of PISA and TIMSS test results in Russia and European countries. Sovremennaja analitika obrazovanija = Modern Analytics of Education [Internet]. 2017 [cited 2022 Aug 17]; 1 (9): 1–32. Available from: https://publications.hse.ru/pubs/share/folder/v01bj4pmvg/202102381.pdf (In Russ.)

21. Tyumeneva Yu. A., Khavenson T. E. Characteristics of teachers and achievements of schoolchildren. Application of the first difference method to TIMSS-2007 data. Voprosy obrazovaniya = Educational Studies Moscow [Internet]. 2012. [cited 2022 Aug 17]; 3: 113– 140. Available from: https://publications.hse.ru/mirror/pubs/share/folder/hlfj5yukmf/direct/68454465

22. Ozdemir B. Equating TIMSS mathematics subtests with nonlinear equating methods using NEAT design: CircleArc equating approaches. International Journal of Progressive Educa­tion [Internet]. 2017 [cited 2022 May 05]; 13 (2): 116–132. Available from: https://ijpe.inased.org/files/2/manuscript/manuscript_241/ijpe-241-manuscript-111607.pdf

23. Lim H., Sireci S. Linking TIMSS and NAEP assessments to evaluate international trends in achievement. Education Policy Analysis Archives [Internet]. 2017 [cited 2021 May 05]; 25 (11). Available from: https://www.redalyc.org/pdf/2750/275050047011.pdf

24. Wang Ji., Ma X. Rejoinder: Response to an examination of plausible score correlation from the trend in mathematics and science study. Athens Journal of Education [Internet]. 2019 [cited 2021 May 05]; 5: 160–169. Available from: https://www.researchgate.net/profile/Jianjun-Wang-11/publication/330636150_Rejoinder_Response_to_An_Examination_of_Plausible_Score_Correlation_from_the_Trend_in_Mathematics_and_Science_Study/links/5e891ad3a6fdcca789f4945d/Rejoinder-Response-to-An-Examination-of-Plausible-Score-Correlation-from-the-Trend-in-Mathematics-and-Science-Study.pdf

25. Wu M. The role of plausible values in large scale surveys. Studies in Educational Evaluation [Internet]. 2005 [cited 2022 May 05]; 31: 114–128. Available from: https://www.acer.org/files/plausiblevaluesinsee.pdf

26. Davier M., Gonzalez E., Mislevy R. What are plausible values and why are they useful? IERI Monograph Series [Internet]. 2009 [cited 2022 May 05]; 2: 9–36. Available from: https://www.ierinstitute.org/fileadmin/Documents/IERI_Monograph/IERI_Monograph_Volume_02_Chapter_01.pdf

27. Gorshkov M. K., Sheregi F. E. Prikladnaya sotsiologiya: metodologiya i metody = Applied sociology: Methodology and methods [Internet]. Moscow: Publishing House Alfa; 2009 [cited 2022 Aug 17]. 416 p. Available from: https://www.isras.ru/files/File/Prikl_Soc_full.pdf (In Russ.)

28. Goodwin J. Issledovaniye v psikhologii: metody i planirovaniye = Research in psychology: Methods and planning [Internet]. St. Petersburg: Publishing House Piter: 2004 [cited 2022 Aug 17]. 558 p. Available from: https://www.studmed.ru/view/gudvin-dzh-issledovanie-v-psihologii-metody-i-planirovanie_2fc857e1293.html (In Russ.)

29. Ilgun Derek M. Effect of extreme and acquiescence response style in TIMSS 2015. Eurasian Journal of Educational Research [Internet]. 2020 [cited 2022 May 15]; 87: 199–219. Available from: https://web.archive.org/web/20200721104047id_/http://ejer.com.tr/public/assets/catalogs/0206512001591274575.pdf

30. Mullis I., Martin M., Davier M. TIMSS 2023 assessment frameworks [Internet]. International Association for the Evaluation of Educational Achievement 2021; 2021 [cited 2022 May 01]. p. 1–97. Available from: https://timss.bc.edu/timss2023/frameworks/pdf/T23_Frameworks.pdf

31.

32.