Enhancing dental and medical students’ self-regulated learning through multiple choice questions: An evaluative study using machine learning
An evaluation of two multiple-choice questions systems using machine learning
DOI:
https://doi.org/10.7146/lom.v17i29.140337Keywords:
MCQ, Machine Learning, Dental, Motivation, Self-Regulated LearningAbstract
This article reports a mixed methods study that uses machine learning and thematic analysis to investigate student evaluation of multiple-choice questions (MCQ). The focus is on medical and dental students' experience of self-regulated learning and motivation. We evaluate two systems developed at Aarhus University: "MED MCQ" used by medical students and "MCQ anatomy" used by dental students. We evaluate through two surveys in SurveyXact with responses from 126 medical students and 70 dental students. We use topic modelling over free text responses. The machine-learning model identifies two groups of students who, in different ways, both experience the system as motivating and facilitating their learning process. The students experience increased self-regulated learning by being able to choose the form of presentation of questions and answer questions independently of the instructor. The article discusses how educators and developers can use MCQs to promote student learning and how to analyze open-ended questions. We discuss the potential for using machine learning and integrating MCQ systems into teaching.
Downloads
References
Ang, E. T., Chan, J. M., Gopal, V., & Li Shia, N. (2018). Gamifying anatomy education. Clinical Anatomy, 31(7), 997-1005.
Armstrong, R. D., Jones, D. H., Koppel, N. B., & Pashley, P. J. (2004). Computerized adaptive testing with multiple-form structures. Applied Psychological Measurement, 28(3), 147-164.
Bergstrom, B. A., & Lunz, M. E. (1999). CAT for certification and licensure. Innovations in computerized assessment, 67-91.
Billings, M., DeRuchie, K., Hussie, K., Kulesher, A., Merrell, J., Morales, A., Paniagua, M.,
Sherlock, J., Swygert, K., & Tyson, J. (2021). NBME item writing guide. Constructing written test questions for the health sciences. Philadelphia: National Board of Medical Examiners.
Bjork, R. A., & Bjork, E. L. (2019). Forgetting as the friend of learning: Implications for teaching and self-regulated learning. Advances in Physiology Education, 43(2), 164-167.
Borakati, A. (2021). Evaluation of an international medical E-learning course with natural language processing and machine learning. BMC Med Educ, 21, 1-10.Chandra, S., Katyal, R., Chandra, S., Singh, K., Singh, A., & Joshi, H. (2018). Medical education/original article creating valid multiple-choice questions (MCQs) Bank with Faculty Development of Pharmacology. Indian J Physiol Pharmacol, 62(3), 359-366.
Deci, E. L., & Ryan, R. M. (2012). Self-determination theory. Handbook of theories of social psychology, 1(20), 416-436.
Decorte, T., Malm, A., Sznitman, S. R., Hakkarainen, P., Barratt, M. J., Potter, G. R., Werse, B., Kamphausen, G., Lenton, S., & Asmussen Frank, V. (2019). The challenges and benefits of analyzing feedback comments in surveys: Lessons from a cross-national online survey of small-scale cannabis growers. Methodological innovations, 12(1), 205979911982560. https://doi.org/10.1177/2059799119825606
dos Santos, J. A., Syed, T. I., Naldi, M. C., Campello, R. J., & Sander, J. (2019). Hierarchical density-based clustering using MapReduce. IEEE Transactions on Big Data, 7(1), 102-114
Downing, S. M., & Haladyna, T. M. (2004). Validity threats: overcoming interference with proposed interpretations of assessment data. Med Educ, 38(3), 327-333.
Drasgow, F., & Olson-Buchanan, J. B. (1999). Innovations in computerized assessment. L. Erlbaum Associates, Publishers.
Durosaro, O., Lachman, N., & Pawlina, W. (2008). Use of knowledge-sharing web-based portal in gross and microscopic anatomy. Annals Academy of Medicine Singapore, 37(12), 998.
Edmondson, D. R., Boyer, S. L., & Artis, A. B. (2012). Self-directed learning: A meta-analytic review of adult learning constructs. International Journal of Education Research, 7(1), 40-48.
Einig, S. (2013). Supporting Students' Learning: The Use of Formative Online Assessments. Accounting education (London, England), 22(5), 425-444. https://doi.org/10.1080/09639284.2013.803868
Eleanor, S., & Mick, P. C. (2017). Some Methodological Uses of Responses to Open Questions and Other Verbatim Comments in Quantitative Surveys. Methoden, daten, analysen, 11(2), 115-135. https://doi.org/10.12758/mda.2017.01
Evergreen, S., & Metzner, C. (2013). Design Principles for Data Visualization in Evaluation: Design Principles for Data Visualization. New directions for evaluation, 2013(140), 5-20. https://doi.org/10.1002/ev.20071
Ghojogh, B., Crowley, M., Karray, F., & Ghodsi, A. (2023). Uniform Manifold Approximation and Projection (UMAP). I B. Ghojogh, M. Crowley, F. Karray & A. Ghodsi (red.), Elements of Dimensionality Reduction and Manifold Learning (s. 479-497). Springer International Publishing. https://doi.org/10.1007/978-3-031-10602-6_17
Grolnick, W. S., & Ryan, R. M. (1987). Autonomy in children's learning: an experimental and individual difference investigation. Journal of personality and social psychology, 52(5), 890.
Guay, F., Ratelle, C. F., & Chanal, J. (2008). Optimal Learning in Optimal Contexts: The Role of Self-Determination in Education. Canadian psychology = Psychologie canadienne, 49(3), 233-240. https://doi.org/10.1037/a0012758
Hendricson, W. D., Andrieu, S. C., Chadwick, D. G., Chmar, J. E., Cole, J. R., George, M. C., Glickman, G. N., Glover, J. F., Goldberg, J. S., Change, A. C. o., & Education, I. i. D. (2006). Educational strategies associated with development of problem‐solving, critical thinking, and self‐directed learning. J Dent Educ, 70(9), 925-936.
Javaeed, A. (2018). Assessment of higher ordered thinking in medical education: multiple choice questions and modified essay questions. MedEdPublish, 7, 128.
Kanzow, P., Schmidt, D., Herrmann, M., Wassmann, T., Wiegand, A., & Raupach, T. (2023). Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Study Involving the Application of Different Scoring Methods. JMIR Med Educ, 9, e43792. https://doi.org/10.2196/43792
Kizilcec, R. F., Pérez-Sanagustín, M., & Maldonado, J. J. (2017). Self-regulated learning strategies predict learner behavior and goal attainment in Massive Open Online Courses. Computers & Education, 104, 18-33. https://doi.org/https://doi.org/10.1016/j.compedu.2016.10.001
Knowles, M. S. (1975). Self-directed learning: A guide for learners and teachers.
Lee, N. W., Shamsuddin, W. N. F. W., Wei, L. C., Anuardi, M. N. A. M., Heng, C. S., & Abdullah, A. N. (2021). Using Online Multiple Choice Questions with Multiple Attempts: A Case for Self-Directed Learning among Tertiary Students. International Journal of Evaluation and Research in Education, 10(2), 553-568.
Little, J. L., Bjork, E. L., Bjork, R. A., & Angello, G. (2012). Multiple-choice tests exonerated, at least of some charges: Fostering test-induced learning and avoiding test-induced forgetting. Psychological science, 23(11), 1337-1344.
Malzer, C., & Baum, M. (2020). A Hybrid Approach To Hierarchical Density-based Cluster Selection. Ithaca.
Mazmanian, P., & Feldman, M. (2011). Theory is needed to improve education, assessment and policy in self‐directed learning. Med Educ, 45(4), 324-326.
McDaniel, M. A., Anderson, J. L., Derbish, M. H., & Morrisette, N. (2007). Testing the testing effect in the classroom. European journal of cognitive psychology, 19(4-5), 494-513.
McDermott, K. B., Agarwal, P. K., D'Antonio, L., Roediger III, H. L., & McDaniel, M. A. (2014). Both multiple-choice and short-answer quizzes enhance later exam performance in middle and high school classes. Journal of Experimental Psychology: Applied, 20(1), 3.
McInnes, L., Healy, J., & Melville, J. (2018). Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426.
Merriam, S. B. (2001). Andragogy and self-directed learning: Pillars of adult learning theory. New directions for adult and continuing education, 2001(89), 3.
Murray, J. H. (1997). Hamlet on the holodeck : the future of narrative in cyberspace. MIT Press.
Nicol, D. (2007). E-assessment by design: using multiple-choice tests to good effect. Journal of further and higher education, 31(1), 53-64. https://doi.org/10.1080/03098770601167922
Pai, K. M., Rao, K. R., Punja, D., & Kamath, A. (2014). The effectiveness of self-directed learning (SDL) for teaching physiology to first-year medical students. The Australasian medical journal, 7(11), 448.
Palmer, E. J., & Devitt, P. G. (2007). Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper. BMC Med Educ, 7(1), 1-7.
Ryan, R. M., & Deci, E. L. (2000). Self-Determination Theory and the Facilitation of Intrinsic Motivation, Social Development, and Well-Being. The American psychologist, 55(1), 68-78. https://doi.org/10.1037/0003-066X.55.1.68 (Positive Psychology)
Say, R., Visentin, D., Cummings, E., Carr, A., & King, C. (2022). Formative online multiple-choice tests in nurse education: An integrative review. Nurse education in practice, 58, 103262.
Schuwirth, L. W. T., & Van Der Vleuten, C. P. M. (2004). Different written assessment methods: what can be said about their strengths and weaknesses? Med Educ, 38(9), 974-979. https://doi.org/10.1111/j.1365-2929.2004.01916.x
Stake, R. E. (2013). Multiple case study analysis. Guilford press.
Tam, M., Hart, A., Williams, S., Holland, R., Heylings, D., & Leinster, S. (2010). Evaluation of a computer program (‘disect’) to consolidate anatomy knowledge: A randomised-controlled trial. Med Teach, 32(3), e138-e142.
Vansteenkiste, M., Simons, J., Lens, W., Sheldon, K. M., & Deci, E. L. (2004). Motivating learning, performance, and persistence: the synergistic effects of intrinsic goal contents and autonomy-supportive contexts. Journal of personality and social psychology, 87(2), 246.
Yin, R. K. (2009). Case study research: Design and methods (årg. 5). sage.
Yu, D., & Xiang, B. (2023). Discovering topics and trends in the field of Artificial Intelligence: Using LDA topic modeling. Expert systems with applications, 225, 120114. https://doi.org/10.1016/j.eswa.2023.120114
Zia, S., Jabeen, F., Atta, K., & Sial, N. A. (2016). Self-directed learning (SDL), an effective method for teaching physiology to medical students. PJMHS, 10(3), 699-702.
Zimmerman, B. J. (2002). Becoming a Self-Regulated Learner: An Overview. Theory into practice, 41(2), 64-70. https://doi.org/10.1207/s15430421tip4102_2
Zimmerman, B. J. (2008). Investigating Self-Regulation and Motivation: Historical Background, Methodological Developments, and Future Prospects. American educational research journal, 45(1), 166-183. https://doi.org/10.3102/0002831207312909
Zimmerman, B. J., & Schunk, D. H. (2011). Self-regulated learning and performance: An introduction and an overview. I Handbook of self-regulation of learning and performance. (s. 1-12). Routledge/Taylor & Francis Group.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Emilie Leth Rasmussen, Malthe Have Musaeus, Mads Ronald Dahl, Henrik Løvschall, Peter Musaeus
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Articles published in the Journal of Learning and Media are licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 Unported Licens.
Authors retain copyright and grant the journal right of first publication; simultaneously articles are licensend under the Creative Commons Attribution license: Attribution-NonCommercial-NoDerviatives (by-nc-nd). Read about this license at https://creativecommons.org/licenses/by-nc-nd/3.0/
---
At LOM.dk, you will also find articles from the discontinued Journal for the Continuing and Further Education of the Danish Universities (UNEV). Note that special rules apply to UNEV articles:
It is the authors and any other copyright holder who have the copyright of articles published under the auspices of UNEV, and access to the articles is contingent on users acknowledging and complying with the associated legal guidelines:
- Users may download and print one copy of any UNEV publication for private studies or research.
- The redistribution of articles or the use of these for revenue-funded activities or commercial purposes are not allowed.
- It is not allowed to distribute the URLs of UNEV articles.