The effects of an AI feedback coach on students’ peer feedback quality, composition, and feedback experience
DOI:
https://doi.org/10.7146/lom.v17i31.148831Nøgleord:
Artificial Intelligence, AI, Natural Language Processing, NLP, Higher Education, Feedback, Peer feedback, Feedback Quality, Cognitive load, Multimedia learning, Prompting, Adaptive prompting, Scaffolding, Sentiment, Instructional designResumé
This study examines the integration of an Artificial Intelligence (AI) feedback coach in a peer feedback activity. Participants provided peers with feedback on their assignments. While providing feedback, they either received real-time adaptive AI coaching (intervention group) or not (control group). Feedback comments from participants were analysed concerning content, text complexity, and sentiment. Survey responses were coded for sentiment and themes. Results show adverse effects of the AI feedback coach. Intervention group participants’ feedback included fewer reflective questions and adhered less to criteria. They provided shorter, more complex feedback. Students indicated mixed views on the AI feedback coach, with some finding it helpful and others distracting. A notable subset of students stated overreliance on the AI coach, prioritising its validation over their own judgment. Results suggest that AI tools need thoughtful integration, possibly with additional scaffolding to counteract overreliance and avoid negative impact on peer feedback quality and feedback experience.
Downloads
Referencer
Alqassab, M., Strijbos, J. W. & Ufer, S. (2018). Training peer-feedback skills on geometric con-struction tasks: role of domain knowledge and peer-feedback levels. European Journal of Psychology of Education, 33, 11–30. https://doi.org/10.1007/s10212-017-0342-0
Bimba, A. T., Idris, N., Al-Hunaiyyan, A., Ibrahim, S. U., Mustafa, N., Supa’at, I., ..., & Ahmad, M. Y. (2021). The effects of adaptive feedback on student’s learning gains. International Journal of Advanced Computer Science and Applications, 12(7), 68-80. https://doi.org/10.14569/IJACSA.2021.0120709
Bouchet, F., Harley, J.M., & Azevedo, R. (2016). Can adaptive pedagogical agents’ prompting strategies improve students’ learning and self-regulation? In A. Micarelli, J. Stamper, & K. Panourgia (Eds). Intelligent Tutoring Systems (Lecture Notes in Computer Science, 9684, 368–374). Springer, Cham. https://doi.org/10.1007/978-3-319-39583-8_43
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling up-take of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354
Chen, T. (2014). Technology-supported peer feedback in ESL/EFL writing classes: a research synthesis. Computer Assisted Language Learning, 29(2), 365–397. https://doi.org/10.1080/09588221.2014.960942
Clarke, D., & Hollingsworth, H. (2002). Elaborating a model of teacher professional growth. Teaching and Teacher Education, 18(8), 947–967. https://doi.org/10.1016/S0742-051X(02)00053-7
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Routledge. https://doi.org/10.4324/9780203771587
Creswell, J. W. (2014). Research Design: Qualitative, Quantitative and Mixed Methods Ap-proaches (4th ed.). Thousand Oaks, CA: Sage.
Dai, Y., Liu, A., & Lim, C. P. (2023). Reconceptualizing ChatGPT and generative AI as a stu-dent-driven innovation in higher education. Procedia CIRP, 119, 84–90. https://doi.org/10.1016/j.procir.2023.05.002
Darvishi, A., Khosravi, H., Sadiq, S., Gašević, D., & Siemens, G. (2024). Impact of AI assis-tance on student agency. Computers & Education, 210, 104967. https://doi.org/10.1016/j.compedu.2023.104967
Deeva, G., Bogdanova, D., Serral, E., Snoeck, M., & De Weerdt, J. (2021). A review of auto-mated feedback systems for learners: Classification framework, challenges and opportu-nities. Computers & Education 162, 104094. https://doi.org/10.1016/j.compedu.2020.104094
Dell’Acqua, F., McFowland, E., Mollick, E. R., Lifshitz-Assaf, H., Kellogg, K., Rajendran, S., Krayer, L., Candelon, F., & Lakhani, K. R. (2023). Navigating the Jagged Technological Fron-tier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality. Harvard Business School Technology & Operations Mgt. Unit Working Paper, (24-013). https://doi.org/10.2139/ssrn.4573321
Evans, C., & Gibbons, N. J. (2007). The interactivity effect in multimedia learning. Computers & Education, 49(4), 1147–1160. https://doi.org/10.1016/j.compedu.2006.01.008
FeedbackFruits (Accessed August 2024). Acai Coach Transparency Note. Retrieved from https://help.feedbackfruits.com/en/articles/7314625-acai-coach-transparency-note
Gan, M. J. S., & Hattie, J. (2014). Prompting secondary students' use of criteria, feedback specificity and feedback levels during an investigative task. Instructional Science, 42(6), 861–878. https://doi.org/10.1007/s11251-014-9319-4
Hansen, L., Olsen, L. R., & Enevoldsen, K. (2023). TextDescriptives: A Python package for cal-culating a large variety of metrics from text. Journal of Open Source Software, 8(84), 5153. https://doi.org/10.21105/joss.05153
Harp, S. F., & Mayer, R. E. (1998). How seductive details do their damage: A theory of cogni-tive interest in science learning. Journal of Educational Psychology, 90(3), 414. https://doi.org/10.1037/0022-0663.90.3.414
Hattie, J. (2023). Visible learning: The sequel: A synthesis of over 2,100 meta-analyses relating to achievement (1st ed.). Routledge. https://doi.org/10.4324/9781003380542
Honnibal, Matthew, Ines Montani, Sofie Van Landeghem, & Adriane Boyd. (2020). spaCy: in-dustrial-strength natural language processing in Python. https://doi.org/10.5281/zenodo.1212303
Hutto, C., & Gilbert, E. (2014). VADER: A parsimonious rule-based model for sentiment analy-sis of social media text. Proceedings of the International AAAI Conference on Web and So-cial Media, 8(1), 216–225. https://doi.org/10.1609/icwsm.v8i1.14550
Jansen, T., Höft, L., Bahr, L., Fleckenstein, J., Möller, J., Köller, O., & Meyer, J. (2024). Empir-ische Arbeit: Comparing generative AI and expert feedback to students’ writing: Insights from student teachers. Psychologie in Erziehung Und Unterricht, 71(2), 80–92. https://doi.org/10.2378/peu2024.art08d
Jensen, L. X., Bearman, M., & Boud, D. (2023). Characteristics of productive feedback en-counters in online learning. Teaching in Higher Education, 30(1), 69–83. https://doi.org/10.1080/13562517.2023.2213168
Jensen, L. X., Bearman, M., & Boud, D. (2023). Feedback encounters: towards a framework for analysing and understanding feedback processes, Assessment & Evaluation in Higher Ed-ucation, 48(1), 121-134, https://doi.org/10.1080/02602938.2022.2059446
Joyner, D. A. (2017). Scaling expert feedback: Two case studies. Proceedings of the Fourth (2017) ACM Conference on Learning @ Scale, 71–80. https://doi.org/10.1145/3051457.3051459
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., …, & Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large language models for educa-tion. Learning and Individual Differences, 103, 102274. https://doi.org/10.1016/j.lindif.2023.102274
Kettanurak, V. (Nui), Ramamurthy, K., & Haseman, W. D. (2001). User attitude as a mediator of learning performance improvement in an interactive multimedia environment: An em-pirical investigation of the degree of interactivity and learning styles. International Journal of Human-Computer Studies, 54(4), 541–583. https://doi.org/10.1006/ijhc.2001.0457
Lawson, A. P., & Mayer, R. E. (2024). Role of individual differences in executive function for learning From distracting multimedia lessons. Journal of Educational Computing Re-search, 62(3),756-784. https://doi.org/10.1177/07356331231215752
Liu, N.-F., & Carless, D. (2006). Peer feedback: The learning element of peer assessment. Teaching in Higher Education, 11(3), 279–290. https://doi.org/10.1080/13562510600680582
Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–16. https://doi.org/10.1145/3313831.3376727
Malmström, H., Stöhr, C., & Ou, W. (2023). Chatbots and other AI for learning: A survey of use and views among university students in Sweden. Chalmers Studies in Communication and Learning in Higher Education, 1(10.17196) https://doi.org/10.17196/CLS.CSCLHE/2023/01
Mayer, R. E. (2002). Multimedia learning. Psychology of learning and motivation, 41, 85-139. Academic Press. https://doi.org/10.1016/S0079-7421(02)80005-6
Meyer, J., Jansen, T., Schiller, R., Liebenow, L. W., Steinbach, M., Horbach, A., & Fleckenstein, J. (2024). Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback increases secondary students’ text revision, motivation, and positive emotions. Computers and Education: Artificial Intelligence, 6, 100199. https://doi.org/10.1016/j.caeai.2023.100199
Nicol, D. J., & Macfarlane‐Dick, D. (2006). Formative assessment and self‐regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090
Nicol, D., Thomson, A., & Breslin, C. (2014). Rethinking feedback practices in higher educa-tion: A peer review perspective. Assessment & Evaluation in Higher Education, 39(1), 102–122. https://doi.org/10.1080/02602938.2013.795518
Nielsen, B. L. (2012). Science teachers’ meaning-making when involved in a school-based professional development project. Journal of Science Teacher Education, 23(6), 621-649. https://doi.org/10.1007/s10972-012-9300-5
Nguyen, N. (2023). Automated feedback coach: Ai Tutor for better feedback writing. https://feedbackfruits.com/blog/automated-feedback-coach-student-ai-tutor-to-write-better-feedback
Nückles, M., Hübner, S., & Renkl, A. (2009). Enhancing self-regulated learning by writing learn-ing protocols. Learning and Instruction, 19(3), 259–271. https://doi.org/10.1016/j.learninstruc.2008.05.002
Nückles, M., Hübner, S., Dümer, S., & Renkl, A. (2010). Expertise reversal effects in writing-to-learn. Instructional Science, 38(3), 237–258. https://doi.org/10.1007/s11251-009-9106-9
Ozogul, G., & Sullivan, H. (2009). Student performance and attitudes under formative evalua-tion by teacher, self and peer evaluators. Educational Technology Research and Develop-ment, 57, 393-410. https://doi.org/10.1007/s11423-007-9052-7
Panadero, E., & Jonsson, A. (2013). The use of scoring rubrics for formative assessment pur-poses revisited: A review. Educational research review, 9, 129-144. https://doi.org/10.1016/j.edurev.2013.01.002
Panadero, E., & Lipnevich, A. A. (2022). A review of feedback models and typologies: Towards an integrative model of feedback elements. Educational Research Review, 35, 100416. https://doi.org/10.1016/j.edurev.2021.100416
Prilop, C. N., & Weber, K. E. (2023). Digital video-based peer feedback training: The effect of expert feedback on pre-service teachers’ peer feedback beliefs and peer feedback quali-ty. Teaching and Teacher Education, 127, 104099. https://doi.org/10.1016/j.tate.2023.104099
Prilop, C. N., Weber, K. E., & Kleinknecht, M. (2021). The role of expert feedback in the devel-opment of pre-service teachers’ professional vision of classroom management in an online blended learning environment. Teaching and Teacher Education, 99, 103276. https://doi.org/10.1016/j.tate.2020.103276
Prilop, C. N., Weber, K. E., & Kleinknecht, M. (2019). How digital reflection and feedback envi-ronments contribute to pre-service teachers’ beliefs during a teaching practicum. Studies in Educational Evaluation, 62, 158–170. https://doi.org/10.1016/j.stueduc.2019.06.005
Prins, F. J., Sluijsmans, D. M. A., & Kirschner, P. A. (2006). Feedback for General Practitioners in Training: Quality, Styles, and Preferences. Advances in Health Sciences Education, 11(3), 289–303. https://doi.org/10.1007/s10459-005-3250-z
Ribeiro, F. N., Araújo, M., Gonçalves, P., André Gonçalves, M., & Benevenuto, F. (2016). Sent-iBench—A benchmark comparison of state-of-the-practice sentiment analysis methods. EPJ Data Science, 5(1), 23. https://doi.org/10.1140/epjds/s13688-016-0085-1
Ruwe, T., & Mayweg-Paus, E. (2023). “Your argumentation is good”, says the AI vs humans – The role of feedback providers and personalised language for feedback effectiveness. Computers & Education: Artificial Intelligence, 5, 100189. https://doi.org/10.1016/j.caeai.2023.100189
Ryan, T., Henderson, M., Ryan, K., & Kennedy, G. (2022). Feedback in higher education: align-ing academic intent and student sensemaking. Teaching in Higher Education, 29(4), 860–875. https://doi.org/10.1080/13562517.2022.2029394
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153-189. https://doi.org/10.3102/0034654307313795
Strijbos, J.-W., Narciss, S., & Dünnebier, K. (2010). Peer feedback content and sender’s competence level in academic writing revision tasks: Are they critical for feedback per-ceptions and efficiency? Learning and Instruction, 20(4), 291–303. https://doi.org/10.1016/j.learninstruc.2009.08.008
Sweller, J. (1994). Cognitive load theory, learning difficulty, and instructional design. Learning & instruction, 4(4), 295-312. https://doi.org/10.1016/0959-4752(94)90003-5
Topping, K. J. (2021). Digital peer assessment in school teacher education and development: a systematic review. Research Papers in Education, 38(3), 472–498. https://doi.org/10.1080/02671522.2021.1961301
Van De Pol, J., Volman, M., & Beishuizen, J. (2010). Scaffolding in teacher–student interaction: A decade of research. Educational Psychology Review, 22(3), 271–296. https://doi.org/10.1007/s10648-010-9127-6
Van der Kleij, F. M., Feskens, R. C. W., & Eggen, T. J. H. M. (2015). Effects of Feedback in a Computer-Based Learning Environment on Students’ Learning Outcomes: A Meta-Analysis. Review of Educational Research, 85(4), 475-511. https://doi.org/10.3102/0034654314564881
Vygotsky, L. S. (1962). The Development of Scientific Concepts in Childhood. In L. Vygotsky & E. Hanfmann, G. Vakar (Eds.), Thought and language (pp. 82–118). MIT Press. https://doi.org/10.1037/11193-006
Wisniewski, B., Zierer, K., & Hattie, J. (2020). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10, 3087. https://doi.org/10.3389/fpsyg.2019.03087
Wood, D., Bruner, J. S., & Ross, G. (1976). The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry, 17(2), 89-100. https://doi.org/10.1111/j.1469-7610.1976.tb00381.x
Zhai, X., Chu, X., Chai, C. S., Jong, M. S. Y., Istenic, A., Spector, M., Liu, J.-B., Yuan, J., & Li, Y. (2021). A review of artificial intelligence (AI) in education from 2010 to 2020. Complexity, 1, 1–18. https://doi.org/10.1155/2021/8812542
Zottmann, J. M., Stegmann, K., Strijbos, J.-W., Vogel, F., Wecker, C., & Fischer, F. (2013). Computer-supported collaborative learning with digital video cases in teacher education: The impact of teaching experience on knowledge convergence. Computers in Human Be-havior, 29(5), 2100–2108. https://doi.org/10.1016/j.chb.2013.04.014
Downloads
Publiceret
Citation/Eksport
Nummer
Sektion
Licens
Copyright (c) 2025 Rasmus Hansen, Christopher Neil Prilop, Tobias Alsted Nielsen, Karen Louise Møller, Rikke Frøhlich Hougaard, Annika Büchert Lindberg
Dette værk er under følgende licens Creative Commons Navngivelse – Ikke-kommerciel – Ingen Bearbejdede Værker (by-nc-nd).
Artikler publiceret i Tidsskriftet for Læring og Medier er licenseret under en Creative Commons Navngivelse-IkkeKommerciel-IngenBearbejdelse 4.0 Unported Licens.
Forfattere bevarer deres ophavsret og giver tidsskriftet ret til første publicering, samtidigt med at værket er omfattet af Creative Commons Attribution-licensen: Navngivelse – Ikke-kommerciel - Ingen Bearbejdede Værker (by-nc-nd). Læs om licensen på http://www.creativecommons.dk/om/.
---
På LOM.dk kan du endvidere finde artikler fra det nu nedlagte Tidsskrift for Universiteternes Efter- og Videreuddannelse (UNEV). Vær opmærksom på, at der gælder særlige regler for UNEV artikler:
Det er forfatterne og evt. andre ophavsret indehavere, der har ophavsretten til artikler udgivet i UNEV regi, og det er en betingelse for adgang til artiklerne, at brugere anerkender og overholder de juridiske retningslinjer forbundet hermed.
- Brugere må downloade og printe én kopi af en hvilken som helst UNEV artikel mhp. private studier eller forskning.
- Det er ikke tilladt at videredistribuere artikler eller anvende disse til indtægtsdækkede aktiviteter eller kommercielle formål.
- Det er tilladt at distribuere URL’en til UNEV artikler.
