Validity assumptions for a multiple-choice test of medical knowledge with open-books and web access. A known groups comparison study.

Authors

  • Lotte Dyhrberg O'Neill Syddansk Universitet
  • Eivind Ortind Simonsen Aarhus Universitet
  • Ulla Breth Knudsen Aarhus Universitet
  • Jesper Stentoft Aarhus Universitet
  • Anders Bonde Jensen Aarhus Universitet
  • Charlotte Green Carlsen Aarhus Universitet
  • Anne Mette Mørcke Københavns Universitet

DOI:

https://doi.org/10.7146/dut.v13i25.97864

Abstract

Relatively little evidence about the validity threats in open-book multiple-choice tests exist. The aim of this study was to examine validity aspects relating to gener-alization, extrapolation and decision of a multiple-choice test of medical knowledge with aids (open-book and internet access). The theoretical framework was modern validity theory, and the study was designed as a ‘known groups com-parison’ study. Test performances of three known groups of test takers hypothe-sized to have different knowledge levels of the test content were compared, and analysis of pass/fail decisions was used to examine implications of decisions based on test scores. Results indicated that it was possible to discriminate between expert and non-expert test taker groups even with the access to aids. In contrast, an inde-fensible passing score was found to be the largest potential threat to test validity. Relatively little evidence about the validity threats in open-book multiple-choice tests exist. The aim of this study was to examine validity aspects relating to gener-alization, extrapolation and decision of a multiple-choice test of medical knowledge with aids (open-book and internet access). The theoretical framework was modern validity theory, and the study was designed as a ‘known groups com-parison’ study. Test performances of three known groups of test takers hypothe-sized to have different knowledge levels of the test content were compared, and analysis of pass/fail decisions was used to examine implications of decisions based on test scores. Results indicated that it was possible to discriminate between expert and non-expert test taker groups even with the access to aids. In contrast, an inde-fensible passing score was found to be the largest potential threat to test validity. 

Author Biographies

Lotte Dyhrberg O'Neill, Syddansk Universitet

Lotte Dyrhberg O’Neill is an Associate Professor at the Centre for Teaching and Learning at the University of Southern Denmark with interests in assessment, student selection, educational attainment and educational development.

Eivind Ortind Simonsen, Aarhus Universitet

Eivind Ortind Simonsen is a computer scientist who has been working on various digitalization projects at Aarhus University, primarily projects with relation to digital exams.

Ulla Breth Knudsen, Aarhus Universitet

Ulla Breth Knudsen is Professor at Aarhus University, Denmark and responsible for the International semester in Gynecology-Obstetrics and Pediatrics. Her main interest is in assessment, students selection, educational development and internationalisation. 

Jesper Stentoft, Aarhus Universitet

Jesper Stentoft is a Consultant at the department of Haematology at Aarhus University Hospital and Professor and educational ambassador at the Institute of Clinical Medicine at Aarhus University.

Anders Bonde Jensen, Aarhus Universitet

Anders Bonde Jensen is Consultant at department of Oncology at Aarhus University Hospital, and Professor at the Institute of Clinical Medicine at Aarhus University with an interest in teaching and educational development. 

Charlotte Green Carlsen, Aarhus Universitet

Charlotte Green Carlsen is an Associate Professor at the Centre of Health Science Education Aarhus University with interests in assessment, student selection, and postgraduate surgical end emergency training.

Anne Mette Mørcke, Københavns Universitet

Anne Mette Morcke is the Director of Copenhagen Academy for Medical Education and Simulation at Rigshospitalet. She also holds a position as associate professor in medical education at the Department of Clinical Medicine at the University of Copenhagen with a particular interest in the more complex aspects of assessment. 

Downloads

Published

2018-09-28

How to Cite

O’Neill, L. D., Simonsen, E. O., Knudsen, U. B., Stentoft, J., Jensen, A. B., Carlsen, C. G., & Mørcke, A. M. (2018). Validity assumptions for a multiple-choice test of medical knowledge with open-books and web access. A known groups comparison study. The Danish Journal of Higher Education, 13(25), 134–150. https://doi.org/10.7146/dut.v13i25.97864

Issue

Section

Scientific article