Validity assumptions for a multiple-choice test of medical knowledge with open-books and web access. A known groups comparison study.

Authors

  • Lotte Dyhrberg O'Neill Syddansk Universitet
  • Eivind Ortind Simonsen Aarhus Universitet
  • Ulla Breth Knudsen Aarhus Universitet
  • Jesper Stentoft Aarhus Universitet
  • Anders Bonde Jensen Aarhus Universitet
  • Charlotte Green Carlsen Aarhus Universitet
  • Anne Mette Mørcke Københavns Universitet

DOI:

https://doi.org/10.7146/dut.v13i25.97864

Abstract

Relatively little evidence about the validity threats in open-book multiple-choice tests exist. The aim of this study was to examine validity aspects relating to gener-alization, extrapolation and decision of a multiple-choice test of medical knowledge with aids (open-book and internet access). The theoretical framework was modern validity theory, and the study was designed as a ‘known groups com-parison’ study. Test performances of three known groups of test takers hypothe-sized to have different knowledge levels of the test content were compared, and analysis of pass/fail decisions was used to examine implications of decisions based on test scores. Results indicated that it was possible to discriminate between expert and non-expert test taker groups even with the access to aids. In contrast, an inde-fensible passing score was found to be the largest potential threat to test validity. Relatively little evidence about the validity threats in open-book multiple-choice tests exist. The aim of this study was to examine validity aspects relating to gener-alization, extrapolation and decision of a multiple-choice test of medical knowledge with aids (open-book and internet access). The theoretical framework was modern validity theory, and the study was designed as a ‘known groups com-parison’ study. Test performances of three known groups of test takers hypothe-sized to have different knowledge levels of the test content were compared, and analysis of pass/fail decisions was used to examine implications of decisions based on test scores. Results indicated that it was possible to discriminate between expert and non-expert test taker groups even with the access to aids. In contrast, an inde-fensible passing score was found to be the largest potential threat to test validity. 

Author Biographies

Lotte Dyhrberg O'Neill, Syddansk Universitet

Lotte Dyrhberg O’Neill is an Associate Professor at the Centre for Teaching and Learning at the University of Southern Denmark with interests in assessment, student selection, educational attainment and educational development.

Eivind Ortind Simonsen, Aarhus Universitet

Eivind Ortind Simonsen is a computer scientist who has been working on various digitalization projects at Aarhus University, primarily projects with relation to digital exams.

Ulla Breth Knudsen, Aarhus Universitet

Ulla Breth Knudsen is Professor at Aarhus University, Denmark and responsible for the International semester in Gynecology-Obstetrics and Pediatrics. Her main interest is in assessment, students selection, educational development and internationalisation. 

Jesper Stentoft, Aarhus Universitet

Jesper Stentoft is a Consultant at the department of Haematology at Aarhus University Hospital and Professor and educational ambassador at the Institute of Clinical Medicine at Aarhus University.

Anders Bonde Jensen, Aarhus Universitet

Anders Bonde Jensen is Consultant at department of Oncology at Aarhus University Hospital, and Professor at the Institute of Clinical Medicine at Aarhus University with an interest in teaching and educational development. 

Charlotte Green Carlsen, Aarhus Universitet

Charlotte Green Carlsen is an Associate Professor at the Centre of Health Science Education Aarhus University with interests in assessment, student selection, and postgraduate surgical end emergency training.

Anne Mette Mørcke, Københavns Universitet

Anne Mette Morcke is the Director of Copenhagen Academy for Medical Education and Simulation at Rigshospitalet. She also holds a position as associate professor in medical education at the Department of Clinical Medicine at the University of Copenhagen with a particular interest in the more complex aspects of assessment. 

Downloads

Published

2018-09-28

How to Cite

O’Neill, L. D., Simonsen, E. O., Knudsen, U. B., Stentoft, J., Jensen, A. B., Carlsen, C. G., & Mørcke, A. M. (2018). Validity assumptions for a multiple-choice test of medical knowledge with open-books and web access. A known groups comparison study. The Danish Journal of Higher Education, 13(25), 134–150. https://doi.org/10.7146/dut.v13i25.97864

Issue

Vol. 13 No. 25 (2018): Universitetspædagogikum

Section

Scientific article

License

DUT udkommer elektronisk via Statsbibliotekets Open Journal System (Tidsskrift.dk) og DUNs hjemmeside (DUN-net.dk) forår og efterår. Det er gratis og frit tilgængeligt at læse og downloade artikler fra tidsskriftet.

Det er ikke muligt at abonnere på Dansk Universitetspædagogisk Tidsskrift, DUT, men hvis du er medlem af DUN, får du tilsendt en nyhedsmail med link til udgivelsen, når den nyeste udgave er online. Linket vil også være tilgængeligt her på siden, så snart tidsskriftet er publiceret. 

© Copyright

Artikler publiseret i Dansk Universitetspædagogisk Tidsskrift, DUT, må bruges (downloades) og genbruges (distribueres, kopieres, citeres) til ikke-kommercielle formål med reference til forfattere og Dansk Universitetspædagogisk Tidsskrift. 

Artikler indsendt til Dansk Universitetspædagogisk Tidsskrift må ikke publiseres i andre tidskrifter.

Betingelser

Artikler i Dansk Universitetspædagogisk Tidsskrift, DUT, er omfattet af ophavsretsloven, og der må citeres fra dem.

Følgende betingelser skal dog være opfyldt:

  • Citatet skal være i overensstemmelse med „god skik“
  • Der må kun citeres „i det omfang, som betinges af formålet“
  • Ophavsmanden til teksten skal krediteres, og kilden skal angives ift. ovenstående bibliografiske oplysninger.