Peer assessment of mathematical understanding using comparative judgement
DOI:
https://doi.org/10.7146/nomad.v22i4.148924Abstract
It is relatively straightforward to assess procedural knowledge and difficult to assess conceptual understanding in mathematics. One reason is that conceptual understanding is better assessed using open-ended test questions that invite an unpredictable variety of responses that are difficult to mark. Recently a technique, called comparative judgement, has been developed that enables the reliable and valid scoring of open-ended tests. We applied this technique to the peer assessment of calculus on a first-year mathematics module. We explored the reliability and criterion validity of the outcomes using psychometric methods and a survey of participants. We report evidence that the assessment activity was reliable and valid, and discuss the strengths and limitations, as well as the practical implications, of our findings.
References
Andrich, D. (1988). Rasch models for measurement. London: Sage. https://doi.org/10.4135/9781412985598
Bisson, M.-J., Gilmore, C., Inglis, M. & Jones, I. (2016). Measuring conceptual understanding using comparative judgement. International Journal of Research in Undergraduate Mathematics Education. Online first. doi: 10.1007/ s40753-016-0024-3 https://doi.org/10.1007/s40753-016-0024-3
Bramley, T. (2015). Investigating the reliability of adaptive comparative judgment (Cambridge Assessment Research Report). Cambridge: Cambridge Assessment.
Bramley, T. (2007). Paired comparison methods. In P. Newton, J.-A. Baird, H. Goldstein, H. Patrick & P. Tymms (Eds.), Techniques for monitoring the comparability of examination standards (pp. 264-294). London: QCA.
Bramley, T., Bell, J. & Pollitt, A. (1998). Assessing changes in standards over time using Thurstone paired comparisons. Education Research and Perspectives, 25, 1-24.
Falchikov, N. & Goldfinch, J. (2000). Student peer assessment in higher education: a meta-analysis comparing peer and teacher marks. Review of Educational Research, 70, 287-322. https://doi.org/10.3102/00346543070003287
Firth, D. (2005). Bradley-Terry models in R. Journal of Statistical Software, 12 (1), 1-12. https://doi.org/10.18637/jss.v012.i01
Heldsinger, S. & Humphry, S. (2010). Using the method of pairwise comparison to obtain reliable teacher assessments. The Australian Educational Researcher, 37, 1-19. https://doi.org/10.1007/BF03216919
Iannone, P. & Simpson, A. (2011). The summative assessment diet: how we assess in mathematics degrees. Teaching Mathematics and Its Applications, 30, 186-196. https://doi.org/10.1093/teamat/hrr017
Johnson, M. & Nádas, R. (2012). A review of the uses of the Kelly's Repertory Grid method in educational assessment and comparability research studies. Educational Research and Evaluation, 18, 425-440. https://doi.org/10.1080/13803611.2012.689715
Jones, I. & Alcock, L. (2014). Peer assessment without assessment criteria. Studies in Higher Education, 39, 1774-1787. https://doi.org/10.1080/03075079.2013.821974
Jones, I. & Inglis, M. (2015). The problem of assessing problem solving: Can comparative judgement help? Educational Studies in Mathematics, 89, 337-355. https://doi.org/10.1007/s10649-015-9607-1
Jones, I., Inglis, M., Gilmore, C. & Hodgen, J. (2013). Measuring conceptual understanding: the case of fractions. In A. M. Lindmeier & A. Heinze (Eds.), Proceedings of the 37th conference of the International Group for the Psychology of Mathematics Education (Vol. 3, pp. 113-120). Kiel: PME.
Jones, I. & Karadeniz, I. (2016). An alternative approach to assessing achievement. In Csíkos, C., Rausch, A. & Szitányi, J. (Eds.), The 40th conference of the International Group for the Psychology of Mathematics Education (Vol. 3, pp. 51-58). Szeged: PME.
Jones, I. & Wheadon, C. (2015). Peer assessment using comparative and absolute judgement. Studies in Educational Evaluation, 47, 93-101. https://doi.org/10.1016/j.stueduc.2015.09.004
Kimbell, R. (2012). Evolving project e-scape for national assessment. International Journal of Technology and Design Education, 22, 135-155. https://doi.org/10.1007/s10798-011-9190-4
McMahon, S. & Jones, I. (2014). A comparative judgement approach to teacher assessment. Assessment in Education: Principles, Policy & Practice, 22, 368-389. https://doi.org/10.1080/0969594X.2014.978839
Newton, P. & Shaw, S. (2014). Validity in educational and psychological assessment. London: Sage. https://doi.org/10.4135/9781446288856
Nisbett, R. E. & Wilson, T. D. (1977). Telling more than we can know: verbal reports on mental processes. Psychological Review, 84, 231-259. https://doi.org/10.1037/0033-295X.84.3.231
Ofqual (2015). A comparison of expected difficulty, actual difficulty and assessment of problem solving across GCSE maths sample assessment materials (Report No. Ofqual/15/5679). London: HMSO.
Pollitt, A. (2012). The method of adaptive comparative judgement. Assessment in Education: Principles, Policy & Practice, 19, 281-300. https://doi.org/10.1080/0969594X.2012.665354
Pollitt, A. (2015). On "reliability" bias in ACJ (Technical report). Cambridge: Cambridge Exam Research.
Rayner, K. (1998). Eye movements in reading and information processing: 20 years of research. Psychological Bulletin, 124, 372-422. https://doi.org/10.1037/0033-2909.124.3.372
Rittle-Johnson, B. & Alibali, M. W. (1999). Conceptual and procedural knowledge of mathematics: Does one lead to the other? Journal of Educational Psychology, 91, 175-189. https://doi.org/10.1037//0022-0663.91.1.175
Star, J. R. (2005). Reconceptualizing procedural knowledge. Journal for Research in Mathematics Education, 36, 404-411.
Tarricone, P. & Newhouse, C. P. (2016). Using comparative judgement and online technologies in the assessment and measurement of creative performance and capability. International Journal of Educational Technology in Higher Education, 1, 13-16. https://doi.org/10.1186/s41239-016-0018-x
Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273-286. https://doi.org/10.1037/h0070288
Thurstone, L. L. (1954). The measurement of values. Psychological Review, 61, 47-58. https://doi.org/10.1037/h0060035
Topping, K. J. (2009). Peer assessment. Theory Into Practice, 48, 20-27. https://doi.org/10.1080/00405840802577569
Topping, K. (2010). Methodological quandaries in studying process and outcomes in peer assessment. Learning and Instruction, 20, 339-43. https://doi.org/10.1016/j.learninstruc.2009.08.003
Wheadon, C. (2015). The opposite of adaptivity? (Blog post, Feb. 10, 2015). Retrieved from https://blog.nomoremarking.com/the-opposite-of-adaptivity-c26771d21d50#.2jably9kx
Downloads
Published
How to Cite
Issue
Section
License
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
