The Future of Education
Marking Reliability and Marker Perceptions: A Comparative Analysis of Face-to-Face and Online Standardisation Approaches
Magda A. Werno, Cambridge International Examinations (United Kingdom)
Sarah Hughes, Cambridge International Examinations (United Kingdom)
Lorna A. Stabler, Cambridge International Examinations (United Kingdom)
Stuart D. Shaw, Cambridge International Examinations (United Kingdom)
Abstract
Recent advancements in technology have contributed to an increase in the use of computer-facilitated administration and management procedures in large-scale educational assessment. Therefore, it is crucial to ensure that online marking and standardisation procedures are associated with at least comparable or higher levels of marking reliability relative to more traditional approaches.
This study adopted a mixed-method approach to examine the operationally available data from 2011–2014 June examination series, as well as perceptions of 21 markers involved in face-to-face and online standardisation in relation to one of the IGCSE (International General Certificate of Secondary Education) Geography components. The objective of the quantitative part of this study was to establish whether marker reliability (measured as deviation from the mark awarded by the Principal Examiner) varies as a function of standardisation mode. The qualitative part of the study aimed to explore the subjective experiences, perceptions, and preferences of markers in relation to the two standardisation methods.
Due to a large amount of variance and a lack of consistent patterns in the 2011–2014 data, no sufficient evidence was found to conclude that marker reliability was directly influenced by the introduction of the online standardisation process in IGCSE Geography. However, the examiners expressed a clear preference for face-to-face standardisation, rather than the online standardisation method. The mode of standardisation was found to be instrumental to examiners’ subjective perceptions and experiences of marking and the extent to which they felt valued, engaged, and confident in terms of internalising and applying the mark scheme.
Based on the findings of this research, a number of recommendations were made in relation to training provision, mark scheme development, and the use of more advanced technologies to enhance group discussions and improve the quality of communication and feedback to markers during the standardisation process.
