Pixel International Conferences

This paper presents one part of the PhD research realized within the broader framework of test versions equivalence in high-stakes testing context, particularly in the Slovak upper-secondary school leaving exam of English (Maturita). Test versions equivalence will be shown as one of the crucial aspects for meaningful and fair interpretation and use of test results at individual, institutional and system level and, at the same time, as one of the aspects of validity frequently taken for granted. The objectives of the research are to investigate the extent of equivalence of the test versions used between 2012 and 2015, to interpret and apply the findings, to propose implementation of processes in the test development with the objective to reach test version equivalence, and to create a theoretical framework for the development of equivalent test versions. In the part of the research presented in this paper, CEFR was used as a descriptive tool for the investigation of content and construct equivalence as the Maturita exam claims to be linked to the B1 CEFR level. The analysis of the content structure by expert judgement using item-descriptor matching method was realized and the agreement and reliability coefficients were calculated. Preliminary findings indicate that CEFR descriptors can be problematic for describing the test content and construct at a discrete, detailed level, as the descriptors differ in terms of their completeness, structure and specificity level. The use of CEFR-based descriptive models is also problematized by the fact that the characteristics of test items are seen as the result of the interaction among test takers' proficiency, design of the item, expert judges' characteristics and their internalization of the judgement task. The key findings of the analysis of content structure and the usefulness of the CEFR for this purpose will be discussed in light of the whole research project and some possible further steps will be presented.

Keywords: expert judgement, CEFR descriptors, content and construct equivalence, test versions equivalence;