Get started
Work

Relating language examinations to the common European framework of reference for languages: learning, teaching, assessment (CEFR): a manual

Year: 2009
Type: report
Abstract: Confirmed Estimation of Overall Examination LevelPage 34 / 152  increase the transparency of the content of examinations (theoretical rationale, aims of examination, etc.);  increase the transparency of the intended level of examinations;  give test takers, test users and teaching and testing professionals the opportunity to analyse the quality of an examination and of the claimed relation with the CEFR;  provide an argumentation why some of the recommended procedures may not have been followed;  provide future researchers with a wider range of techniques to supplement those outlined in this Manual.It is important to note that while the Manual covers a broad range of activities, its aim is limited: It provides a guide specifically focused on procedures involved in the justification of a claim that a certain examination or test is linked to the CEFR.Standard Setting (Chapter 6): The crucial point in the process of linking an examination to the CEFR is the establishment of a decision rule to allocate students to one of the CEFR levels on the basis of their performance in the examination.Usually this takes the form of deciding on cut-off scores, borderline performances.The preceding stages of Familiarisation, Specification and Standardisation can be seen as preparatory activities to lead to valid and rational decisions.Chapter 6 describes procedures to arrive at the final decision of setting cut scores.The material presented there draws on an extensive literature on standard setting, and the procedures presented in Chapter 6 are a selection from the many available procedures deemed to be suitable in the context of language testing.Additional procedures based on the exploitation of teacher judgments and IRT to incorporate an external criterion (e.g.CEFR illustrative items, or teacher assessments with CEFR illustrative descriptors) into a linking study are presented in Extra Material provided by Brian North and Neil Jones.Validation (Chapter 7): While the preceding stages of Familiarisation, Specification, Standardisation and Standard Setting can be conceived roughly to represent a chronological order of activities, it would be naïve to postpone validation activities until everything has been done, and to conceive it as an ultimate verdict on the quality of the linking process.Validation must rather be seen as a continuous process of quality monitoring, giving an answer to the general question: "Did we reach the aims set for this activity?"A simple, but nevertheless important example has already been referred to: it is important to provide CEFR familiarisation and standardisation training, but it is equally important to check if such activities have been successful; this is precisely what is meant by validation.Aspects of validity and procedures to collect validity evidence are described in this final chapter.Aspects of validity and procedures on how to collect validity evidence have been put together in the final chapter (Chapter 7) of this Manual. Use of the CEFRA common framework of reference enables different examinations be to related to each other indirectly without any claim that two examinations are exactly equivalent.The focus of examinations may vary but their coverage can be profiled with the categories and levels of the framework.In the same way that no two learners at Level B2 are at Level B2 for the same reason, no two examinations at Level B2 have completely identical profiles.The parts of the CEFR most relevant for linking examinations are:  Chapter 3 "The Common Reference Levels";  Chapter 4 "Language Use and the Language User"with scales for Communicative Language Activities and for Communicative Language Strategies;  Chapter 5 "The User/Learner's Competences", particularly Section 5.2 "Communicative LanguageCompetences" with the illustrative scales for aspects of linguistic, pragmatic and socio-linguistic competence.Users of this Manual will find the full text of the CEFR and related documents, plus a number of useful tools on the Council of Europe website, including the following: Documents  The CEFR in English and French, including appendices. Links to other language versions on the Council of Europe website (www.coe.int/lang;www.coe.int/portfolio) The Manual, including appendices. The forms and reference Grids included in the Manual. The Reference Supplement. Content Analysis Grids CEFR Content Analysis Grid for listening and reading (sometimes referred to as "the Dutch CEFR Grid"): Appendix B1.  CEFR Content Analysis Grids for speaking and writing, developed by ALTE: Appendix B2.Illustrative Descriptors (www.coe.int/portfolio) The descriptors from the CEFR (in English). The descriptor bank from the European Language Portfolio, demonstrating the relationship between those descriptors and the original CEFR descriptors. A collation of C1/C2 descriptors (in English) from the CEFR and related projects that marks which descriptors were calibrated to CEFR levels and which were not.Illustrative Samples  Documentation for DVDs of illustrative samples of spoken performance by adults, available at the time of writing for English, French, Italian and Portuguese 1 . Illustrative samples of written performance, available at the time of writing for English, French, German, Portuguese and Italian. Illustrative items for listening and reading for English, French, German, Italian and Spanish.Other related resources will be added to this CEFR "Toolkit" listed on www.coe.int/lang and www.coe.int/portfolio as they become available. Especially Relevant Parts of the CEFRIn considering which specific resources in the CEFR to consult, the user may find the following scales and descriptions of the levels especially useful from a global perspective:English Version French Version Overviews of the Common Reference Levels  Table 1 "Common Reference Levels" in Chapter 3 Page 24 Page 25  Section 3.6."Content Coherence in Common Reference Levels" Pages 3336 Pages 3234  Document B5 "Coherence in Descriptor Calibration" Pages 223224 Pages 159160  "Levels of Proficiency in the ALTE Framework" Pages 249-50 Pages 176177 Overviews of Communicative Activities  Table 2, Portfolio Self-assessment Grid Pages 2627 Pages 2627  DIALANG Document C3 "Elaborated Descriptive Scales …"
Language: English
Open Access status: green
OpenAlex ID: W2618994416
Cites: 27
Cited by: 119
Related to: 10