64 | Chapter 3 Figure 1 The process that led to the definitive rating of portfolios. Proof testing of the codebook (RG) Training session with research assistant (RG & CB) Independently rating the same 24 randomly drawn portfolios (RG & CB) Discussing items lacking conformity (RG & CB) Independently rating the same 10 randomly drawn portfolios (RG & CB) Discussing items lacking conformity (RG &CB) De nitive rating of 90 portfolios (RG & CB) Development of the codebook, based on the literature on good practice (RG) – Addition of question 4 – Decision to pass portfolios without learning goals on the pre-structured forms – IRR measurement 1 (Appendix B) – IRR measurement 2 (Appendix B) – IRR measurement 3: 20 portfolios were rated by the two raters (Appendix B). Items that did not correspond were discussed leading to a nal decision for these items. – Elimination of question 12 – Addition of question 25 – Fine-tuning of instructions – Fine-tuning of instructions – De nitive version of codebook (Appendix A)
RkJQdWJsaXNoZXIy MjY0ODMw