63 Content analysis | 3 The operationalisation of monitoring,12 was challenging due to limited information on good monitoring practice.44,45 Therefore, it was decided to complement a single rating item (‘Does the portfolio show signs of monitoring?’) with memos of the observed monitoring activities. During the rating of monitoring we focused on the coherence between different elements of the portfolio: to what extent did themes and topics considered during feedback and reflections recur in learning goals and plans, and vice versa. After a first version of the codebook was approved by all authors, two raters (RG & CB) proof tested the codebook on a sample of portfolios, leading to adaptations of the codebook. Subsequently, two sittings were performed during which the raters used the codebook to rate the same randomly drawn portfolios, after which adaptations were made. During this process, interrater reliabilities (IRR) were calculated multiple times using Krippendorff’s alpha46,47 and percentage of agreement. IRR improved with practice and proved to be acceptable during the definitive rating (see Appendix B). Figure 1 shows the process that led to the final version of the codebook and the definitive rating of portfolios. During the ratings, the content analysed was confined to material covering one study year, to make sure that the rating period was equal for all portfolios included in the content analysis, regardless of the cohort the portfolio belonged to. To secure that we rated material that was collected during an entire study year, we chose to include the content from the year prior to the cohort the portfolio belonged to, e.g. content from the first year was included for portfolios that belonged to the second year cohort, content from the second year was included for portfolios that belonged to the third year cohort, etc. After the rating was completed the ratings were transferred to an SPSS data file. Frequencies were calculated to gain insight in the number of portfolios that met the different SRL criteria for good practice from the codebook. Ethical Approval and Reflexivity Statement The ethical review board of the Dutch Organization of Medical Education (NVMO) approved the study under NERB number 786. The research team consisted of a psychologist (RG), physicians (BT, NS, AK, RL) and a health scientist and educationalist (SH). Before starting her psychology study, the first researcher also studied mass communication, the field from which content analysis originates. The research assistant (CB), who supported the content analysis, has a background in psychology.
RkJQdWJsaXNoZXIy MjY0ODMw