Time to reflect How portfolio use helps and hinders self-regulated learning T IME TO REFLECT How portfolio use helps and hinders self-regulated learning Rozemarijn van der Gulden
Time to reflect How portfolio use helps and hinders self-regulated learning Rozemarijn van der Gulden
ISBN 978-94-6473-062-3 Cover Przewalski Ontwerpers Design/lay-out and print Promotie In Zicht | www.promotie-inzicht.nl De uitgave van dit proefschrift is mede ondersteund door de Nederlandse Vereniging voor Medisch Onderwijs. © Rozemarijn van der Gulden, 2023 All rights are reserved. No part of this book may be reproduced, distributed, stored in a retrieval system, or transmitted in any form or by any means, without prior written permission of the author.
Proefschrift ter verkrijging van de graad van doctor aan de Radboud Universiteit Nijmegen op gezag van de rector magnificus prof. dr. J.H.J.M. van Krieken, volgens besluit van het college voor promoties in het openbaar te verdedigen op dinsdag 2 mei 2023 om 10.30 uur precies door Rozemarijn van der Gulden geboren op 26 september 1991 te Nijmegen Time to reflect How portfolio use helps and hinders self-regulated learning
Promotoren Prof. dr. N.D. Scherpbier-de Haan Prof. dr. S. Heeneman (UM) Copromotoren Dr. B.P.A. Thoonen Dr. A.A. Timmerman (UM) Manuscriptcommissie Prof. dr. C.R.M.G. Fluit Dr. A.A.E.M. van der Velden Prof. dr. R.A.M.J. Damoiseaux (UMCU)
Content Chapter 1 Introduction 7 Chapter 2 How does portfolio use affect self-regulated learning in clinical workplace learning: What works, for whom, and in what contexts? R. van der Gulden, A.A. Timmerman, J.W.M. Muris, B.P.A. Thoonen, S. Heeneman, N.D. Scherpbier-de Haan Published in: Perspectives in Medical Education. 2022;11(5):247-257. doi:10.1007/s40037-022-00727-7 19 Chapter 3 How is self-regulated learning documented in portfolios of trainees? A content analysis R. van der Gulden, S. Heeneman, A.W.M. Kramer, R.F.J.M. Laan, N.D. Scherpbier-de Haan, B.P.A. Thoonen Published in: BMC Medical Education. 2020;20(1):205. doi:10.1186/s12909-020-02114-4 55 Chapter 4 How does portfolio use support self-regulated learning during General Practitioner specialty training? A qualitative focus group study R. van der Gulden, A.A. Timmerman, M.H. Sagasser, A.W.M. Kramer, N.D. Scherpbier-de Haan, B.P.A. Thoonen, S. Heeneman Published in: BMJ Open. 2023;13(2):e066879. doi:10.1136/bmjopen-2022-066879 85 Chapter 5 How to manage tensions between accountability and learner agency when using a multipurpose portfolio? R. van der Gulden, B.P.A. Thoonen, S. Heeneman, J.W.M. Muris, M.H. Sagasser, A.A. Timmerman, N.D. Scherpbier-de Haan Under review 107 Chapter 6 A philosophical discussion of the support of self-regulated learning in medical education: the treasure hunt approach versus the (Dutch) ‘dropping’ approach R. van der Gulden, M. Veen, B.P.A. Thoonen Accepted by Teaching and Learning in Medicine 125 Chapter 7 Discussion 139 Chapter 8 Appendices Summary Nederlandse samenvatting Data management statement Dankwoord Curriculum vitae Portfolio 153 155 159 165 167 169 171
Introduction 1
9 Introduction | 1 Medical training programmes across the world have implemented portfolios for a variety of purposes, e.g. assessment, guidance and/or competency development.1 This thesis focuses on the value of portfolio use for one specific purpose: the support of self-regulated learning (SRL). SRL concerns ‘the degree to which students are metacognitively, motivationally, and behavio[u]rally active participants in their own learning process’ and is considered of eminence during and after education.2(p167) Since portfolio research within the field of clinical workplace learning (WPL) has mainly focused on other purposes of portfolio use, much is still unknown regarding the effectiveness of portfolio use for the support of SRL in this educational setting. Therefore, this thesis was conducted to answer the following research question: How does portfolio use affect SRL during clinical WPL? Since the use of portfolios fits within a tradition of educational practices, three of the practices that define current medical curricula will be discussed before the use of portfolios for the support of SRL is further introduced. Furthermore, the context in which the different studies were performed, a reflexivity statement, the aim, research question and outline of this thesis will be presented in this chapter. Educational practices that define current medical curricula The use of portfolios for the support of SRL is not an isolated practice within medical curricula, instead it strongly relates to other current educational practices. Consequently, knowledge of these practices is fundamental to understand the studies that are part of this thesis. Therefore, three common educational practices within medical curricula are described below: 1. Competency based education3-5 2. Programmatic assessment6-8 and 3. Focus on self-actualisation. 1. Competency based education is centred around the competencies that are considered vital to the field being educated.3 In medical education these vital competencies are often derived from the Canadian Medical Education Directions for Specialists (CanMEDS) physician competency framework. This framework describes competencies of physicians by use of seven roles that are considered essential to the profession: Medical Expert, Communicator, Collaborator, Leader, Health Advocate, Scholar and Professional.9 Competency based education is outcome-based, as it only prescribes the competencies that learners ought to possess in order to graduate. It is therefore considered more flexible and learner-centred than traditional education, that also specifies how and when learners (and teachers) should achieve the desired results.3,10
10 | Chapter 1 2. Programmatic assessment aims to combine two purposes of assessment: learning and decision making.7,8 These aims are pursued by developing an integrated programme of assessment that is intertwined with the curriculum, rather than assessing knowledge and/or skills (once at the end) of individual courses.6 It is assumed that continuous low-stake assessments (no formal decisions are based on these individual assessments) are an impetus for learning, as they inform learners about their progress. Furthermore, aggregation and triangulation of multiple low-stake assessments is expected to be valuable for high-stake decision making (important pass or fail decisions), as it provides information about a learner’s performance during a variety of contexts and moments.8 Moreover, it is expected that by integrating a multiplicity of assessment methods and sources, biases and problems that are tied to individual methods/sources are cancelled out.8 3. The focus of (western) society on self-actualisation is mirrored within medical education. This shows in the intention of medical curricula to support concepts like lifelong learning (‘actively engag[ing] in one’s own learning in different circumstances and contexts’)11(p35) and agency (‘one’s capacity to act purposefully and autonomously’). 12(p1) The introduction of portfolios In light of these educational practices, portfolios have been introduced within medical education around 1990.1 Portfolios originate from the arts, where it is custom to compile ones work in a folder, in order to present it to teachers and/or potential clients. 13 In medical education portfolios are used to collect, amongst others: feedback, reflections, learning objectives, assessment outcomes and professional work.14 While medical educational portfolios started similar to those in the arts (i.e. paper folders), current portfolios are mostly provided in a digital format. Portfolios in this thesis are therefore defined as a purposeful aggregation of (digital) items (e.g. evidence, reflections, feedback) that demonstrate learning, experience or professional growth.14,15 Portfolios are implemented in medical education for a variety of purposes, e.g. assessment, guidance and/or competency development.1 In most cases portfolios are used to serve multiple purposes at the same time.1 As stated above, this thesis focuses specifically on one purpose of portfolio use: the support of SRL.16,17 Before discussing the potential relationship between portfolio use and SRL, information on what defines SRL is provided.
11 Introduction | 1 Self-regulated learning The first models of SRL were introduced in the field of educational psychology around 1986 to better understand the cognitive, motivational and emotional aspects of learning.18 In the years that followed, authors from different disciplines developed and adjusted models of SRL.19 Panadero compared six influential models and concluded that all models present SRL as cyclical and consisting of different phases and subprocesses. He describes SRL as an umbrella term that refers to a variety of constructs related to learning, that cover different domains: (meta)cognition, behaviour, motivation and emotion.18 Sitzmann and Ely performed a meta-analysis and identified sixteen constructs that are considered part of SRL: goal level, planning, monitoring, metacognition, attention, learning strategies, persistence, time management, environmental structuring, help seeking, motivation, emotion control, effort, self-evaluation, attributions and self-efficacy.19 Accordingly, it is difficult to provide a distinct definition of SRL. For the purpose of this thesis a definition of Zimmerman (one of the first authors to describe SRL) was used: ‘the degree to which students are metacognitively, motivationally, and behavio[u]rally active participants in their own learning process’.2(p167) SRL is usually considered important for two reasons. Firstly, SRL is considered to help learners during education, as those learners with a high degree of SRL are expected to learn more effectively than those with a low degree of SRL.20 Proof for this idea is often provided by research that showed a positive correlation between the degree of SRL (usually measured with the use of self-evaluation surveys) and academic performance (e.g. grade point average).21-23 Secondly, those with a high degree of SRL are supposed to engage more easily in lifelong learning after graduation, as there is a high degree of overlap in skills and activities described by the two constructs.24 This second reason can explain the increased popularity of SRL within medical education in particular. In the last decades medical training programmes have focused on lifelong learning, as it is expected that a mindset favouring lifelong learning helps physicians to adequately respond to the ever-changing demands of their profession.25 Moreover, SRL is considered valuable during clinical WPL, because it is expected to help learners keep track of individual learning needs in the unpredictable and sometimes chaotic clinical workplace.24 Consequently, medical training programmes aim to invoke SRL in learners during education. In order to support SRL of learners it is important to know more about differences in the degree of SRL between learners. These differences can be explained by various (interacting) factors: - P ersonality Some learners are more inclined to engage in high degrees of SRL than others, due to differences in personality characteristics. Studies have found a relationship between
12 | Chapter 1 SRL and the dimensions of the Big Five personality traits (these five traits are derived through factor analysis and considered crucial dimensions of personality). In these studies four of the traits (openness to experience, emotional stability, agreeableness and conscientiousness) were positively correlated with the degree of SRL.26-28 - Learning environment The learning environment is another factor that can explain differences in the degree of SRL between learners, as there seem to be certain environmental characteristics that can scaffold SRL. Studies have, for example, identified aspects of the classroom environment that are associated with higher degrees of SRL, e.g. enthusiastic teachers and clear learning goals.29,30 - Co-regulated learning It is proposed that co-regulated learning (‘learners regulate their cognitions, motivation and behaviour together with other individuals in the environment’) improves SRL over time.31(p235),32 Co-regulated learning can consist of formal instructions, e.g. studies have showed the effectivity of training programmes that aim to improve SRL of children in primary and secondary school.21,33 But also more informal interactions, e.g. working on a group assignment with peers, can be accompanied by co-regulated learning.31,32 Portfolios used to support SRL Portfolios are implemented as a tool to support SRL, as they are assumed to provide a learning environment that can scaffold SRL. Moreover, portfolios are usually embedded in interactions with peers, supervisors and/or faculty, and are expected to contribute to co-regulated learning. Studies have indicated that portfolio use can support SRL within various educational settings, such as high school,34 vocational education (e.g. a hairdressing programme),35 higher education (e.g. teaching programmes),36-38 and continuous education (e.g. a language course).39 However, reviews of portfolio use within the field of clinical WPL show that studies in this field have mainly focused on other purposes of portfolio use, e.g. assessment.14,16 Therefore, it is unclear to what extent, in which ways and under what circumstances portfolio use actually supports SRL during clinical WPL. This is troublesome as students, trainees and faculty of medical training programmes invest a lot of time and effort into portfolio use under the assumption that this supports SRL.
13 Introduction | 1 The context of this thesis The studies of this thesis were performed in the context of the Dutch General Practitioner (GP) specialty training. This training programme is provided by eight training institutes across the Netherlands. Three of the training institutes participated in the studies (Radboudumc, Maastricht University, Leiden University Medical Centre). The GP specialty training has a formal competency framework and guidelines for training and assessment (e.g. educational plan and the assessment protocol), but the implementation of these outlines differs between the institutes. The duration of the specialty training programme is generally three years. GP trainees learn while working in general practice and adjacent fields (emergency-, mental health- and chronic care). This WPL is guided by experienced doctors (mostly GPs), who work on site with the trainee and function as supervisors. In addition, trainees receive education in peer trainee groups during a weekly academic day, which is provided by faculty of the training institutes (GPs and behavioural scientists). Trainees are obligated to document information concerning learning and assessment in a digital portfolio. The portfolio is intended to support SRL and is an essential component during programmatic assessment. Accordingly, portfolio content is used to inform annual progress decisions. Content and structure of the portfolio are based on the research- informed NijMaas guidelines.40 The portfolio offers eleven unique pre-structured forms that trainees can fill out themselves and/or send to others in order to obtain feedback (e.g. a mini-CEX and a form to formulate learning objectives and plans). Trainees have the opportunity to select the various forms at their own discretion. Alongside the pre-structured forms, trainees can add their own (learning) documents to a separate folder of the portfolio (e.g. individual trainings plans and test results). Reflexivity statement My experience with (qualitative) research was limited before I started this PhD trajectory. I studied psychology, which prepared me to be quantitative researcher. While I followed a course on qualitative research after my study and was a research assistant for an interview study, I learned most about qualitative research throughout this PhD project. I started with a deductive approach to quantify the presence of SRL within portfolio content (Chapter 3) and grew towards exploring the factors and processes related to portfolio use for the support of SRL through an inductive, qualitative approach (Chapter 4). This development was accompanied by an expansion of worldviews, from post-positivism towards contextualism. While I did learn about ontology, epistemology and other aspects of research philosophy during my study, it was during this project that I first started to realise why this even matters.
14 | Chapter 1 An advantage of my background as psychologist was that I was already familiar with theories of human behaviour and a systems thinking view of complex processes. It was refreshing that some of my attributes that previously had led to frustrations (e.g. meticulous, critical), were of value during the research process. I also consider it valuable that I had no personal experiences with portfolio use (in the context of medical training) and/or ties with the GP specialty training programme, as I was therefore able to approach this research project with an open mind. My PhD trajectory was an intriguing learning process, but at times I was also disappointed in the academic world of which I was now part. For example, when it became clear that it was difficult to publish my studies, not necessarily because of their quality, but because the topic (portfolio use) was no longer of interest to a variety of journals. Furthermore, I found it hard to do justice to the complexity of this topic, as it felt like people (e.g. colleagues, peer-reviewers) seemed to be primarily interested in clear-cut answers and recommendations. Aim and research question of this thesis The aim of this thesis is to gain insight into what extent, in which ways and under what circumstances portfolio use supports SRL during clinical WPL. In order to reach this aim, one central research question was formulated for this thesis: How does portfolio use affect SRL during clinical WPL? Outline of this thesis In order to address the research question three studies based on different research methods were employed to assess current portfolio practices: a realist review, a content analysis and a focus group study. Based on the results of these studies, it was decided to use a systems thinking methodology to propose suggestions for future portfolio use. Lastly, a philosophical discussion of the support of SRL during medical education was written, as multiple complexities concerning the support of SRL were encountered during the different studies of this thesis. This material is presented in the following chapters: - Chapter 2 presents the realist review, which consisted of two phases. During the first phase, a programme theory was formulated. During the second phase an in-depth literature searchwas performed. Through extraction of context-mechanism-outcome configurations a model was proposed that describes how contextual factors and portfolio mechanisms can impact SRL outcomes.
15 Introduction | 1 - Chapter 3 presents the content analysis. In order to gain insight into the presence of SRL constructs within portfolio content of GP trainees, a codebook was developed. The codebook was used to rate the presence of criteria for good SRL practices within ninety portfolios of GP trainees. - Chapter 4 presents the focus group study. Nine focus groups were performed with three different stakeholder groups (trainees, supervisors and faculty). Template analysis of the verbatim transcripts was performed to formulate overarching themes that could clarify the role of portfolio use for the support of SRL. - Chapter 5 describes the use of a systems thinking methodology that was based on the polarity thinkingTM framework, to explore directions to manage tensions that are inextricably linked to multipurpose portfolio use. - Chapter 6 presents a philosophical discussion of SRL. For this discussion, an analogy based on two (Dutch) childhood games (a treasure hunt and a ‘dropping’) was used to reassess approaches to support SRL. - Chapter 7 contains the general discussion of this thesis. In this chapter the main conclusions of the thesis are provided and reflected on.
16 | Chapter 1 References 1. Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach. 2009;31(9):790-801. doi:10.1080/01421590903139201 2. Zimmerman BJ. Investigating self-regulation and motivation: Historical background, methodological developments, and future prospects. Am Educ Res J. 2008;45(1):166-183. doi:10.3102/0002831207312909 3. Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T. Toward a definition of competency-based education in medicine: a systematic review of published definitions. Med Teach. 2010;32(8):631-637. doi:10.310 9/0142159X.2010.500898 4. Morcke AM, Dornan T, Eika B. Outcome (competency) based education: an exploration of its origins, theoretical basis, and empirical evidence. Adv Health Sci Educ Theory Pract. 2013;18(4):851-863. doi:10.1007/s10459-012-9405-9 5. Touchie C, ten Cate O. The promise, perils, problems and progress of competency-based medical education. Med Educ. 2016;50(1):93-100. doi:10.1111/medu.12839 6. van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ. 2005;39(3):309-317. doi:10.1111/j.1365-2929.2005.02094.x 7. Schuwirth LW, Van der Vleuten CP. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach. 2011;33(6):478-485. doi:10.3109/0142159X.2011.565828 8. van der Vleuten CP, Schuwirth LW, Driessen EW, et al. A model for programmatic assessment fit for purpose. Med Teach. 2012;34(3):205-214. doi:10.3109/0142159X.2012.652239 9. Frank JR, Snell L, Sherbino J, Boucher A. CanMEDS 2015: Physician Competency Framework Series I. Royal College of Physicians and Surgeons of Canada; 2015. 10. IobstWF, Sherbino J, CateOT, et al. Competency-basedmedical education in postgraduatemedical education. Med Teach. 2010;32(8):651-656. doi:10.3109/0142159X.2010.500709 11. Berkhout JJ, Helmich E, Teunissen PW, van der Vleuten CPM, Jaarsma ADC. Context matters when striving to promote active and lifelong learning in medical education. Med Educ. 2018;52(1):34-44. doi:10.1111/medu.13463 12. Nieminen JH, Tai J, Boud D, Henderson M. Student agency in feedback: beyond the individual. Assess Eval High Educ. 2021;47(1):95-108. doi:10.1080/02602938.2021.1887080 13. Van Tartwijk J, Driessen E, Van Der Vleuten C, Stokking K. Factors influencing the successful introduction of portfolios. Qual High Educ. 2007;13(1):69-79. doi:10.1080/13538320701272813 14. Colbert CY, Ownby AR, Butler PM. A review of portfolio use in residency programs and considerations before implementation. Teach Learn Med. 2008;20(4):340-345. doi:10.1080/10401330802384912 15. Gordon JA, Campbell CM. The role of ePortfolios in supporting continuing professional development in practice. Med Teach. 2013;35(4):287-294. doi:10.3109/0142159X.2013.773395 16. Tochel C, Haig A, Hesketh A, et al. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12. Med Teach. 2009;31(4):299-318. doi:10.1080/01421590902883056 17. Buckley S, Coleman J, Davison I, et al. The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Med Teach. 2009;31(4):282298. doi:10.1080/01421590902889897 18. Panadero E. A Review of Self-regulated Learning: Six Models and Four Directions for Research. Front Psychol. 2017;8:422. doi:10.3389/fpsyg.2017.00422 19. Sitzmann T, Ely K. A meta-analysis of self-regulated learning in work-related training and educational attainment: what we know and where we need to go. Psychol Bull. 2011;137(3):421-442. doi:10.1037/a0022777 20. Siddaiah-Subramanya M, Nyandowe M, Zubair O. Self-regulated learning: why is it important compared to traditional learning in medical education?. Adv Med Educ Pract. 2017;8:243-246. doi:10.2147/AMEP.S131780 21. Dignath C, Büttner G. Components of fostering self-regulated learning among students. A meta-analysis on intervention studies at primary and secondary school level. Metacogn Learn. 2008;3(3):231-264. doi:10.1007/ s11409-008-9029-x 22. Richardson M, Abraham C, Bond R. Psychological correlates of university students’ academic performance: a systematic review and meta-analysis. Psychol Bull. 2012;138(2):353-387. doi:10.1037/a0026838 23. Cho KK, Marjadi B, Langendyk V, Hu W. The self-regulated learning of medical students in the clinical environment - a scoping review. BMC Med Educ. 2017;17(1):112. Published 2017 Jul 10. doi:10.1186/s12909-0170956-6
17 Introduction | 1 24. van Houten-Schat MA, Berkhout JJ, van Dijk N, Endedijk MD, Jaarsma ADC, Diemers AD. Self-regulated learning in the clinical context: a systematic review. Med Educ. 2018;52(10):1008-1015. doi:10.1111/medu.13615 25. Murdoch-Eaton D, Whittle S. Generic skills in medical education: developing the tools for successful lifelong learning. Med Educ. 2012;46(1):120-128. doi:10.1111/j.1365-2923.2011.04065.x 26. Kirwan JR, Lounsbury JW, Gibson LW. An investigation of the Big Five and narrow personality traits in relation to self-regulated learning. J Psycholog Behav Sci. 2014;2(1):1-11. 27. Bruso J, Stefaniak J, Bol L. An examination of personality traits as a predictor of the use of self-regulated learning strategies and considerations for online instruction. Educ Technol Res Dev. 2020;68(5):2659-2683. doi:10.1007/s11423-020-09797-y 28. Bidjerano T, Dai DY. The relationship between the big-five model of personality and self-regulated learning strategies. Learn Individ Differ. 2007;17(1):69-81. doi:10.1016/j.lindif.2007.02.001 29. Young MR. The motivational effects of the classroom environment in facilitating self-regulated learning. J Mark Educ. 2005;27(1):25-40. doi:10.1177/0273475304273346 30. Boekaerts M. Self-regulated learning: Where we are today. Int J Educ Res. 1999;31(6):445-457. doi:10.1016/S08830355(99)00014-2 31. Bransen D, Govaerts MJB, Sluijsmans DMA, Driessen EW. Beyond the self: The role of co-regulation in medical students’ self-regulated learning. Med Educ. 2020;54(3):234-241. doi:10.1111/medu.14018 32. Bransen D, Govaerts MJB, Panadero E, Sluijsmans DMA, Driessen EW. Putting self-regulated learning in context: Integrating self-, co-, and socially shared regulation of learning. Med Educ. 2022;56(1):29-36. doi:10.1111/ medu.14566 33. Dignath C, Buettner G, Langfeldt H-P. How can primary school students learn self-regulated learning strategies most effectively?: A meta-analysis on self-regulation training programmes. Educ Res Rev. 2008;3(2):101-129. doi:10.1016/j.edurev.2008.02.003 34. Chang C-C, Tseng K-H, Liang C, Liao Y-M. Constructing and evaluating online goal-setting mechanisms in web-based portfolio assessment system for facilitating self-regulated learning. Comput Educ. 2013;69:237-249. doi:10.1016/j.compedu.2013.07.016 35. Kicken W, Brand-Gruwel S, Van Merriënboer J, Slot W. Design and evaluation of a development portfolio: how to improve students’ self-directed learning skills. Instruc Sci. 2009;37(5):453-473. doi:10.1007/s11251-008-9058-5 36. Jarvinen A, Kohonen V. Promoting professional development in higher education through portfolio assessment. Assess High Educ. 1995;20(1):25-36. doi:10.1080/0260293950200104 37. Abrami PC, Venkatesh V, Meyer EJ, Wade CA. Using electronic portfolios to foster literacy and self-regulated learning skills in elementary students. J Educ Psychol. 2013;105(4):1188-1209. doi:10.1037/a0032448 38. Welsh M. Student perceptions of using the PebblePad e-portfolio system to support self-and peer-based formative assessment. Technol Pedagogy Educ. 2012;21(1):57-83. doi:10.1080/1475939X.2012.659884 39. Chau J, Cheng G. Towards understanding the potential of e-portfolios for independent learning: A qualitative study. Australas J Educ Technol. 2010;26(7):932-950. doi:10.14742/ajet.1026 40. Sagasser MH, Schreurs ML, Kramer AWM, Maiburg B, Mokking H. Richtlijn portfolio voor de huisartsopleiding. Huisartsopleiding Nederland; 2010.
How does portfolio use affect self-regulated learning in clinical workplace learning: What works, for whom, and in what contexts? R. van der Gulden A.A. Timmerman J.W.M. Muris B.P.A. Thoonen S. Heeneman N.D. Scherpbier-de Haan Perspectives in Medical Education. 2022;11(5):247-257. doi:10.1007/s40037-022-00727-7 2
20 | Chapter 2 Abstract Introduction Portfolio use to support self-regulated learning (SRL) during clinical workplace learning is widespread, but much is still unknown regarding its effectiveness. This review aimed to gain insight in the extent to which portfolio use supports SRL and under what circumstances. Methods A realist review was conducted in two phases. First, stakeholder interviews and a scoping search were used to formulate a programme theory that explains how portfolio use could support SRL. Second, an in-depth literature search was conducted. The included papers were coded to extract context-mechanism-outcome configurations (CMOs). These were synthesised to answer the research question. Results Sixteen papers were included (four fulfilled all qualitative rigour criteria). Two primary portfolio mechanisms were established: documenting as a moment of contemplation (learners analyse experiences while writing portfolio reports) and documentation as a reminder of past events (previous portfolio reports aid recall). These mechanisms may explain the positive relationship between portfolio use and self-assessment, reflection and feedback. However, other SRL outcomes were only supported to a limited extent: formulation of learning objectives and plans, and monitoring. The partial support of the programme theory can be explained by interference of contextual factors (e.g., system of assessment) and portfolio related mechanisms (e.g., mentoring). Discussion Portfolio research is falling short both theoretically - in defining and conceptualising SRL - and methodologically. Nevertheless, this review indicates that portfolio use has potential to support SRL. However, the working mechanisms of portfolio use are easily disrupted. These disruptions seem to relate to tensions between different portfolio purposes, which may undermine learners’ motivation.
21 Realist review | 2 Introduction The use of portfolios to support self-regulated learning (SRL) is common practice in medical education. 1,2 Portfolios are a purposeful aggregation of (digital) items (e.g. evidence, reflections, feedback) that demonstrate learning, experience or professional growth.3,4 SRL refers to ‘the degree to which students are metacognitively, motivationally, and behavio[u]rally active participants in their own learning process’.5(p167) The literature indicates that higher levels of SRL are associated with better academic performance and lifelong learning.5,6 However, effective SRL is not self-evident, especially during clinical workplace learning (WPL), since it is difficult for learners to monitor their individual learning needs in the unpredictable and complex clinical setting.6,7 It is assumed that portfolios can mitigate this difficulty.4,8 Previous reviews have examined portfolio use for a variety of purposes, including competency development and assessment.1,2,4,9 With regard to SRL-related outcomes, the reviews are most informative about reflection. Although portfolio use was associated with an increased incidence of reflection, the quality of reflection did not necessarily improve with portfolio use.1 This might be due to learners’ reluctance to disclose their introspections in a document accessible to faculty members who can influence their study prospects.2,9 Furthermore, some evidence suggests that portfolio use can support self-assessment and identification of learning needs.1,2,4,9 However, it was also shown that the use of a portfolio alone is not sufficient for these processes to occur, as several preconditions for successful portfolio use were mentioned, such as encouragement by a mentor2,4,9 and clear portfolio goals and instructions.9 Given the popularity of portfolios, it is important to gain insight in the extent to which and under what circumstances portfolio usage is effective for supporting SRL. Therefore, we conducted a realist review to better understand when and how portfolio use supports SRL during clinical WPL. We used the following research question: How does portfolio use affect SRL during clinical WPL: What works, for whom, and in what contexts? Methods A realist review is suitable to provide a rich and practice-oriented understanding of complex social interventions, such as portfolio use.10 The aim of realist reviews is ‘to unpack the mechanism of how complex programmes work (or why they fail) in particular contexts and settings’.10(p21) To do so, the first step is to formulate a programme theory, that can explain why the programme under review is expected to work. Subsequently, literature is included to search for context-mechanism-outcome configurations (CMOs). In other words: what works for whom in which circumstances? Finally, a synthesis of these CMOs provides insight into the contexts and mechanisms that can explain different outcomes
22 | Chapter 2 of the programme. This approach fitted our aim to better understand when and how portfolio use supports SRL during clinical WPL. Review process The review process consisted of two phases, which are described in more detail below (See Appendix A for a visualisation of the review process). Two reviewers (AT, RG) performed the data-collection and -analysis. They discussed their approach and dilemmas that arose during the review process on a regular basis with the other authors. The standards of the RAMESES project were used to guide our decisions.11 Phase 1: The goal was to formulate a programme theory, which describes how portfolio use is expected to support SRL during clinical WPL. Step 1: Stakeholder interviews. We conducted individual interviews to gather ideas and experiences from portfolio users. We included eight stakeholders from different institutes of the Dutch general practitioner specialty training, all of whom had previous experience with portfolio use and/or guidance. RG performed the interviews using a semi-structured interview guide. The interviews were audio recorded and later summarised. (See Appendix B for information on the interviews). Step 2: Exploratory scoping search. Simultaneously with the stakeholder interviews, we performed a scoping search of PubMed and Web of Science in collaboration with a librarian to gather papers that explain how portfolio use supports SRL (July 2018; See Appendix C for search strings). We selected search terms that included portfolios and SRL or Self-Directed Learning (SDL) during WPL. Although, there are intrinsic differences between SRL and SDL we included both, given the interchangeable use of these terms in the literature.12 The search resulted in 53 references; 45 references remained after removing duplicates. We considered 14 papers useful to formulate a programme theory, as these papers theorised about how portfolio use can support SRL.13-26 Step 3: Formulating the programme theory. To establish a first version of the programme theory, RG and AT examined the interview summaries and included papers to extract ideas and theories that explain how portfolio use supports SRL. Subsequently, the theory was clarified and adapted in discussion with the other authors. During these discussions we recognised that interviewees and papers often (implicitly) assumed a relationship between portfolio use and the completion of learning cycles. The stages of the learning cycles explained by interviewees and in the included papers were similar to the experiential learning cycle described in Kolb’s theoretical framework.27 Therefore, we utilised this experiential learning cycle as middle range theory. This resulted in a final version of the programme theory provided in Figure 1.
23 Realist review | 2 Phase 2: The goal was to provide an overview of the current research regarding portfolio use for the support of SRL during clinical WPL. Step 4: In-depth literature search. The in-depth literature search (February 2019) was supported by the librarian who had also assisted during the scoping search. The original search strings were revised to retrieve references regarding the SRL outcomes that are part of the programme theory, e.g. self-assessment and learning cycle (Appendix D). We searched: Pubmed, CINAHL, ERIC, PsycInfo, Embase, and Web of Science. Step 5a: Title and abstract screening. RG and AT screened title and abstract of the references for inclusion, which required that references needed to concern primary research into portfolio use for the support of SRL/SDL (or related outcomes) during clinical WPL. The first three hundred references were assessed and discussed by both reviewers to establish the approach and definitive list of inclusion criteria (Appendix E). The remaining 1744 references were assessed by one reviewer. However, references that raised any doubts were discussed between the two reviewers. Step 5b: Full-text review. The remaining papers were read and assessed by one of the two reviewers, and discussed if there were any doubts regarding inclusion. Step 6: Assessing rigour. We evaluated the rigour of the included papers, as is prescribed by the realist tradition, to provide information on the credibility and trustworthiness of the papers.28 We discussed information on quality assessment criteria and procedures used in other realist reviews,28-30 to compose the following criteria for the rigour evaluation: 1. There is a clear statement of, and rationale for, the research question/aims. 2. Design and study methods are appropriate to answer the research question. 3. Study findings and conclusions are supported by the data. RG paired with the other authors to decide whether the included papers met these criteria. The criteria were evaluated individually first, and later discussed between the pairs to reach consensus.
24 | Chapter 2 Figure 1 The programme theory that describes how portfolio use is expected to support SRL during clinical WPL. Portfolio use is expected to facilitate learning from experiences during clinical workplace learning (WPL), through the support of the following self-regulated learning outcomes: Attaining self-assessment Learners gain understanding of their own competence level, strengths and weaknesses13-17, by: • Writing reflections on experiences in their portfolio13,15,18-25. • Accumulating feedback in their portfolio21,26. Formulating learning objectives and plans Learners direct their future learning when encountering gaps between current and desired performance levels17,23-25, by: • Formulating learning objectives in their portfolio. Thereby, describing the specific knowledge, skill or attitude that they want to achieve23. • Formulating learning plans in their portfolio. Thereby, describing how they want to achieve their learning objectives and how they will know that they have achieved their objectives. Monitoring Learners monitor their progress towards previously set learning objectives by: • Using self-assessments13,15-17,23,24. Clinical WPL Portfolio use Attaining self-assessment Formulating learning objectives Formulating learning plans Writing re ections Accumulating feedback
25 Realist review | 2 Step 7: Extracting CMOs. The research question guided the formulation of applied definitions for context, mechanism and outcome: Context: T he external factors that affect portfolio use for the support of SRL. These factors could still exist without the portfolio present. Mechanism: T he processes set in motion by portfolio use that influence the degree and/or level of SRL. These processes would not exist without the portfolio present. Outcome: SRL that is generated by portfolio use. The reviewers individually coded the included papers for phrases that described contexts, mechanisms or outcomes as defined above. We only included phrases directly linked to the study that was performed by the authors of the paper, as the focus was on primary data. Subsequently, the reviewers discussed their coding to reach consensus about the CMOs present. It was decided to also include incomplete CMOs, as these can help to clarify how portfolio use works for the support of SRL. Multiple iterations of coding and discussion were performed to formulate the final CMOs. Coding was supported by NVIVO (https://www.qsrinternational.com/nvivo) and Atlas.ti (https://atlasti.com/). Step 8: Synthesising CMOs. To synthesise the CMOs, we visualised each CMO by putting arrows between its context, mechanism and outcome with the use of mindmapping software (http://www.mindomo.com). This enabled us to easily (re)order different contexts, mechanisms and/or outcomes inductively, without losing the connection between individual CMOs. In that way, we could identify overarching contextual factors, by grouping similar contexts together. Likewise, similar mechanisms were abstracted into overarching portfolio (related) mechanisms. We deductively organised the outcomes of the CMOs according to the SRL outcomes of the programme theory: self-assessment, reflection, feedback, learning objectives and plans, and monitoring. Finally, we used the identified contextual factors, portfolio (related) mechanisms and SRL outcomes to compose a model that illustrates how portfolio use supports SRL during clinical WPL. Results Study characteristics We included sixteen papers (See Appendix F for a flowchart of the extraction process).31-46 These papers originated from eleven countries across Africa (2), Asia (3), Europe (8) and North America (3). Studies were conducted in undergraduate medical settings - (5) and postgraduate medical ones (11). Most papers describe an evaluation of portfolio implementation (14). Although quantitative, qualitative and mixed methods designs are present, the questionnaire was the most popular method of data collection (10). Only four
26 | Chapter 2 papers fulfilled the three rigour criteria (See Appendix G for a more elaborate description of the included papers). Overview of identified CMOs An overview of the identified CMOs is provided in table 1. The different CMOs are organised according to the SRL outcome they were assigned to. Amodel of how portfolio use works for the support of SRL during WPL The relationship between contextual factors, portfolio (related) mechanisms and SRL outcomes is visualised in Figure 2. Below we explain the components of the model. Contextual factors The contexts of the CMOs were abstracted into four contextual factors. First, we identified a relationship between portfolio use and characteristics of workplace learning.33,36,39,41-44 Multiple papers referred to aspects of WPL that can complicate learning, such as limited access to computers, time constraints resulting from a high workload and the frenetic pace of the clinical setting. Some papers explained that portfolio requirements added to these pressures.36,39 In contrast, other papers described that portfolio use resolved issues related to WPL by creating moments of contemplation, i.e. learners that were able to use their portfolio during busy workdays were provided with an opportunity to pause and think about what had happened.36,39,43,44 The second contextual factor concerns the system of assessment in place.36,39 Two studies showed that formal requirements concerning the amount and/or quality of portfolio reports in the context of summative assessment, were related to stress and anxiety of learners.36,39 These assessment requirements might interfere with the support of portfolio use for SRL, as learners in one of these studies did not experience any educational benefits of the portfolio.39 Two papers mentioned geographical/cultural characteristics in relation to portfolio use.36,40 The face-saving culture present in Taiwan was thought to influence feedback seeking of learners with the portfolio, due to a fear of negative feedback.36 In addition, Jenkins et al. describe how the exceptionally high service demands in South-Africa, limited time available for portfolio use.40 In the same two studies the final contextual factor surfaced: prior experience with SRL constructs.36,40 It was described that limited experience with reflection on the part of learners and supervisors resulted in low awareness and documentation of reflection in the portfolio.40 Furthermore, the learners in the study of Fu et al. found it difficult to individually perform a learning needs assessment, which resulted in inauthentic portfolio reports.36
27 Realist review | 2 Portfolio (related) mechanisms The mechanisms of the CMOs were distilled into: • Portfolio mechanisms: primary mechanisms inherent to portfolio use, that seem to affect SRL directly. • Portfolio related mechanism: mechanisms related to portfolio use that seem to affect the primary portfolio mechanisms and thereby also SRL. The first portfolio mechanism affecting SRL concerns documenting as a moment of contemplation.33,35,37,42-45 Documenting was reported to help learners analyse their experiences; writing a portfolio report helped learners to capture the essence of their experiences.35,42,43,45 The second portfolio mechanism is documentation as a reminder of past events.32,33,44 Previously documented information helped learners to remember what happened before, which provided an opportunity to mentally return to these events.33,44 The first two portfolio related mechanisms are controlled by training institutes and their faculty. First, some papers mentioned conditions of portfolio use (e.g. a digital format, the provided instructions or privacy matters) that affected portfolio use and thus the potential for documenting and use of documentation.31,33,36,41,43,44,46 The second portfolio related mechanism controllable by training institutes and/or faculty concerns mentoring during portfolio use.33,36,39,44-46 Some papers describe how portfolio use supported mentoring and, in this way, SRL: feedback was exchanged more easily, since sensitive or otherwise neglected topics were included in the portfolio and therefore discussed.44,45 However, others described that supervisors could only provide valuable feedback when learners provided suitable portfolio reports.36,46 Also WPL (contextual factor) can interfere with the possibility to exchange feedback through the portfolio: learners were hesitant to ask busy supervisors for portfolio contributions, as they did not want to add to the supervisor’s workload.39 The other portfolio related mechanisms concern different aspects of learners’ reactions to portfolio use. Multiple papers described learners’ assumptions about portfolio use.33,36,40,42,44 Two papers showed that positive assumptions about the potential of the portfolio for reflection and feedback were related to the occurrence of these SRL outcomes.36,44 Likewise, Kjaer et al. identified negative assumptions about portfolio use during clinical care that related to doubts about the portfolio’s educational benefit.42 Two other papers reported that learners only considered the portfolio suitable to provide evidence of competence, and not for (extensive) reflection.33,40 Furthermore, two papers referred to learners’ feelings about portfolio use related to SRL.36,39 There were learners that experienced stress and anxiety in reaction to the system of assessment, which potentially limited SRL.36,39 In contrast, Fu et al. also mention positive effects of portfolio use on feelings. They mentioned, for example, how learners experienced a positive self-image when supervisors took the effort to provide them individualised feedback.36
28 | Chapter 2 Table 1 The context–mechanism–outcome configurations identified during the extraction process of the sixteen included papers focusing on portfolio use in clinical workplace learning (WPL), ordered according to SRL outcomes. Contexta Mechanismb Outcomec It was difficult to ensure protected time within the clinical hospital setting (WPL) Learners thought that time spent on the portfolio reduced the time available to spend on patients (AP) Learners doubted the educational benefit of the portfolio.42 - Learners did not have a personal work and/or storage space within the hospital (WPL) - The busy, frenetic pace of the clinical setting (WPL) - Summative assessment; more specifically, requirements regarding the (number of ) portfolio reports (SA) Learners struggled to collect the required portfolio forms, because they usually did not have access to their portfolio at the workplace. And because they were reluctant to add to the workload of colleagues by asking them to observe and provide feedback on routine procedures (M). Consequently, the portfolio requirements induced stress, anxiety, and other negative feelings (F) Learners doubted the educational benefit of the portfolio.39 Context Mechanism Outcome Self-assessment (+) The busy, frenetic pace of the clinical setting, which can result in the training year passing by without any concrete developments (WPL) The portfolio provided a structure to document information during busy workdays (DC). Subsequently, this documentation reminded learners of what happened before and thus provided an opportunity to look back (DR) Learners reviewed their weak and strong points.44 Learners documented frank and open portfolio reports about their deficiencies and how they had tried to remedy them (DC) Learners were aware of their feelings, attitudes and concerns.37 Distilling clinical experiences into portfolio reports helped to analyse these experiences (DC) Learners engaged in constructive self-criticism, thereby clarifying thoughts and feelings and identifying proficiencies and deficiencies in performance.35 The portfolio facilitated that all information was stored in one place (DR) Learners identified gaps in learning.32
29 Realist review | 2 Context Mechanism Outcome Self-assessment (-) - Learners did not have a personal work and/or storage space within the hospital (WPL) - The busy, frenetic pace of the clinical setting (WPL) - Summative assessment; more specifically, requirements regarding the (number of ) portfolio reports (SA) Learners struggled to collect the required portfolio forms, because they usually did not have access to their portfolio at the workplace. And because they were reluctant to add to the workload of colleagues by asking them to observe and provide feedback on routine procedures (M). Consequently, the portfolio requirements induced stress, anxiety, and other negative feelings (F) Learners did not experience the portfolio as help in the identification of strengths and weaknesses in one’s own performance.39 Context Mechanism Outcome Reflection (+) - Time-pressure (WPL) - Upcoming job interviews (WPL/A) Learners documented short, superficial, selective and/or strategic portfolio reports (DC), because of time constraints, the idea that the portfolio was a record of achievement (AP), and privacy concerns (CP). Nevertheless, these short notes helped in remembering the events that had taken place (DR) Although documented reflections were superficial, learners could engage in deep reflection at a later moment when they were reminded of the events that had taken place.33 The busy, frenetic pace of the clinical setting (WPL) Brief dedicated time to write in the portfolio needed to be secured (CP). Reflections could be captured as they happened.41 The busy, frenetic pace of the clinical setting (WPL) Learners experienced limited time for reflection. Yet portfolio writing was considered to take up no additional time, as it was possible to do this in between the regular tasks and responsibilities (AP) Learners engaged in reflection.44 The busy, frenetic pace of the clinical setting made it difficult to synthesize learning experiences (WPL) However, documenting in the portfolio provided an intentional deliberate moment to pause and think about what had happened during the day (DC). Learners engaged in reflection.43 Learners documented frank and open portfolio reports about their deficiencies and how they had tried to remedy them (DC) Learners engaged in reflection.37
www.proefschriften.netRkJQdWJsaXNoZXIy MjY0ODMw