Skip to main content

How is self-regulated learning documented in e-portfolios of trainees? A content analysis

Abstract

Background

It is assumed that portfolios contribute to self-regulated learning (SRL). Presence of these SRL processes within the documentation kept in portfolios is presupposed in common educational practices, such as the assessment of reflective entries. However, questions can be asked considering the presence of SRL within portfolios. The aim of this study was to gain insight into the documentation of SRL processes within the electronic (e)-portfolio content of medical trainees. SRL consists of numerous processes, for this study the focus was on self-assessment via reflection and feedback, goal-setting and planning, and monitoring, as these are the processes that health professions education research mentions to be supported by portfolios.

Methods

A database containing 1022 anonymous e-portfolios from General Practitioner trainees was used to provide descriptive statistics of the various available e-portfolio forms. This was followed by a quantitative content analysis of 90 e-portfolios, for which, a codebook was constructed to rate the documentation of the included SRL processes.

Results

The numbers of forms in the e-portfolios varied to a great extent. Content analysis showed a limited documentation of reflective entries, and available entries mainly described events and experiences without explanations and context. Feedback was generally limited to comments on what went well and lacked specificity, context and suggestions for future action. Learning goals and plans were short of specificity, but did contain challenging topics and different goals were compatible with each other. 75% of the e-portfolios showed (limited) signs of monitoring.

Conclusions

The e-portfolio content showed limited documentation of SRL processes. As documentation of SRL requires time and asks for a high level of introspection and writing skills, one cannot expect documentation of SRL processes to appear in e-portfolio content without efforts.

Peer Review reports

Background

It is assumed that portfolios can contribute to self-regulated learning (SRL) [1,2,3,4,5]. SRL refers to ‘the degree to which students are metacognitively, motivationally, and behavio[u]rally active participants in their own learning process’ ([6], p. 167). It can be seen as an umbrella term that covers a multitude of processes that supposedly influence learning, including skills (e.g. time management), affective constructs (e.g. self-efficacy) and metacognitive processes (e.g. reflection) [7]. Portfolios, ‘the collection of evidence that learning has taken place’ ([8], p.192), are used to support and document some of these SRL processes. The health professions education literature generally reports the following SRL processes to be supported by portfolios: self-assessment, which can be informed internally via reflection and externally via feedback, goal-setting and planning, and monitoring [9,10,11].

The supposed value of portfolios for SRL is one factor explaining the widespread implementation of portfolios in medical training institutes [12], as SRL is considered an essential skill for medical students and physicians [13,14,15]. Working and learning in a clinical workplace can be unpredictable and sometimes chaotic [16], and it is expected that only those who regulate their learning well are able to keep track of individual educational needs in such a hectic environment [14]. SRL proficiency is thus expected to be of eminence during workplace-based education [14]. Medical educators are therefore trying to identify and optimise tools and procedures that foster SRL during this type of education.

While educators see in portfolios the potential to function as such a SRL fostering tool, users are more sceptical about the educational value of portfolios. The results of studies that rely on user experiences and perceptions show that opinions of learners concerning the support of their portfolio for SRL are mixed at best [17,18,19,20,21,22,23,24]. In some studies, users indicated that their learning is supported by the portfolio [19,20,21]. However, other studies showed that users experience limited value of portfolios for feedback, reflection, and in achieving learning goals [17, 18, 22,23,24]. Users indicated, for example, that reflecting via the portfolio is too restrictive and face-to-face feedback gets neglected or compromised by the portfolio [22, 23].

The negative appraisals of users might be prompted by unfavourable experiences with portfolios, that can be accounted for by factors other than SRL, such as the burden and time needed to complete a portfolio [12, 22]. Likewise, unclarity about the purpose of the portfolio can add to negative user perceptions [25]. Trainees have stated that they experience the portfolio as a way to provide evidence of learning to faculty instead of a tool for self-development [23]. On the other hand, considering the comprehensiveness of SRL and the complexities associated with supporting and/or scaffolding SRL [26, 27], it is also possible that portfolios do not contribute to the SRL processes for which they are deployed within health profession education. Studies looking into user experiences and perspectives cannot quantify the extent to which SRL processes are taking place, therefore also other data sources and methods are required when aiming to establish if and how portfolios contribute to SRL processes.

An additional data source is the content of portfolios, which could clarify to what extent the different SRL processes are present in the documentation that learners keep in their portfolio. In common educational practice, such as the assessment of reflections and learning goals and the use of portfolio content as starting point for supervision, the presence of SRL processes in portfolio documentation is presupposed. However, to our knowledge previous studies have only analysed the content of specific (assessment) instruments, such as the mini-CEX, for the presence and quality of documented SRL processes [28,29,30]. These studies show that the presence and quality of feedback, reflection and action plans documented could be improved substantially. Consequently, there is too little empirical evidence to substantiate the assumption that SRL processes are present in the documentation of portfolios.

The aim of the current study is to gain insight into the documentation of different SRL processes within the e-portfolio content of trainees, thereby focusing on those SRL processes that are, within the field of health professions education, expected to be supported by and documented in portfolios. The research question is: To what extent are SRL processes, specifically self-assessment via reflection and feedback, goal-setting and planning, and monitoring, documented in e-portfolio content?

Methods

Context

The setting of this research is the Dutch General Practitioner (GP) speciality training, as provided by the eight institutes related to the eight University Medical Centres in the Netherlands. The formal framework and guidelines of the speciality training, such as the assessment protocol, are similar for the eight institutes. During the 3 years of speciality training, GP trainees learn, while working in general practice (during their first- and third year) and adjacent fields such as psychiatry and hospital emergency care (during their second year). This workplace-based learning is guided by experienced doctors (mostly GPs), who work on site with the trainee and function as supervisors. In addition, GP trainees receive education in peer-trainee groups, during a weekly academic day, which is provided by GP teachers and behavioural scientists at the different institutes.

Trainees are obligated to document information concerning assessment and learning in an e-portfolio. Content and structure of the e-portfolio are based on the research-informed NijMaas guidelines [31]. The mission of these guidelines was to propose an e-portfolio that combines programmatic assessment [32] with support for SRL. The e-portfolio contains eleven unique forms (see Table 1). These eleven forms can be added as often as required by trainees. Alongside the pre-structured forms, trainees can add their own documents to a separate folder of the e-portfolio.

Table 1 Description of the different pre-structured e-portfolio forms and their – during design - envisioned supportive value for SRL processes (with meaning supportive and – meaning no additional support)

Trainees share their e-portfolio with teachers, supervisors and heads of the GP Training Institute. Every 4 months, teachers and supervisors assess competency development (via a Compass-form, see Table 1), and provide their advice on advancement in the e-portfolio. Besides that, the e-portfolio content is used to inform the annual summative progress decision made by the head of the GP Training Institute. For this, the e-portfolios also contain the results of mandatory progress tests and video-assessments of doctor-patient communications.

Trainees have access to technical instructions of the e-portfolio and the (programmatic) assessment procedures of their institute, which are derived from the national assessment protocol.

Procedure

A mirror version of the actual e-portfolio database was constructed. In this mirror database anonymous e-portfolios of trainees from three institutes were stored, from introduction of the e-portfolio (2013) to the end of March 2018. The folder containing personal documents could not be anonymised, therefore, only the pre-structured forms were transferred to the mirror database.

Records containing data of users other than trainees, such as administrators, were excluded from analyses. The database also consisted of e-portfolios that were not (appropriately) used due to e.g. trainees transferring to another institute or dropping-out. To exclude these e-portfolios, absence of any Compass-form (an obligatory form that needs to be completed three times a year, see Table 1) was used as exclusion criterion. All e-portfolios (N = 128) without a Compass-form were excluded. In addition, e-portfolios of trainees that were already in their second- or third year during implementation of the e-portfolio were excluded, as these trainees in many cases kept using their old paper portfolio alongside the e-portfolio (N = 112). This resulted in a final dataset consisting of 1022 e-portfolios.

To consider differences in duration of training between trainees, the duration of e-portfolio use was specified for all e-portfolios by calculating the difference between March 2018 and the date the specific e-portfolio was created. This duration was used to assign the e-portfolios into three cohorts, which globally represented the 3 years of speciality training.

Design

This study was conducted in two phases using quantitative measures. The first phase, was carried out to provide descriptive statistics of the use of pre-structured forms of the 1022 included e-portfolios. During the second phase, a quantitative content analysis was performed to rate the manifestation of the included SRL processes in the content of 90 e-portfolios.

These 90 e-portfolios were selected using stratified random sampling, to ensure that the three institutes and 3 years of speciality training were equally represented in the sample. It was decided within the research team to select ten e-portfolios per group (institute x year) to keep the rating process feasible. As the e-portfolio was implemented later in one institute there were too little portfolios available to rate material from the third year, so the decision was made to rate fifteen e-portfolios for the first 2 years for this institute.

Analyses

Phase one: number of forms

Data were exported from the mirror database into eleven different data files that signified the eleven unique pre-structured forms. Within these data files, each case represented one completed form. In subsequent steps, single cases were aggregated per e-portfolio, to determine the number of completed forms per e-portfolio. Different descriptive statistics were calculated using IBM SPSS Statistics 25.

An exception was made concerning the form: ‘learning goals and plans’. Since trainees differed to a great extent in the number of learning goals that they formulated on one form (ranging from one to twenty-four), it was considered inconsequential to compare numbers of forms. So, number of learning goals were compared instead. A goal was included if any text was provided within the first of five text boxes (‘description of the learning goal’), relevance of content was not considered at this point.

Phase two: content analysis

A quantitative content analysis of e-portfolios was performed, such an analysis consists of ratings made by the use of a deductively designed coding scheme which is elaborated in a codebook [36,37,38]. The aim of our analysis was to identify the presence of effective SRL behaviour within the documentation kept in the e-portfolios, thereby focusing on the SRL processes that, according to health professions education literature, can be supported by portfolio use: self-assessment, goal setting and planning, and monitoring [9,10,11]. As there was no existing instrument available that was suitable to rate the presence of these SRL processes within written documentation, we used research literature regarding the different processes to develop our codebook (see Appendix A).

For the development of the codebook, we searched the research literature for descriptions of good practice of the included SRL processes, so we could formulate criteria that can be rated for their presence. We decided to focus on two processes that inform self-assessment - reflection and feedback - instead of self-assessment itself, as this process, which is for the most part cognitively performed, is difficult to objectify [39].

For the formulation of criteria concerning reflection we used the framework of Hatton and Smith that distinguishes different types of writing with an increasing level of reflection [40, 41]. With regard to feedback we used the conditions for effective feedback formulated by Gibbs and Simpson [42]. Literature by Zimmerman was used to formulate the criteria on effective goal setting and planning [43].

The operationalisation of monitoring [12], was challenging due to limited information on good monitoring practice [44, 45]. Therefore, it was decided to complement a single rating item (‘Does the e-portfolio show signs of monitoring?’) with memos of the observed monitoring activities. During the rating of monitoring we focused on the coherence between different elements of the portfolio: to what extent did themes and topics considered during feedback and reflections recur in learning goals and plans, and vice versa.

After a first version of the codebook was approved by all authors, two raters (RG & CB) proof tested the codebook on a sample of e-portfolios, leading to adaptations of the codebook. Subsequently, two sittings were performed during which the raters used the codebook to rate the same randomly drawn e-portfolios, after which adaptations were made. During this process, interrater reliabilities (IRR) were calculated multiple times using Krippendorff’s alpha [46, 47] and percentage of agreement. IRR improved with practice and proved to be acceptable during the definitive rating (see Appendix B). Fig. 1 shows the process that led to the final version of the codebook and the definitive rating of e-portfolios.

Fig. 1
figure 1

The process that led to the definitive rating of e-portfolios

During the ratings, the content analysed was confined to material covering one study year, to make sure that the rating period was equal for all e-portfolios included in the content analysis, regardless of the cohort the e-portfolio belonged to. To secure that we rated material that was collected during an entire study year, we chose to include the content from the year prior to the cohort the portfolio belonged to, e.g. content from the first year was included for e-portfolios that belonged to the second year cohort, content from the second year was included for e-portfolios that belonged to the third year cohort, etc.

After the rating was completed the ratings were transferred to an SPSS data file. Frequencies were calculated to gain insight in the number of e-portfolios that met the different SRL criteria for good practice from the codebook.

Ethical approval and reflexivity statement

The ethical review board of the Dutch Organization of Medical Education (NVMO) approved the study under NERB number 786.

The research team was comprised of a psychologist (RG), physicians (BT, NS, AK, RL) and a health scientist and educationalist (SH). Before starting her psychology study, the first researcher studied mass communication, the field from which content analysis originates. The research assistant (CB), who supported the content analysis, has a background in psychology.

Results

Phase one: number of forms

Table 2 provides the descriptive statistics of the number of pre-structured forms used within the e-portfolios. Statistics are split up into the three cohorts, based upon the duration of e-portfolio use. The table shows high standard deviations, representing a considerable difference in numbers of forms present in individual e-portfolios.

Table 2 Descriptive statistics of the numbers of pre-structured forms used in the e-portfolios. For each form it is indicated if the form is envisioned to support F(eedback), R(eflection), G(oal-setting) and/or Mo(nitoring). NB: The data comprise the cumulative numbers of forms for the entire period the e-portfolio was in use. Consequently, the statistics of the first cohort (0–12 months) contain data covering up to a maximum of twelve months, whereas the third cohort (> 24 months) covers at least two years of forms

The percentage of e-portfolios that contained a particular form is also presented for each of the three cohorts. These figures show substantial percentages of non-use, even in the second- and third cohorts, indicating that there are GP trainees in their second- or third-year that miss a number of (mandatory) forms in their e-portfolio.

The table also shows if the pre-structured forms are mandatory or optional and if the form was, during design of the e-portfolio, envisioned to support the different SRL processes. No patterns in percentage of use or number of forms could be detected on basis of these characteristics.

Phase two: content analysis

The results of the content analysis are displayed in Table 3. It shows that forms aimed at reflection were present in just over half of the e-portfolios (54.4%). In case an entry was present it could often (83.7%) not be qualified as reflective, because only experiences and events were described. Entries that were reflective did not exceed descriptive reflection (a singular explanation or justification was given for an event). In general, trainees described experiences with a positive focus, and did not discuss reasons, motives and context. For example: “Things are going very well; well on track, tight schedule; meeting commitments” and “I could think more about the patient’s context, although sometimes I’m managing well enough already. I also know the families and their backgrounds and, if necessary, take this into consideration during the consult.” There were no examples of dialogic or critical reflection in our sample: trainees did not document a variety of possible explanations for events, whereby also considering contextual influences.

Table 3 The percentage of e-portfolios that met the different SRL criteria for good practice from the codebook. The numbers in front of the criteria correspond to the item numbers of the codebook (Appendix A). The codebook describes the instructions used to decide if the criteria were met

Feedback was in many cases confined to a summary of what went well or needed improvement, often lacking specificity, context and direction for future action. For example: “Nice constructive attitude, friendly. Responds well to feedback.” and “Engages actively in the learning process, open for feedback. You are struggling with doubts about the discipline. Sometimes it is hard to keep one’s distance to the patient and not to take your work home.” Feedback was specific in one quarter to one third of the e-portfolios (27.8 and 35.6% for teachers and supervisors, respectively). In about 90% of the e-portfolios, feedback was focused on performance and learning under the trainees’ control. Feedback was aligned with the purpose of the form in about two third of the e-portfolios (58.9% for teachers and 73.3% for supervisors). In these cases, the feedback concerned topics relevant for the competence at hand. A foundation or reasoning for the provided feedback was present in over half of the e-portfolios, such a ‘source’ for feedback was formulated in 57.8% of the e-portfolios. Tips for improvement were present in 13.3% (teachers) and 33.3% (supervisors) of the e-portfolios. Future-directed comments that were given generally missed specificity, for example: “Now just demonstrate it” and “Keep taking (more) care of yourself here”.

Learning goals and plans were specific in less than half (44,0%) of the e-portfolios, mainly because motives for the learning goals (‘why’) and a time or place indication (‘when’ and ‘where’) were missing. Trainees were able to formulate learning goals that are compatible with each other (87.8% of the e-portfolios) and they chose challenging goals to work with (97.8%). Almost two thirds of the e-portfolios contained learning goals with a personal origin. Notably, a number of learning goals appeared in numerous e-portfolios, for example, the placement of an intrauterine device and dealing with polypharmacy.

Signs of monitoring were present in three-quarters of the e-portfolios (74.4%). However, in one-third of these cases (24 of 67) the monitoring behaviour was limited, i.e. monitoring was mostly restricted to a feedback request, via a Mini-CEX, that related to one of the learning goals. In more elaborate cases of monitoring, trainees also noted the progress on learning goals, or adapted their original plans for goal attainment. In general, the themes and topics discussed in the feedback of teachers and reflections of trainees were unrelated to other components of the e-portfolio, such as the learning goals of trainees.

Discussion

Portfolios are implemented to foster SRL of medical students and trainees, as SRL is considered an essential skill for those studying and/or working in the clinical setting [13, 14, 48]. It is expected that documenting reflections, learning goals and plans in a portfolio, stimulates the occurrence and depth of these SRL processes [2, 49, 50]. In addition, it is assumed that portfolio documentation, in which these reflections and learning goals are stored together with feedback from third parties, can help learners to self-assess their performance [51, 52] and with that offers the information needed to monitor one’s own learning process [12]. Common educational practice, such as the assessment of reflections and learning goals and the use of portfolio content as starting point for supervision (meetings), are based on these assumptions.

Our study shows, however, that the documentation of these SRL processes – reflection, feedback, goal-setting and planning, and monitoring – is limited within the e-portfolios of GP trainees. Previous studies have explained that (mandatory) written reflections have certain predicaments, that can potentially reduce reflection into a ‘tick-box-exercise’ which shows through documentation of shallow reflections [53,54,55,56]. An example of such a predicament is the apprehension that can be felt towards written reflections, i.e. the documentation of vulnerabilities, as such documentation might be considered harmful for assessment, professional development or legal issues [50, 56, 57]. Furthermore, our results concerning feedback are not uncommon, as the review of Bing-You et al. shows that more than a quarter of the 51 included articles on feedback exchange reported problems with low quality feedback, due to limited information, lack of specificity and absence of action plans [58]. With regard to goal-setting and planning previous studies have shown that learners hardly ever integrate goals into their workplace-based learning, unless they receive tailored coaching focused on effective goal-setting [59,60,61,62]. Although, the importance of monitoring of learning is often mentioned, there is still limited evidence of monitoring behaviour within medical education [45].

Considering the near absence of the included SRL processes in the e-portfolio content, could result in the simple explanation that trainees do not engage in SRL. However, a previous study using interviews, also targeting Dutch GP trainees, indicated that trainees did purposively regulate their learning [63]. The occurrence of SRL related cognitions, motivations and behaviour of trainees that probably took place - whether or not instigated by the e-portfolio - is apparently not captured in e-portfolio forms or content.

This could be explained by a variety of factors of some have been described above. In addition, trainees might not find the right words to accurately describe the SRL processes taking place. As SRL processes are complex, and might (partly) take place on a subconscious level [64], formulating if and how they were present asks for a high level of introspection and writing skills. Difficulties to adequately document the presence of SRL processes in a portfolio might be fuelled by another factor that is part of workplace-based medical education: a need to prioritise. When working in the clinical setting, where time pressure and a certain level of unpredictability are inevitable [14, 16, 65], trainees might not experience the time required to appropriately document the learning processes taking place. However, SRL processes may still occur in the head of trainees or be a topic of discussion with supervisors and teachers.

Another factor explaining the limited documentation of the different SRL processes might concern the difficulties that can occur when combining multiple purposes – assessment, accountability and support for SRL – in one portfolio [66]. The way the current GP e-portfolio was implemented might also have contributed. During implementation the focus was on assuring an appropriate functioning of the e-portfolio, especially regarding assessment, which needed to be covered sufficiently for accountability reasons. It was expected that the potential of the e-portfolio for the support and documentation of SRL would surface naturally when the e-portfolio was adequately used. Consequently, trainees received instructions concerning the use of the e-portfolio and its programmatic assessment function, but no specific instructions regarding the documentation of SRL.

With the choice to only provide information on the technical and assessment aspects of the e-portfolio we potentially overestimated the knowledge and skills that trainees and their teachers/supervisors have concerning (the documentation of) SRL. Research shows that support of a teacher/supervisor is important for the development of SRL [67], in addition, teachers/supervisors often do not know how to support SRL effectively [68, 69]. Thus, trainees, teachers and supervisors not only need to be supported in the technical aspects of a portfolio, but also need training concerning (documentation of) SRL, e.g. on effective reflection, feedback and goal-setting.

Strengths and limitations

This is the first study to conduct content analysis on e-portfolios of medical trainees in order to achieve insights in the documentation of multiple SRL processes. For this purpose, a codebook, based on research literature, was constructed to rate the documentation of different criteria for good SRL practice in e-portfolio content.

To ensure anonymity of trainees, we chose to only include the pre-structured forms, which make up a considerable part of the e-portfolio. Consequently, it cannot be excluded that material stored in the folder for personal documents does show different patterns concerning SRL than those found in the pre-structured forms.

This study focused on self-assessment, via feedback and reflection, goal setting and planning, and monitoring, as these SRL processes are commonly mentioned in the health professions education literature in concern to portfolio use. However, SRL is a comprehensive concept that covers more than the processes that were considered within this research.

Implications for future research

We focused on the documentation of SRL processes within e-portfolios, by using e-portfolio content as data source for quantitative analysis. Qualitative research would be valuable to assess how different factors, such as level of introspection and writing skills, influence the (limited) documentation of SRL processes. Likewise, other questions that still exist concerning the use of portfolios for the support of SRL require combined use of quantitative and qualitative measures. Future studies should, thus, use different data sources – e.g. portfolio content, portfolio users - and methods – e.g. questionnaires, observations, interviews - to attain further understanding of portfolio use for the support of SRL via triangulation. Thereby, clarifying to what extent and how portfolio use can support SRL.

Conclusion

This study adds insights into the intricate relationship between portfolio use and SRL. Content analysis of e-portfolios used by GP trainees showed limited documentation of the SRL processes that are, according to health professions education literature, influenced by portfolio use: self-assessment, via reflection and feedback, goal-setting and planning, and monitoring. As the documentation of SRL in a portfolio asks for high level of introspection and writing skills, and requires time that is not always available within the clinical setting, one cannot expect documentation of SRL processes to appear in e-portfolio content without efforts.

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

SRL:

Self-Regulated Learning

Electronic portfolio:

e-portfolio

GP:

General Practitioner

References

  1. Tochel C, Haig A, Hesketh A, Cadzow A, Beggs K, Colthart I, et al. The effectiveness of portfolios for post-graduate assessment and education: BEME guide no 12. Med Teach. 2009;31(4):299–318.

    Google Scholar 

  2. Beckers J, Dolmans D, Van Merriënboer J. E-portfolios enhancing students' self-directed learning: a systematic review of influencing factors. Australas J Educ Technol. 2016;32(2):32–46.

  3. Abrami PC, Wade A, Pillay V, Aslan O, Bures EM, Bentley C. Encouraging self-regulated learning through electronic portfolios. E-learn: world conference on E-learning in corporate, government, healthcare, and higher education. In: Association for the Advancement of computing in education (AACE); 2007.

    Google Scholar 

  4. Alexiou A, Paraskeva F. Enhancing self-regulated learning skills through the implementation of an e-portfolio tool. Procedia Soc Behav Sci. 2010;2(2):3048–54.

    Google Scholar 

  5. Lam R. Promoting self-regulated learning through portfolio assessment: testimony and recommendations. Assess Eval High Educ. 2014;39(6):699–714.

    Google Scholar 

  6. Zimmerman BJ. Investigating self-regulation and motivation: historical background, methodological developments, and future prospects. Am Educ Res J. 2008;45(1):166–83.

    Google Scholar 

  7. Sitzmann T, Ely K. A meta-analysis of self-regulated learning in work-related training and educational attainment: what we know and where we need to go. Psychol Bull. 2011;137(3):421.

    Google Scholar 

  8. Snadden D, Thomas M. The use of portfolio learning in medical education. Med Teach. 1998;20(3):192–9.

    Google Scholar 

  9. Nothnagle M, Goldman R, Quirk M, Reis S. Promoting self-directed learning skills in residency: a case study in program development. Acad Med. 2010;85(12):1874–9.

    Google Scholar 

  10. Garrett BM, Jackson C. A mobile clinical e-portfolio for nursing and medical students, using wireless personal digital assistants (PDAs). Nurse Educ Today. 2006;26(8):647–54.

    Google Scholar 

  11. Foucault M-L, Vachon B, Thomas A, Rochette A, Giguère C-É. Utilisation of an electronic portfolio to engage rehabilitation professionals in continuing professional development: results of a provincial survey. Disabil Rehabil. 2018;40(13):1591–9.

    Google Scholar 

  12. Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE guide no. 45. Med Teach. 2009;31(9):790–801.

    Google Scholar 

  13. Berkhout JJ, Helmich E, Teunissen PW, van der Vleuten CP, Jaarsma ADC. Context matters when striving to promote active and lifelong learning in medical education. Med Educ. 2018;52(1):34–44.

    Google Scholar 

  14. van Houten-Schat MA, Berkhout JJ, van Dijk N, Endedijk MD, Jaarsma ADC, Diemers AD. Self-regulated learning in the clinical context: a systematic review. Med Educ. 2018;52(10):1008–15.

    Google Scholar 

  15. Siddaiah-Subramanya M, Nyandowe M, Zubair O. Self-regulated learning: why is it important compared to traditional learning in medical education? Adv Med Educ Pract. 2017;8:243.

    Google Scholar 

  16. Irby DM. Teaching and learning in ambulatory care settings: a thematic review of the literature. Acad Med. 1995;70(10):898–931.

    Google Scholar 

  17. Hrisos S, Illing JC, Burford BC. Portfolio learning for foundation doctors: early feedback on its use in the clinical workplace. Med Educ. 2008;42(2):214–23.

    Google Scholar 

  18. Vance G, Williamson A, Frearson R, O’Connor N, Davison J, Steele C, et al. Evaluation of an established learning portfolio. Clin Teach. 2013;10(1):21–6.

    Google Scholar 

  19. Webb TP, Merkley TR. An evaluation of the success of a surgical resident learning portfolio. J Surg Educ. 2012;69(1):1–7.

    Google Scholar 

  20. Pearson DJ, Heywood P. Portfolio use in general practice vocational training: a survey of GP registrars. Med Educ. 2004;38(1):87–95.

    Google Scholar 

  21. Jenkins L, Mash B, Derese A. The national portfolio of learning for postgraduate family medicine training in South Africa: experiences of registrars and supervisors in clinical practice. BMC Med Educ. 2013;13(1):149.

    Google Scholar 

  22. Vance GH, Burford B, Shapiro E, Price R. Longitudinal evaluation of a pilot e-portfolio-based supervision programme for final year medical students: views of students, supervisors and new graduates. BMC Med Educ. 2017;17(1):141.

    Google Scholar 

  23. De Swardt M, Jenkins LS, Von Pressentin KB, Mash R. Implementing and evaluating an e-portfolio for postgraduate family medicine training in the Western Cape. South Africa. BMC Med Educ. 2019;19(1):251.

    Google Scholar 

  24. Tailor A, Dubrey S, Das S. Opinions of the ePortfolio and workplace-based assessments: a survey of core medical trainees and their supervisors. Clin Med (Northfield Il). 2014;14(5):510–6.

    Google Scholar 

  25. Driessen E. Portfolio critics: do they have a point? Med Teach. 2009;31(4):279–81.

    Google Scholar 

  26. Devolder A, van Braak J, Tondeur J. Supporting self-regulated learning in computer-based learning environments: systematic review of effects of scaffolding in the domain of science education. J Comput Assist Learn. 2012;28(6):557–73.

    Google Scholar 

  27. Bol L, Garner JK. Challenges in supporting self-regulation in distance education environments. J Comput High Educ. 2011;23(2–3):104–23.

    Google Scholar 

  28. Pelgrim EA, Kramer AW, Mokkink HG, Van der Vleuten CP. Quality of written narrative feedback and reflection in a modified mini-clinical evaluation exercise: an observational study. BMC Med Educ. 2012;12(1):97.

    Google Scholar 

  29. Djajadi R, Claramita M, Rahayu G. Quantity and quality of written feedback, action plans, and student reflections before and after the introduction of a modified mini-CEX assessment form. Afr J Health Prof Educ. 2017;9(3):148–52.

    Google Scholar 

  30. Shaughness G, Georgoff PE, Sandhu G, Leininger L, Nikolian VC, Reddy R, et al. Assessment of clinical feedback given to medical students via an electronic feedback system. J Surg Res. 2017;218:174–9.

    Google Scholar 

  31. Saggasser MH, Schreurs ML, AWM K, Maiburg B, Mokking H. Richtlijn portfolio voor de huisartsopleidingUMC st Radboud, Maastricht University; 2010.

    Google Scholar 

  32. van der Vleuten CP, Schuwirth L, Driessen E, Dijkstra J, Tigelaar D, Baartman L, et al. A model for programmatic assessment fit for purpose. Med Teach. 2012;34(3):205–14.

    Google Scholar 

  33. Tromp F, Vernooij-Dassen M, Grol R, Kramer A, Bottema B. Assessment of CanMEDS roles in postgraduate training: the validation of the compass. Patient Educ Couns. 2012;89(1):199–204.

    Google Scholar 

  34. Van Nuland M, Van Den Noortgate W, Degryse J, Goedhuys J. Comparison of two instruments for assessing communication skills in a general practice objective structured clinical examination. Med Educ. 2007;41(7):676–83.

    Google Scholar 

  35. Al Ansari A, Ali SK, Donnon T. The construct and criterion validity of the mini-CEX: a meta-analysis of the published research. Acad Med. 2013;88(3):413–20.

    Google Scholar 

  36. Boettger RK, Palmer LA. Quantitative content analysis: its use in technical communication. IEEE Trans Prof Commun. 2010;53(4):346–57.

    Google Scholar 

  37. Kondracki NL, Wellman NS, Amundson DR. Content analysis: review of methods and their applications in nutrition education. J Nutr Educ Behav. 2002;34(4):224–30.

    Google Scholar 

  38. White MD, Marsh EE. Content analysis: a flexible methodology. Libr Trends. 2006;55(1):22–45.

    Google Scholar 

  39. Ward M, Gruppen L, Regehr G. Measuring self-assessment: current state of the art. Adv Health Sci Educ Theory Pract. 2002;7(1):63–80.

    Google Scholar 

  40. Pee B, Woodman T, Fry H, Davenport ES. Appraising and assessing reflection in students' writing on a structured worksheet. Med Educ. 2002;36(6):575–85.

    Google Scholar 

  41. Hatton N, Smith D. Reflection in teacher education: towards definition and implementation. Teach Teach Educ. 1995;11(1):33–49.

    Google Scholar 

  42. Gibbs G, Simpson C. Conditions under which assessment supports students’ learning. Learning Teaching Higher Educ. 2005;1:3–31.

    Google Scholar 

  43. Zimmerman BJ. Goal setting: a key proactive source of academic self-regulation. In: Motivation and self regulated learning: Theory, research, and applications; 2008. p. 267.

    Google Scholar 

  44. Epstein RM, Siegel DJ, Silberman J. Self-monitoring in clinical practice: a challenge for medical educators. J Contin Educ Health Prof. 2008;28(1):5–13.

    Google Scholar 

  45. De Bruin AB, Dunlosky J, Cavalcanti RB. Monitoring and regulation of learning in medical education: the need for predictive cues. Med Educ. 2017;51(6):575–84.

    Google Scholar 

  46. Feng GC. Mistakes and how to avoid mistakes in using intercoder reliability indices. Methodol Eur J Res Methods Behav Soc Sci. 2015;11(1):13.

    Google Scholar 

  47. Hayes AF, Krippendorff K. Answering the call for a standard reliability measure for coding data. Commun Methods Meas. 2007;1(1):77–89.

    Google Scholar 

  48. Murdoch-Eaton D, Whittle S. Generic skills in medical education: developing the tools for successful lifelong learning. Med Educ. 2012;46(1):120–8.

    Google Scholar 

  49. Wald HS, Borkan JM, Taylor JS, Anthony D, Reis SP. Fostering and evaluating reflective capacity in medical education: developing the REFLECT rubric for assessing reflective writing. Acad Med. 2012;87(1):41–50.

    Google Scholar 

  50. Sandars J. The use of reflection in medical education: AMEE guide no. 44. Med Teach. 2009;31(8):685–95.

    Google Scholar 

  51. Sargeant J, Eva KW, Armson H, Chesluk B, Dornan T, Holmboe E, et al. Features of assessment learners use to make informed self-assessments of clinical performance. Med Educ. 2011;45(6):636–47.

    Google Scholar 

  52. Driessen E, van Tartwijk J, van der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ. 2007;41(12):1224–33.

    Google Scholar 

  53. Curtis P, Gorolay S, Curtis A, Harris M. What do general practitioners think of written reflection? A focus group study. Educ Prim Care. 2016;27(4):292–8.

    Google Scholar 

  54. Driessen E. Do portfolios have a future? Adv Health Sci Educ Theory Pract. 2017;22(1):221–8.

    Google Scholar 

  55. de la Croix A, Veen M. The reflective zombie: problematizing the conceptual framework of reflection in medical education. In: Perspectives on Medical Education; 2018. p. 1–7.

    Google Scholar 

  56. O’Connell TS, Dyment JE. The case of reflective journals: is the jury still out? Reflective Pract. 2011;12(1):47–59.

    Google Scholar 

  57. Vivekananda-Schmidt P, Marshall M, Stark P, Mckendree J, Sandars J, Smithson S. Lessons from medical students’ perceptions of learning reflective skills: a multi-institutional study. Med Teach. 2011;33(10):846–50.

    Google Scholar 

  58. Bing-You R, Varaklis K, Hayes V, Trowbridge R, Kemp H, McKelvy D. The feedback tango: an integrative review and analysis of the content of the teacher–learner feedback exchange. Acad Med. 2018;93(4):657–63.

    Google Scholar 

  59. Lockspeiser TM, Li S-TT, Burke AE, Rosenberg AA, Dunbar AE, Gifford KA, et al. In pursuit of meaningful use of learning goals in residency: a qualitative study of pediatric residents. Acad Med. 2016;91(6):839–46.

    Google Scholar 

  60. Nothnagle M, Anandarajah G, Goldman RE, Reis S. Struggling to be self-directed: residents' paradoxical beliefs about learning. Acad Med. 2011;86(12):1539–44.

    Google Scholar 

  61. George P, Reis S, Dobson M, Nothnagle M. Using a learning coach to develop family medicine residents' goal-setting and reflection skills. J Grad Med Educ. 2013;5(2):289–93.

    Google Scholar 

  62. Li S-TT, Burke AE. Individualized learning plans: basics and beyond. Acad Pediatr. 2010;10(5):289–92.

    Google Scholar 

  63. Sagasser MH, Kramer AW, van der Vleuten CP. How do postgraduate GP trainees regulate their learning and what helps and hinders them? A qualitative study. BMC Med Educ. 2012;12(1):67.

    Google Scholar 

  64. Fitzsimons GM, Bargh JA. Automatic self-regulation; 2004.

    Google Scholar 

  65. Lewis CE, Tillou A, Yeh MW, Quach C, Hiatt JR, Hines OJ. Web-based portfolios: a valuable tool for surgical education. J Surg Res. 2010;161(1):40–6.

    Google Scholar 

  66. Heeneman S, Driessen EW. The use of a portfolio in postgraduate medical education–reflect, assess and account, one for each or all in one? GMS J Med Educ. 2017;34(5):1–12.

  67. Cazan A-M. Teaching self regulated learning strategies for psychology students. Procedia Soc Behav Sci. 2013;78:743–7.

    Google Scholar 

  68. Perry NE, Hutchinson L, Thauberger C. Talking about teaching self-regulated learning: scaffolding student teachers’ development and use of practices that promote self-regulated learning. Int J Educ Res. 2008;47(2):97–108.

    Google Scholar 

  69. English MC, Kitsantas A. Supporting student self-regulated learning in problem-and project-based learning. Interdiscip J Problem-Based Learning. 2013;7(2):6.

    Google Scholar 

Download references

Acknowledgements

We would like to show our gratitude to Parantion for their efforts in designing the mirror e-portfolio database. We would also like to acknowledge Carolien Boogaard, for her support during the e-portfolio ratings, and Carlo Hagemann, for his advice on content analysis. Lastly, a thanks to the other members of the Opt-e-MisE group – Greetje Sagasser, Angelique Timmerman and Jean Muris - for their input.

Funding

The research was for the main part internally funded, and was additionally funded by unrestricted grants from Parantion (https://parantion.com/) and SBOH (https://www.sboh.nl/).

Author information

Authors and Affiliations

Authors

Contributions

The article is the result of a collaboration between the six authors. BT initiated the research by bringing together the research team and realising the mirror e-portfolio database in collaboration with the supplier of the e-portfolio (Parantion). RG and BT did a first proposition for the design of the study, which was finalised through discussion with the other authors. RG executed the research and was the primary writer of the article. Guidance was given by the other authors, via monthly group meetings, feedback on drafts of the article and one-on-one consultations. All authors agreed on a final version of the article.

Corresponding author

Correspondence to R. van der Gulden.

Ethics declarations

Ethics approval and consent to participate

The ethical review board of the Dutch Organization of Medical Education (NVMO) approved the study under NERB number 786.

Consent for publication

Not applicable.

Competing interests

Sylvia Heeneman is an Associate Editor for BMC Medical Education.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Additional file 1.

Appendix A. How is SRL documented in e-portfolio content. Codebook. In this appendix the codebook used for the content analysis is displayed.

Additional file 2.

Appendix B. How is SRL documented in e-portfolio content. Interrater reliabilities. This table shows the interrater reliabilities that were calculated during the design of the codebook.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

van der Gulden, R., Heeneman, S., Kramer, A.W.M. et al. How is self-regulated learning documented in e-portfolios of trainees? A content analysis. BMC Med Educ 20, 205 (2020). https://doi.org/10.1186/s12909-020-02114-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12909-020-02114-4

Keywords