- Research article
- Open Access
Managing residents in difficulty within CBME residency educational systems: a scoping review
BMC Medical Education volume 20, Article number: 235 (2020)
Best practices in managing residents in difficulty (RID) in the era of competency-based medical education (CBME) are not well described. This scoping review aimed to inventory the current literature and identify major themes in the articles that address or employ CBME as part of the identification and remediation of residents in difficulty.
Articles published between 2011 to 2017 were included if they were about postgraduate medical education, RID, and offered information to inform the structure and/or processes of CBME. All three reviewers performed a primary screening, followed by a secondary screening of abstracts of the chosen articles, and then a final comprehensive sub-analysis of the 11 articles identified as using a CBME framework.
Of 165 articles initially identified, 92 qualified for secondary screening; the 63 remaining articles underwent full-text abstracting. Ten themes were identified from the content analysis with “identification of RID” (41%) and “defining and classifying deficiencies” (30%) being the most frequent. In the CBME article sub-analysis, the most frequent themes were: need to identify RID (64%), improving assessment tools (45%), and roles and responsibilities of players involved in remediation (27%). Almost half of the CBME articles were published in 2016–2017.
Although CBME programs have been implemented for many years, articles have only recently begun specifically addressing RID within a competency framework. Much work is needed to describe the sequenced progression, tailored learning experiences, and competency-focused instruction. Finally, future research should focus on the outcomes of remediation in CBME programs.
Best practices for managing residents in difficulty are neither well defined nor consistently applied. The term “residents in difficulty” generally refers to graduate medical trainees who have demonstrated a significant or sustained pattern of underperformance compared to expectations. As postgraduate medical education training programs transition to competency-based medical education (CBME), the need for definitions and approaches may increase.
In a recent article about CBME, Van Melle  et al. identified five key features of CBME educational designs: 1) an outcomes-based competency framework; 2) using a sequenced progression of competence; 3) including tailored learning experiences for the achievement of competencies; 4) tailoring teaching to competencies that, for residents, include multiple workplace observations followed by feedback and coaching; and 5) taking a purposeful and programmatic approach to assessment. We will explore whether the literature about residents in difficulty considers these features of CBME.
Several articles attempted to quantify the prevalence, characteristics, areas of weakness, and outcomes of residents on remediation or probation. Zbieranowski  and Guerrasio  each looked at a 10-year period at one university, ending in 2009 and 2012 respectively. In both articles, competency-based models were not in effect during the eras studied. Dupras  reported on a survey across Internal Medicine (IM) programs that sought to define prevalence, problems, and remediation in the era of core competencies, which preceded the roll-out of CBME residency training programs.
Very little literature exists on residents in difficulty within the context of CBME programs. Past reviews have looked at prevalence, which competencies residents underperform in, and what interventions are used.
This scoping review takes stock of what is known about residents in difficulty within the CBME context and connects it to the available literature on best practices for residents in difficulty. In North America, where the vast majority of the identified studies were published, CBME was first launched in the United States by the Accreditation Council for Graduate Medical Education (ACGME) in the 6 Core Competencies in 2001 and more specifically developed through the Milestones Project since 2013 . In Canada, the College of Family Physicians of Canada (CFPC) launched their Triple C curriculum in 2011 ; and the Royal College of Physicians and Surgeons (RCPSC) adopted the CanMEDS framework in 1996, updating it in 2005 and 2015 , before developing a more structured CBME curriculum, Competence by Design, implemented gradually beginning in 2017 . Specifically, we considered papers about residents in difficulty in a pre- and post-CBME context and considered what has been written on such topics as identifying a resident in difficulty, implementing remedial education plans, monitoring resident progress during the remedial period, and determining successful resident outcomes following the remedial period.
The purpose of this paper is to provide a review of the current literature on residents in difficulty, spanning the era of the introduction of CBME to post-graduate training programs. The specific objectives of this scoping review were to
inventory current literature
identify major themes
report on the frequency and types of articles that address CBME as part of the identification and remediation of residents in difficulty.
A scoping review methodology, as outlined by Arksey and O’Malley  and Levac et al.  (2010), was used to provide a comprehensive overview of the current literature on residents in difficulty, spanning the era of the introduction of CBME to post-graduate training programs.
The main stages in our methodology consisted of 1) identifying the research question; 2) developing a search strategy for capturing literature relevant to the research question; 3) screening the literature for inclusion; 4) analyzing the results using a grounded theory approach; and 5) collating, summarizing, and reporting of the results.
We employed multiple quality assurance processes, including calibration of reviewers during the screening, abstracting, and coding stages. The three reviewers performed calibration exercises to achieve clarification and consensus around the screening and abstracting criteria and outcomes (e.g. What constitutes as inclusion? What is meant by the abstracting criteria? What information is being captured / excluded?). For example, during the article title and abstracting stage, the three reviewers were assigned six articles each to screen for inclusion (‘yes’, ‘no’ or ‘unsure’), with each article being reviewed by two separate reviewers. Once screened, the reviewers met to discuss the results, discrepancies and questions that arose. This process was repeated until consensus was reached and the reviewers were confident to begin screening articles independently. The same process was applied for article abstracting. Throughout the screening and abstracting processes, reviewers were encouraged to flag articles as “unsure” or “in need of discussion” for discussion as a group. During qualitative analysis, a similar calibration exercise was used to establish the article coding framework for identifying themes across articles and findings of interest / impact.
The key research question was: What does the current literature say about the remediationFootnote 1of graduate medical trainees using competency-based educational approaches?
We first established our article inclusion criteria to narrow down the body of literature to articles solely on our topic of interest. The three criteria are listed and described in Table 1.
We searched the electronic databases of ERIC, MEDLINE, and EMBASE, with the aid of the Department of Family and Community Medicine (DFCM) librarian, using a search framework from a previous literature review conducted by two members of the research team (SGT, LSA) (See Supplement A for sample search strategy). Sample keywords used to search various article fields including titles, abstracts, subject heading and author keywords included: “physician,” “trainee,” “intern,” “residency”, “house officer”, “remediation,” “academic difficulty” and “residents in difficulty” (along with all relevant wildcards, plural forms, and other relevant terms). The subject headings of “education, medical, graduate/” and “remedial teaching” were also used. By combining search terms, articles were only included if they had discussed both residency education and academic difficulty / remediation, accounting for a narrower scope (See Supplement B for Sample Search Strategy).
The last major literature review on remediation, conducted by Cleland et al. , looked at articles from 1984 to 2012, inclusive. Thus, we chose to focus our scoping review on the most recent body of literature, published from 2011 to 2017, inclusive. Another reason for that focus was that residency education has only very recently moved purposefully toward implementing CBME systems.
Of the 165 articles identified from the literature search, 131 were included in the primary screen (34 were duplicates and therefore excluded). Three reviewers (LSA, MR,Footnote 2 and SGT) were calibrated on the primary screening of 10 of the 131 article titles and abstracts to ensure screening consistency and to refine the screening criteria as necessary. The remaining article titles and abstracts were divided among the members of the research team and screened independently.
Secondary screen and data extracting
The team brainstormed on the type of data to collect that would be relevant to our topic, and to the aims of our local Best Practices in Evaluation and Assessment (BPEA) Working GroupFootnote 3 at large. We constructed an online form to collect data of interest from included full-text articles. (See Supplement C for complete list of data extraction criteria, their descriptions, and examples from the reviewed literature).
A second calibration exercise was performed with three reviewers (JP, LSA, and MR) on the screening and abstracting of five of the 92 full-text articles included from the primary screen. This helped us to ensure consistency among reviewers in the screening of full-text articles, refine the data abstracting tool, and form a common understanding of the extraction criteria. We improved and expanded upon the data extraction criteria as a result.
The remaining full-text articles were divided among the team (JP, LSA, and MR) to be screened and abstracted independently.
Descriptive and qualitative summary of results
Data collected from eligible articles were summarized descriptively (frequencies, crosstabulations), using IBM SPSS Statistics Software v24. Qualitative data were collected about each article’s purpose or objective(s); the main findings related to the structures to support competence that were discussed; and any other findings related to residents in difficulty, remediation, and/or BOE. All qualitative data were analyzed with NVivo v11 using summative content analysis methodology, as described by Hsieh and Shannon , to identify major themes.
A secondary, more comprehensive analysis was done of the 11 articles that discussed or used a CBME orientation or framework.
Risk of Bias
In such research, there is the potential for reviewer bias during the primary and secondary screens, as well as the qualitative analysis. To mitigate reviewer bias, calibration exercises were performed at each stage until all reviewers achieved consensus; and the themes and abstracting criteria were discussed, iteratively reviewed and refined throughout. There is also inherent bias due to the criteria for inclusion selected by the authors, which might have excluded other relevant studies from this review. Lastly, there is also the risk of bias in the gathered articles due to the interests of publishers and researchers, alike, thereby possibly limiting completed understanding and/or coverage of these topics.
Figure 1 shows the literature review screening process and results. Of the 165 articles identified from our literature search, 92 article titles and abstracts passed the primary screen and went on to secondary screening. A total of 63 full-text articles were included from the secondary screen and underwent data extraction and analysis.
Descriptive overview of included articles (n = 63)
This section outlines some key results based on frequency of occurrence. Note that frequency does not infer importance per se, but rather reflects the topics and contexts of greatest interest to researchers and publishers.
Interestingly, the year 2012 saw the most articles published on residents in difficulty, remediation, and/or BOE (n = 16) (see Fig. 2). Most articles reviewed were primary research studies (n = 36, 57%) — e.g. correlation, observation, and prediction studies — with only 5 review articles captured (8%). The majority of articles were published in the United States (n = 45, 71%).
The specialties most often discussed were Emergency Medicine (n = 11, 17%), Surgery (n = 11, 17%), Family Medicine (n = 6, 10%), and Internal Medicine (n = 6, 10%); a quarter of articles (n = 15, 24%) were not associated with any particular specialty.
Medical Expert (n = 33, 52%), Professionalism (n = 24, 38%), and Communicator (n = 11, 17%) were the competencies most often discussed in association with residents in difficulty, remediation, and/or BOE.
“Guidelines for program” was the structure most frequently used to support resident competence (n = 45, 71%). Other commonly discussed structures were: “Design of residency educational program” (n = 21, 33%) and “Design of individual resident educational plan/program” (n = 19, 30%). Many articles that discussed “guidelines for program” were primary research studies (n = 24, 38%). “Promotion of resident systems” was rarely discussed (n = 7, 11%); it was defined as the system(s) in place for promoting residents from one stage of training or PG year to the next (e.g. How are promotion decisions made? What data points inform the decision to promote residents? How is competence assessed?).
Ten themes were identified from the content analysis of qualitative data collected on all 63 reviewed articles (Table 2). “Identification of deficiencies” (n = 33, 52%) and the “importance of defining and classifying resident problems or deficiencies as a first step to remediating them” (n = 19, 30%) were the two most common themes identified. Additionally, the subgroup of papers that were explicit about remediation within a CBE context were analysed to see whether they differed from the 10 themes identified and if the emphasis within the subgroup was consistent or not.
The top two themes arising from our qualitative analysis are explored in greater detail below.
Identification of residents in difficulty
Early assessment and identification of resident problems is considered essential to the success of remediation efforts in terms of resident learning and behavioural outcomes. Early identification has the following impacts: minimizes the use of resources (deficiencies increase in severity the longer they persist in training, thus requiring more time and effort to remediate); reduces negative impacts to patient safety and quality of care (identifying and remediating resident problems earlier means less exposure of underperforming residents to patients); decreases negative effects on the functioning of health professional teams and the system at large (health professional teams are negatively affected by the actions of underperforming residents).
Many types of assessments were discussed that could be used to identify resident deficiencies, such as:
In-training specialty exams, especially if occurring early in the learning experience or residency year 
Innovative in-training assessment tools for providing formative feedback, such as the Clinical Skills Verification program described by Dalack and Jibson [14, 15]
Objective Standardized Clinical Examinations (OSCEs) to uncover knowledge gaps in trainees 
Standardized Direct Observation Tools (SDOT) 
Simulation laboratories 
Simulated oral board cases 
Post-curriculum multiple choice examinations, administered yearly 
TriMetrix, a tool used to measure the behaviours and motivations of residents 
Administering pre- and post-lecture questions to allow early identification of gaps in medical knowledge 
General faculty evaluations 
Besides early targeted assessments, almost a quarter of the articles deemed faculty development important for identifying residents in difficulty (21%, n = 4/19) [21,22,23,24].
Defining and Classifying Deficiencies
Separate from the identification of residents in difficulty is the need to have a literature-informed standardized classification system for categorizing and defining the many different kinds of resident deficiencies.
Defining and classifying resident problems includes rating the level of seriousness of the deficiency  and exploring potential causes  (e.g. mental health issues, cognitive disabilities).
Having standardized definitions and a means of classifying deficiencies is necessary for developing more uniform approaches to remediation that are targeted to improving this issue . Such means also allow Program Directors and faculty assessors to have a mental model or framework for identifying these deficiencies in practice.
Repeatedly, articles discussed the importance of defining and classifying as a necessary first step to understanding the best course of action to take in improving a resident’s performance to meet the standards of the program, whether it is through informal coaching, mentoring and monitoring of progress, more formal remediation, or probation if the issue is non-remediable.
Some of the literature we reviewed offered categorizations and definitions of commonly occurring resident problems, or referenced other articles that did, such as surgical clinical performance , professional behaviour, and competence and collaboration [25, 29,30,31]. Information on these articles can be found in Supplement D.
CBME article sub-analysis (11 articles)
Competency-based medical education was explicitly discussed infrequently. Eleven (17%) articles discussed a competency-based educational orientation or framework. Half of these papers were from the last 2 years (2016–2017), reflecting the increasing global interest in CBME.
The most common themes were similar in the CBME-related articles: “Identification of residents in difficulty” (64%) and “Improving assessment tools for tracking progress of residents undergoing remediation” (45%). The theme “Roles and responsibilities of players involved in remediation” was also cited frequently (27%) in CBME-related articles, often describing the structure and role of clinical competency committees. The frequency of themes did not differ significantly from non-CBME–related articles, although the numbers are small, making comparisons difficult.
Below is a summary of some of the articles representing these themes (Theme #):
Identifying residents in difficulty (1. A. Assessment):
R. Domen et al. Resident Remediation, Probation, and Dismissal. 2014.
Describes a new approach for pathology training programs. They discuss the identification of resident performance problems using close observation by multiple observers, using multiple evaluation tools with 360-degree evaluation, at multiple points during the training period. They describe an eight-step remediation development plan and see the burden of responsibility lying between PDs and CCC .
B. Kinnear et al. Critical Deficiency Ratings in Milestone Assessment: A Review and Case Study. 2017.
Examines the proportion of ACGME milestone sets that include critical deficiencies (CD) ratings and describes one internal medicine residency program’s experiences using CD ratings in assessment. Identification of CDs may help programs develop remediation and improvement plans .
K. Walburton et al. Comprehensive Assessment of Struggling Learners Referred to a Graduate Medical Education Remediation Program. 2017.
Describes the development of and composition of an Early Intervention Remediation Committee (EIRC), a process for identifying struggling learners (RID), the categorizing of primary deficits, the assessment of learners, and the outcomes. Over 2 years, 4% of learners required remediation plans and all were successfully completed. Time commitment averaged 45 h per learner .
Assessment and improving assessment tools and methods for tracking progress (3.)
B. Gas et al. Objective Assessment of General Surgery Residents Followed by Remediation. 2016.
Describes the development of an objective assessment of general surgery residents using a five-station skills assessment event called “X-Games.” Those residents who failed stations (89%) were “remediated” with take-home and deliberate practice, scoring rubrics, and video re-assessment. Of the residents who attempted remediation (58%), all achieved a score greater than or equal to 4/5 .
L. McMurray et al. The Nightmares Course: A Longitudinal, Multidisciplinary, Simulation-Based Curriculum to Train and Assess Resident Competence in Resuscitation. 2017.
Describes the use of a formative simulation-based resuscitation “Nightmares” course: a novel, competency-based, transitional curriculum and assessment program in resuscitation medicine with competency-based thresholds using entrustable decisions. This was followed by a summative OSCE in which 35% of residents did not meet the competency threshold and subsequently participated in remediation sessions .
Roles and responsibilities of players involved in remediation (10.)
E. Ketteler et al. Competency Champions in the Clinical Competency Committee: A Successful Strategy to Implement Milestone Evaluations and Competency Coaching. 2014.
Describes the development of a CCC for surgical residents. The role of the CCC was to identify RID, the players/roles of the members, and the frequency of evaluation, using the ACGME competencies. The selection of “competency champions” is described, along with the process to identify RID who require remediation and more formal performance evaluation .
J. Smith et al. Defining Uniform Processes for Remediation, Probation and Termination in Residency Training. 2017.
An attempt to establish standardized definitions of levels of RID, management processes, documentation expectations, and appropriate notifications. The authors then develop a “remediation schema for residents at risk of not meeting educational milestones during their training,” including the roles of the PD, the designated institutional offices (DIO), and the GME offices .
K. Walburton et al. Comprehensive Assessment of Struggling Learners Referred to a Graduate Medical Education Remediation Program. 2017 (see 1. above) .
Other areas included (11.)
D. Dupras et al. “Problem Residents”: Prevalence, Problems and Remediation in the Era of Core Competencies. 2012.
The purpose here was to assess the experiences of internal medicine PDs, including the incidence and prevalence of resident problems in programs using ACGME competencies. The study concludes that identification of RID is difficult prior to residency, that residents have difficulties in multiple competencies, and have the most trouble with unprofessional behaviour. The authors advocate for a more comprehensive and multisource evaluation system in order to identify RID .
M. Lacasse et al. Expectations of Clinical Teachers and Faculty Regarding Development of the CanMEDS-Family Medicine Competencies: Laval Developmental Benchmarks Scale for Family Medicine Residency Training. 2014.
Uses a modified Delphi methodology to describe the development of expected timeframes for achievement of the CanMEDS Family Medicine (FM) competencies during FM residency training. The authors develop the Laval Developmental Benchmarks Scale for FM, which helps to identify outlier residents who are either excelling or needing remediation .
Transitioning to a CBME system requires many policy and practice changes, both at the level of the residency program and in the larger institution. Our research group employed an evidence-informed approach to determining the best practices around resident remediation, and the management of residents in difficulty, by reviewing the literature in these topic areas. Since a more purposeful move toward CBME implementation only occurred very recently, we focused our review on the last 7 years of published literature.
Identifying residents in difficulty, mainly through early and accurate assessment and faculty development, is thought to be a necessary first step in managing these residents. Management requires clear performance benchmarks by competence area, clear definitions and classifications of commonly encountered resident problems, and clear guidelines for identifying specific issues (e.g. how to identify a professionalism issue; how to gauge its severity; determining in which context(s) it is observed). Programs, and their faculty, must understand these criteria so they can to make sure residents in difficulty will be identified early and get access to remediation and other supports as early as possible. Assessments methods (e.g. advocating for more frequent formal and informal evaluation and feedback) and tools need to be improved for better tracking of each resident’s progress during remediation.
There is benefit in tailoring remediation programs to both the specialty and the resident; however, the impacts of these efforts on faculty, health care teams, and patient care must be considered closely. Customization of remediation plans can be very resource intensive. Thus, programs need to determine the appropriate amount of resources to devote to any one resident by considering the type of problem, its severity, and any pertinent resident characteristics (e.g. resident’s personality, which may be more “hard-wired” and thus more difficult to remediate).
Some researchers have pointed to the refinement of postgraduate resident selection processes as a means of limiting the percentage of residents in their programs at risk of requiring remediation. Factors such as past medical school performance can predict learner difficulty later in postgraduate training. Thus, it may be useful to consider such factors when screening applicants for entry to residency programs.
When interventions were mentioned, key information and uniformity were lacking across studies. The study by Meinema et al. encourages other authors to use a checklist to enable more “complete, comparable, and replicable descriptions of educational interventions” (2019) .
Programs will need to do some thinking around how the results of this review can best be applied to their program, for the following reasons:
Most findings are not specific to CBME systems
Despite the increasing popularity of CBME systems globally, very few articles explicitly discussed CBME in the context of resident remediation and/or residents in difficulty. In the last 2 years, the number of articles discussing CBME increased from 14 to 30%. A key implication of these findings is that universities and programs will need to translate the research findings around resident remediation to make them applicable and/or functional for their specific CBME frameworks.
Systems to oversee the promotion of residents from year to year or phase to phase were also rarely discussed. This is worrying. As more and more programs aim to implement CBME, designing better systems for oversight of resident promotion will become a crucial element in ensuring each resident’s competence throughout each stage of training.
Many of the articles may not be based in the same geographic context or may be specific to a particular specialty
As most studies were either conducted in the United States and/or published by American authors, it will be important for readers to consider the generalizability of these results. For example, attention must be paid to the different educational context(s) in which they were applied.
The literature focuses on identifying, characterizing, assessing, and remediating only certain types of resident problems
In about a quarter of the articles, remediation was not discussed relative to, or in association with, a particular specialty. Programs will have to consider whether the solutions offered in such a way can be applied to their particular specialty. Nonetheless, even if one’s specialty is not captured in this review, it might be useful to consider the practices of others so as to encourage innovative alternatives. The reviewed literature focused predominantly on the three competency areas of Medical Expert, Professionalism, and Communicator. This may be particularly useful if these are the sorts of resident problem areas that need to be managed locally.
While the findings are largely based in traditional, time-based models of education, this review can still offer general principles to guide the implementation of CBME systems for managing residents in difficulty.
Findings of CBME sub-analysis
From the CBME sub-analysis there continues to be an emphasis on identifying RID with more novel assessment methods, which are linked to milestones. Additionally, many articles describe increased frequency of assessment and link that to the role of either an Early Identification Competency Committee (Walburton ) or CCC (Domen , Ketteler , Kinnear ). These studies typically describe the members of the CCC and the frequency of meetings, along with some general statements regarding the identification of RID and tracking of progress.
A core premise with CBME is that learners will take different lengths of time to achieve competency in certain milestones. Knowing when a learner is “falling behind” versus on a slower trajectory is one of the difficult components in identifying RID. Lacasse et al.  developed the Laval Developmental Benchmark Scale (LDBS) for Family Medicine CanMEDS competencies using a Delphi methodology. They identified RID by establishing outliers based on expected developmental timelines. This work highlights the need for further research into identifying at what point on a continuum learners need more coaching (“little R” remediation) versus more formal remediation within the context of CBME.
The increase in frequency of CBME related articles in the 2015–2017 time period is encouraging. The domains from Van Melle’s framework most often map to “outcome competencies” and “programmatic assessment.”  However, there is a paucity of literature on progressive sequencing, and how instructional methods and learning experiences are tailored to those trainees.
Limitations and strengths
It is possible that the authors missed papers in their search and follow up strategies.
A limitation of the scoping review methodology, used in this paper, is that the appraisal process does not have the same rigour as a systematic review. This should be kept in mind when interpreting the results. Scoping reviews offer a bird’s-eye view of a topic and can identify key themes. This study is best used to inform about general trends and stimulate more in-depth research in the topic areas discussed.
Additionally, a small number of studies (n = 11) were found that specifically studied residents in difficulty who were in a competency-based curriculum. More research reports on residents in competency-based curricula with identified performance problems would help inform the many educators now using competency-based approaches.
A strength of this paper is the experience among the research team in both the topic and methodology: Residents in Difficulty (JP, SGT) and Scoping Reviews (LSA, SGT). Given that the use of competency-based residency educational approaches is accelerating, this review can serve as a baseline for what is known at or near the beginning, with future studies and more in-depth studies building on this starting point.
Suggestions for further research
More longitudinal studies on the impact of both competency-based approaches and remediation are needed. There is also a need to understand the effectiveness of different approaches to supporting the success of residents in difficulty. Some of the arguments about the value of competency-based approaches is that underperformance will be uncovered earlier, with more specificity of problems, with additional assessments. Research to explore the accuracy of these assertions would be helpful.
While the findings of this review are largely based in traditional, time-based models of education, it can still offer general principles that can guide the implementation of CBME systems for managing residents in difficulty.
The major themes identified in both the non-CBME and CBME articles were “identification of deficiencies” and “importance of defining and classifying resident problems or deficiencies as a first step to remediating them.” There were twice as many articles related to CBME in the last 2 years of the review as in the previous 5 years. Several studies described increased attention to identifying RID using milestones and the role of the competence committees.
Mapping of major themes to Van Melle’s  framework for CBME programs showed that much work still needs to be done to describe sequenced progression, tailored learning experiences, and competency-focused instruction. Further, there is little literature on the outcomes of remediation in CBME programs. Finally, more studies are needed that specifically describe the processes or interventions in sufficient detail to make them applicable and/or functional for the CBME frameworks of other Universities and programs.
Availability of data and materials
All data generated or analysed during this study are included in this published article [and its supplementary information files].
Remediation refers to the identification, intervention, and assessment processes at the individual and systems levels.
MR - Mariela Ruétalo. Research Officer, Post MD Education, University of Toronto
The Best Practices in Evaluation and Assessment (BPEA) Working Group was established 2016 and worked for about 18 months to help the University of Toronto Postgraduate Medical Education Office inform best practices in the area of resident evaluation and assessment to support the curricular redesign to include a competency-based approach. Information was gathered in nine areas including residents in difficulty, remediation, and board of examiners. The BPEA Working Group including its 9 teams reviewed the literature, conducted focus groups, and consulted experts, leading to the identification of recommended priority actions in the area of resident evaluation and assessment for competency based educational programs.
Accreditation Council for Graduate Medical Education
Board of Examiners
Best Practices in Evaluation and Assessment
Canadian Medical Education Directives for Specialists
Competency Based Education
Competency Based Medical Education
Clinical Competency Committee
Department of Family and Community Medicine
Designated Institutional Offices
Early Intervention Remediation Committee
Excerpta Medica dataBASE
Education Resources Information Center
Graduate Medical Education
Laval Developmental Benchmark Scale
Lisa St. Amant
Medical Literature Analysis and Retrieval System Online
Objective Standardized Clinical Examinations
Residents in Difficulty
Standardized Direct Observation Tools
Susan Glover Takahashi
Van Melle E, Frank JR, Holboe ES, Dagnone D, Stockley D, Sherbino J. A Core components framework for evaluating implementation of competency-based medical education programs. Acad Med. 2009;94(7):1002–9.
Zbieranowski I, Takahashi SG, Verma S, Spadafora SM. Remediation of residents in difficulty: a retrospective 10-year review of the experience of a postgraduate board of examiners. Acad Med. 2013;88(1):111–6.
Guerrasio J, Garrity MJ, Aagaard EM. Learner deficits and academic outcomes of medical students, residents, fellows, and attending physicians referred to a remediation program, 2006-2012. Acad Med. 2014;89(2):352–8.
Dupras DM, Edson RS, Halvorsen AJ, Hopkins RH, McDonald FS. “Problem residents”: prevalence, problems and remediation in the era of Core competencies. Am.J. Med. 2012;125(4):421–5.
Edgar L, McLean S, Hogan S, Hamstra S, Holmboe E. The Milestones Guidebook. Accreditation Council for Graduate Medical Education (ACGME); 2020 [cited 19 June 2020]. Available from: https://www.acgme.org/Portals/0/MilestonesGuidebook.pdf.
Oandasan I, Saucier D, Triple C. Competency-based curriculum. Mississauga: The College of Family Physicians of Canada; 2013. Available from: https://portal.cfpc.ca/resourcesdocs/uploadedFiles/Education/_PDFs/TripleC_Report_pt2.pdf.
About CanMEDS [Internet]. Royal College of Physicians and Surgeons of Canada. 2020 [accessed 2020 Jun 19; cited 2020 Jun 19]. Available from: http://www.royalcollege.ca/rcsite/canmeds/about-canmeds-e.
CBD. How to implement [Internet]. Royal College of Physicians and Surgeons of Canada. 2020 [accessed 2020 Jun 19; cited 2020 Jun 19]. Available from: http://www.royalcollege.ca/rcsite/cbd/cbd-implementation-e.
Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol. 2005;8(1):19–32.
Levac D, Colquhoun H, O’Brien KK. Scoping studies: advancing the methodology. Implement Sci. 2010;5:69.
Cleland J, Leggett H, Sandars J, Costa MJ, Patel R, Moffat M. The remediation challenge: theoretical and methodological insights from a systematic review. Med Educ. 2013;47:242–51.
Hsieh HF, Shannon S. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.
Aldeen AZ, Salzman DH, Gisondi MA, Courtney DM. Faculty prediction of in-training examination scores of emergency medicine residents: a multicenter study. J Emerg Med. 2014;46(3):390–5.
Dalack G, Jibson M. Clinical skills verification, formative feedback, and psychiatry residency trainees. Acad Psychiatry. 2012;36(2):122–5.
Gaviola G, Smith SE. Assessment of fellowship trainee clinical competency and growth with an objective standardized clinical examination within the musculoskeletal fellowship program: Initial experience. Miami Beach: Annual meeting of the Society of Skeletal Radiology; 2012; 2012.
House JB, Dooley-Hash S, Kowalenko T, et al. Prospective comparison of live evaluation and video review in the evaluation of operator performance in a pediatric emergency airway simulation. J Graduate Med Educ. 2012;4(3):312–6.
Webb TP, Paul J, Treat R, Codner P, Anderson R, Redlich P. Surgery residency curriculum examination scores predict future American Board of Surgery in-training examination performance. J Surg Educ. 2014;71(5):743–7.
Yost M, Gardner J, McMurtry Bell R, et al. Predicting academic performance in surgical training. J Surg Educ. 2015;72(3):491–9.
Platt MP, Davis EM, Grundfast K, Grillone G. Early detection of factual knowledge deficiency and remediation in otolaryngology residency education. Laryngoscope. 2014;124(8):E309–11.
Riebschleger MP, Haftel HM. Remediation in pediatric residency programs: a survey of pediatric program directors. Acad Paediatrics. 2011;11(4):e7.
Audétat M-C, Dory V, Nendaz M, et al. What is so difficult about managing clinical reasoning difficulties? Med Educ. 2012;46(2):216–27.
Audétat M-C, Laurin S, Dory V. Remediation for struggling learners: putting an end to 'more of the same'. Med Educ. 2013;47(3):224–31.
Audétat M-C, Voirol C, Béland N, Fernandez N, Sanche G. Remediation plans in family medicine residency. Can Fam Physician. 2015;61(9):e425–34.
Domen RE. Resident remediation, probation, and dismissal basic considerations for program directors. Am J Clin Pathol. 2014;141(6):784–90.
Anderson F, Cachia PG, Monie R, Connacher AA. Supporting trainees in difficulty: a new approach for Scotland. Scott Med J. 2011;56(2):72–5.
Katz ED, Goyal DG, Char D, Coopersmith CM, Fried ED. A novel concept in residency education: case-based remediation. J Emerg Med. 2013;44(2):493–8.
Papadakis MA, Paauw DS, Hafferty FW, Shapiro J, Byyny RL. For the Alpha Omega Alpha Honor Medical Society think tank. Perspective: the education community must develop best practices informed by evidence-based research to remediate lapses of professionalism. Acad Med. 2012;87(12):1694–8.
Sanfey H, Williams R, Dunnington G. Recognizing residents with a deficiency in operative performance as a step closer to effective remediation. J Am Coll Surg. 2013;216(1):114–22.
Barnhoorn PC. Resident remediation: start from scratch: to the editor. Am J Clin Pathol. 2015;144(3):525.
Hodges BD, Ginsburg S, Cruess R, et al. Assessment of professionalism: recommendations from the Ottawa 2010 conference. Med Teach. 2011;33(5):354–63.
Jones T, Armstrong E. The Scottish Government [Internet]. Scottish Executive. 2001 [accessed 2020 Jun 20; cited 2020 Jun 20]. Available from: https://www2.gov.scot/resource/doc/158740/0043085.pdf.
Kinnear B, Bensman R, Held J, O’Toole J, Schauer D, Warm E. Critical deficiency ratings in milestone assessment. Acad Med. 2017;92(6):820–6.
Walburton KM, Goren E, Dine CJ. Comprehensive assessment of struggling learners referred to a graduate medical education remediation program. J Grad Med Educ. 2017;9(6):763–7.
Gas BL, Buckarma EH, Mohan M, Pandian TK, Farley DR. Objective assessment of general surgery residents followed by remediation. J Surg Educ. 2016;73(6):e71–6.
McMurray L, Hall AK, Rich J, Merchant S, Chaplin T. The nightmares course: a longitudinal, multidisciplinary, simulation-based curriculum to train and assess resident competence in resuscitation. J Grad Med Educ. 2017;9(4):503–8.
Ketteler ER, Auyang ED, Beard KE, McBride EL, McKee R, Russell JC, Szoka M, Nelson T. Competency champions in the clinical competency committee: a successful strategy to implement milestone evaluations and competency coaching. J Surg Educ. 2014;71(1):36–8.
Smith JL, Lypson M, Silverberg M, Weozberg M, Murano T, Lukela M, Santen SA. Defining uniform processes for remediation, probation and termination in residency training. West J Emerg Med. 2017;18(1):110–3.
Lacasse M, Théorêt J, Tessier S, Arsenault L. Expectations of clinical teachers and faculty regarding development of the CanMEDS-family medicine competencies: Laval developmental benchmarks scale for family medicine residency training. Teach.Learn.Med. 2014;26(3):244–51.
Meinema JG, Haafkens JA, Jaarsma DADC, van Weert HCPM, van Dijk N. Development and evaluation of a culturally appropriate hypertension education (CAHE) training program for health care providers. PLoS One. 2017;12(6):e0178468.
Mariela Ruétalo. Research Officer, Post MD Education, University of Toronto was responsible for assisting in the primary screening.
The Postgraduate Medical Education (PGME) Office, University of Toronto and Associate Dean: PGME Dr. Glen Bandiera for providing administrative and operational assistance.
Ethics approval and consent to participate
Consent for publication
The authors declare they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Additional file 1:
Supplement A: References of Articles Reviewed.
Additional file 2: Supplement B:
Sample Search Strategy.
Additional file 3: Supplement C:
List of Abstracting Criteria and their Descriptions.
Additional file 4: Supplement D:
Additional file 5: Supplement E:
Other Important Resources.
Additional file 6: Supplement F:
Summary of Remediation Practices from the Literature.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Pirie, J., St. Amant, L. & Glover Takahashi, S. Managing residents in difficulty within CBME residency educational systems: a scoping review. BMC Med Educ 20, 235 (2020). https://doi.org/10.1186/s12909-020-02150-0
- Residents in difficulty
- Competency-based medical education