This article has Open Peer Review reports available.
Residents in difficulty—just slower learners? a case–control study
© O'Neill et al.; licensee BioMed Central. 2014
Received: 26 June 2014
Accepted: 15 December 2014
Published: 30 December 2014
Recent meta-analyses have found small-moderate positive associations between general performance in medical school and postgraduate medical education. In addition, a couple of studies have found an association between poor performance in medical school and disciplinary action against practicing doctors. The aim of this study was to examine if a sample of Danish residents in difficulty tended to struggle already in medical school, and to determine whether administratively observable performance indicators in medical school could predict difficulties in residency.
The study design was a cumulative incidence matched case–control study. The source population was all active specialist trainees, who were medical school graduates from Aarhus University, in 2010 to June 2013 in two Danish regions. Cases were doctors who decelerated, transferred, or dropped out of residency. Cases and controls were matched for graduation year. Medical school exam failures, grades, completion time, and academic dispensations as predictors of case status were examined with conditional logistic regression.
In total 89 cases and 343 controls were identified. The total number of medical school re-examinations and the time it took to complete medical school were significant individual predictors of subsequent difficulties (deceleration, transferral or dropout) in residency whereas average medical school grades were not.
Residents in difficulty eventually reached similar competence levels as controls during medical school; however, they needed more exam attempts and longer time to complete their studies, and so seemed to be slower learners. A change from “fixed-length variable-outcome programmes” to “fixed-outcome variable-length programmes” has been proposed as a way of dealing with the fact that not all learners reach the same level of competence for all activities at exactly the same time. This study seems to support the logic of such an approach to these residents in difficulty.
KeywordsMedical school performance Postgraduate training Problem residents Residents in difficulty Case–control study Assessment
The term “resident in difficulty” and many other synonymous terms have been used for medical residents who do not meet the expectations of the training programmes . Residents in difficulty may suffer and also strain the learning environment and patient care . It has been reported in the international literature that around 3–10% of doctors in postgraduate training are struggling to comply with educational requirements ,,. Early identification and support of residents in difficulty has been proposed as an important investment in the development and training of future health professionals, and as the gold standard for educational supervision ,-. The question is, if it is possible to identify risk factors during medical school before the unwanted outcome of a problematic residency manifests, so that resources may be directed towards preventive support of new residents at risk, instead of being used for damage control. At least three recent meta-analyses and one literature review examined the general association between performance in medical school and postgraduate medical education -. One meta-analysis including international studies reported a small effect size for previous academic performance as predictors of postgraduate performance . In this study, the predictors of previous academic performance examined were solely grade-based and the outcome measure was mainly based on performance in the first year in training after qualification. A second meta-analysis based on international literature found undergraduate grades and rankings to be moderately correlated with internship and residency performance . The third and most recent meta-analysis found performance measures available for US resident selection committees to be moderately, positively associated with resident performance . Finally, a review of studies correlating mainly US medical student performances with subsequent residency performances also found positive correlations although the overall strength of the associations examined was unclear from this study . In addition, a couple of studies have examined factors associated with extremely poor performances of practicing doctors ,,. Papadakis et al. found that the number of medical courses not passed during pre-graduate years 1–2, the MCAT z score, and reports of unprofessional behaviour in medical school were significant predictors of subsequent disciplinary action against practicing physicians . British researchers also found failed exams in the early/preclinical courses in medical school to predict professional misconduct . Generally, much of the existing research on residents in difficulty appears to be originating from English-speaking countries, in particular the US and the UK -, a situation which may lead to systematic (location/regional) bias. No cohort study on the risk factors for being a resident in difficulty in a Danish context has previously been published nationally or internationally.
Danish residents are typically selected for specialist training positions based on their postgraduate curriculum vitae and on personal interviews. There is no tradition for using medical school performance indicators in the planning of residencies, i.e. residents are not evaluated on their profiles as learners in medical school, their habits or their learning needs in advance. Whether such an evaluation of medical school performance is justified at all in a Danish context has never been examined before, even though the aggregated international evidence as described above seem to generally suggest a small-moderate association between medical school performance and subsequent doctor performance. An important reason for the lack of validity studies for medical school performance in a Danish context is most likely that postgraduate medical performance is not graded with a score or a mark. The aim of this study was to examine if residents in difficulty in a Danish context tended to struggle already in medical school or not compared to controls, and to determine if/which administratively observable performance indicators in medical school would predict difficulties in residency. This study reports the results of a case–control study that examined medical school predictors of struggling to complete residency training as planned. The results will be discussed and compared with relevant literature in medical education.
The study design was a cumulative incidence matched case–control study, which is rooted in epidemiological methodology, but the method is also used by the research community in medical education ,,,. This design was chosen because residents in difficulty (cases) are relatively rare and because the time from exposure (early medical school) to manifest (struggling in residency) is long. The incidence of residents in difficulty has recently been estimated to be about 7% in the regions of our source population .
The source population in this study was all active specialist trainees, who were graduates from Aarhus University, in 2010–June 2013 in two Danish regions (Central and North), which until recently was the only medical school in these regions. In other words, the source population constitutes medical doctors with at least two years of basic postgraduate clinical training who had begun their specialist training programmes. The general contract for specialist training is that residents are to complete their residency in four or five years depending on specialty to be authorized. Within this timeframe, they need to comply with a set of minimum competences, which are based on the CanMEDS physician competency framework , and complete a number of obligatory course and rotations in order to pass training. No grades are given in postgraduate training. The definition of residents in difficulty used by the postgraduate medical training authorities in our region was outlined in an official document entitled “Handling Problematic Residencies”. In this document a resident in difficulty is defined as “a doctor who does not acquire the required competences within the planned timeframe of the programme”. The three main strategies for dealing with these residents in the region are outlined in this document to be either: programme deceleration, ward transferral or even resident dismissal in the rare cases where problems remain unresolved and severe . We therefore chose to define cases by a composite outcome, as doctors in residency who either: decelerated, transferred or dropped out. We included voluntary resignations and changes of speciality in the dropout category, because they are also indicative of problems with completing a programme as planned. We defined deceleration as having had long or repeated episodes of: leave of absence, unknown absence, illness, or delays due to inadequate development of competences. “Long” episodes of absence were defined as episodes lasting more than two months, and “repeated’ episodes of absence were defined as two or more episodes of absence. Transferral was defined as having had unplanned changes of ward of training due to: failure to thrive or due to inadequate development of competences. Residents were considered to be dropouts if they were dismissed from, had resigned from, or changed their speciality. Exclusion criteria were: decelerations due to maternity leave, including unknown absence three months before/after maternity leave, as well as changes of specialty during the introductory training positions (non-residency positions), which were not uncommon events. Controls were a random sample of doctors in the source population, who were not identified as cases in the case extraction period. We aimed at a case–control ratio of 1:4 and matched cases and controls on graduation year.
Cases were identified after evaluation of individual residency records in an existing database (evaluer.dk), which tracks the progression of individual doctors in postgraduate training in Denmark, and via the records of the regional Council for Postgraduate Medical Training. Lists of all potential controls (graduates from the same year and institution) were supplied by administrators at Aarhus University (AU). The order of graduates in each cohort of potential controls was randomized by researchers using the website random.org. Subsequently, potential controls were checked for membership and source population and picked in this random order if they belonged to the source population. Data on medical school performance and progression was extracted from the AU administrative databases, and data on dispensations (dates and causes for seeking academic dispensations) were extracted manually from paper files in a physical archive at AU. Data extraction took place between June 2013 and February 2014.
The study was exempt from ethics review by the regional ethics committee. The authors also obtained permission from the Danish Data Protection Agency to use and combine the specific data extracted from the specific sources for the purpose of this study as required by Danish law.
Potential predictor variables representing available measures of medical school performances relating to re-examinations, grades, completion time, and dispensations were examined, as was demographic variables.
A variable for gender was supplied with the data set from AU. We formed a variable representing the age at admission to medical school (“Admission age”), by subtracting students’ birth date from the date they were registered as starting medical school at AU.
Three variables were formed representing the number of re-examinations in medical school’ taken by students during the first year (“‘Year 1 re-examinations”), the first two years (“Year 1 & 2 re-examinations”), and in total (“Total re-examinations”).
Medical school grades
Medical school grades were calculated as an equally weighted average of all number grades given to a student during medical school, i.e. pass/fail grades were excluded. The grading scale was the Danish seven-point grades scale that contains the grades: −3 (unacceptable), 0 (inadequate), 2 (adequate), 4 (fair), 7 (good), 10 (very good), and 12 (excellent).
Programme completion time (“Completion time”) was calculated as the number of years between the day students commenced the medical programme administratively and the day of the last examination. The current rules of progression allow students up to 12 years to complete the six-year medical school curriculum.
Students had to seek dispensations if their progression deviated from academic rules and regulations, i.e. if they required extra time for examinations for some reason, or if they needed more than three attempts to pass an examination, etc. We generated one variable for the number of dispensations sought during the first year of study (“Year 1 dispensations”) and one for the total number of dispensations sought in medical school (“Total dispensations”). In addition we categorized the causes given for seeking dispensations into a number of relevant categories. The categorization was performed independently by two researchers.
Differences amongst cases and controls were examined with either t-tests (continuous data), or Χ2 tests or Fisher’s exact tests (categorical data). We used conditional logistic regression to examine predictor variables of case-status and to estimate odds ratios (OR) and their 95% confidence intervals (95% CI). Before multivariate analysis all predictor variables were screened for collinearity and zero cells. We examined a multivariate model using backwards hierarchical elimination , and one in which we included only the significant (p < 0.05) univariate predictor variables. Descriptive summary statistics and analyses were performed using STATA/IC 12.
Causes for seeking academic dispensations in cases and controls (n = 432)
Poor organizational skills
Applications for dispensations due to missed deadlines, e.g. late registrations for exams or submission of assignments, etc.
Applications for extended examination time, changes in order and timing of courses and examinations, etc. due to pregnancy.
Applications grounded in problematic or serious familial events or situations, e.g. burials, parental responsibilities, life threatening disease, etc.
Applications—typically relating to examinations (i.e. extended examination time, extra exam attempts, re-examinations)—which were caused by chronic illnesses, handicaps, or dyslexia.
Applications for a 4th or 5th exam attempt where no causes were given, as well as applications for remediation activities.
Applications for re-examinations on grounds of shorter lasting illnesses such as influenza, etc.
Study plan deviations
Applications for changes from the standard study plan, i.e. changes in the order of subjects, time of examination, etc. We distinguished between three subcategories of study plan deviations: positive, neutral or negative. In the positive subcategory the causes related to relevant research or international exchange activities, etc. In the neutral category we were either unable to ascertain a cause or unable to determine whether a cause was positive or negative. Finally, in the negative deviations category, there were clear signs of students prioritizing other activities above medical school activities, for example: holidays booked during terms, family birthdays, sport activities, etc., or signs that students were struggling to keep up with the ordinary study plan.
Applications for merits for courses taken in other programs, or advance approval of planned relevant educational activities such as electives and placements, etc.
Summary of medical school performance indicators for cases (n = 89) and controls (n = 343)
Residents in difficulty n
Admission age (years)
Graduation year (calendar year)
Year 1 re-examinations
Year 1 & 2 re-examinations
Medical school grades (Danish 7-point scale)
Completion time (years)
Year 1 dispensations
Poor organizational skills
Study plan deviations
Predictors of struggling in residency
OR (95% CI)
OR (95% CI)
OR (95% CI)
Year 1 re-examinations
Year 1 & 2 re-examinations
Total no. of re-examinations
Average medical school grades
Completion time (years)
No. of year 1 dispensations
Total no. of dispensations
Exposures and odds ratios for struggling in residency
Unadjusted OR (95% CI)
Medical school completion time:
No. of medical school re-examinations:
We found the total number of re-examinations during medical school and the time it took to complete medical school to be significant univariate predictors of observable difficulties in residency training (deceleration, transferral or dropout). We were not able to model case status with a multivariate model when using backwards hierarchical elimination. We suspect that the bivariate model (model 2) that included both of the significant univariate predictors (total re-examinations and completion) were unstable due to collinearity, as the two predictors were correlated (r = 0.432, p = 0.000), because repeated examination failures generally tend to mess up students’ study plans and delay their completion. Another cause of late completion may be leaves of absence during medical school, which has previously been shown to be associated with drop out from this particular medical programme . It is therefore likely that late completion may be associated not only with academic problems like failed examinations, but perhaps also with factors such as stress/burnout, or perhaps with a waning commitment to medicine. One review of the literature on residents in difficulty found that they were most often identified by programme directors on the grounds of insufficient medical knowledge and poor clinical judgment, but also on inefficient use of time . Other authors have reported absenteeism, poor attendance, being slow or late as observable behaviours of both struggling medical students and residents in difficulty ,,-. Our results suggest that both prior academic challenges (repeated exam failures) and being a slow learner (longer completion times) were associated with struggling to comply with the demands of subsequent residency training. In contrast to recent meta-analyses and reviews -, we did not find any association between medical school grades and residency performances (Table 3). There is certainly a universal tendency for “restriction of range” when using medical school grades as predictors of performance in postgraduate training, which may diminish the chance of finding strong associations.8 Restriction of range probably also affected our results, since the academically worst performing medical students at AU tend to drop out early in medical school, and the overall dropout rate has been found to be quite high (20%) ,. While correction for restriction of range is statistically possible , we do not find it valid considering the aim of this study (early identification), as residents do no present themselves in the wards with academic records, which are corrected for unreliability or restriction of range. Another, and most likely better explanation for the lack of predictive power of medical school grades in this context, is the fact that Danish universities are bound by national laws to be very liberal with examination attempts (minimum three per subject) and completion times (maximum 12 years for a six-year programme). That is given some extra learning time and examination attempts, our cases eventually reached similar levels of competence in medical school as controls (Table 2). In addition, we did not find significant differences in the patterns of dispensation applications amongst our cases and controls either (Tables 2), which we would expect if cases were considerably weaker than controls academically. Local researchers recently examined dispensation applications in this same medical programme with a retrospective cohort study of 1,056 medical students admitted between 2003 and 2006. A significantly larger proportion of dispensations seekers dropped out compared to non-dispensations seekers (58% versus 17.6% dropout) . It is therefore likely that students with the most severe academic problems reflected in academic dispensations never made it through to graduation, residency or to our sample.
Our results would suggest that ignoring past performances is not the best way to help at-risk doctors with a smooth transition to postgraduate training. Instead, specific medical school performance indicators could be included in the introductory dialogues between doctors and workplaces in order to align expectations before designing and implementing tailored, individual learning plans. Our results would suggest that residents in difficulty were slower learners, and they could be under time pressure in residency programmes because of the fixed-length employment relationships. They may struggle because fixed-length training programmes are too rigid for “slow learners”—not necessarily because they will become poorer doctors long term. In other words, the “one-size-fits-all” approach to residency training could be seen as the main problem, and the proven learning potential of residents in difficulty as the solution. Recent developments in competency-based education have highlighted the need for a shift from “fixed-length variable-outcome programmes” to “fixed-outcome variable-length programmes” -. Ten Cate et al. emphasized that “the goal of competency-based education should be linked to observed competence, rather than to the current proxy measure, length of training”, and that “as educators, we should recognize that all trainees do not reach the same level of competence for all activities at exactly the same time” . Our results seem to support that line of thinking, because while there were no significant differences in the final average medical school grades or “competences” of cases and controls, there were differences in the “process” of reaching similar competency levels.
Strengths and limitations
The strength of this case–control study was that there was no missing data in the data extracted from the database, and that recall bias was not an issue. On the other hand, the databases and archives from which data was extracted, were kept and maintained for administrative purposes, and they may have contained errors and have been incomplete (e.g. missing student dispensation case paper files or missing documents within these files). We had no control over these types of bias with this design. Misclassification of variables is also always a possibility when handling large amounts of complex data as in this case. The collinearity mentioned above is a likely limitation, which may have prevented the control for confounding. A methodological limitation is that matching for graduation year does not secure the exact same exposures during medical school for cases and controls, since completion time does vary; however, there is no immediate solution to that problem . It was a strength that we were able to examine predictors of difficulties in residency across all types of specialties in two of five Danish regions. On the other hand, it was a limitation that we only had access to medical school performance data for doctors graduating from one particular institution. Finally, it is a strength that we are able to report results from a different educational and cultural setting than most of the previously published literature on the subject -—a setting with liberal progression rules for pre-graduate medical education.
Residents in difficulty eventually reached similar competence levels as controls during medical school when given the opportunity. However, they needed more exam attempts and longer time to complete their pre-graduate studies, and so seemed to be slower learners in general. Knowledge of relevant medical school performance indicators could be used constructively in discussions with the resident to tailor and plan a more appropriate and individual residency. A change from “fixed-length variable-outcome programmes” to “fixed-outcome variable-length programmes” has been proposed as a way of dealing with the fact that not all learners reach the same level of competence for all activities at exactly the same time. This study seems to support the logic of such an approach to residents in difficulty.
The authors would like to thank Professor Berit Eika, Vice-Chancellor for Education at Aarhus University, as well as Marianne Toftegaard Hansen and Eline Skjøttgaard Pallesen, both from AU Studies Administration for their unyielding support and all their help with data extraction. The study was funded partly by the governing body of public health care in the central region of Denmark (Region Midtjylland) and partly by the Centre for Medical Education at Aarhus University. The authors were autonomous in all decisions relating to the research process and publication of this paper.
- Steinert Y: The “problem” learner: whose problem is it? AMEE Guide No. 76. Med Teach. 2013, 35: e1-11. 10.3109/0142159X.2013.774082.View ArticleGoogle Scholar
- Brenner AM, Mathai S, Satyam J, Mohl PC: Can we predict “problem residents”?. Acad Med. 2010, 85: 1147-1151. 10.1097/ACM.0b013e3181e1a85d.View ArticleGoogle Scholar
- Tabby DS, Majeed MH, Schwartzman RJ: Problem neurology residents. Neurology. 2011, 76: 2119-2123. 10.1212/WNL.0b013e31821f4636.View ArticleGoogle Scholar
- Zbieranowski I, Takahashi SG, Verma S, Spadafora SM: Remediation of residents in difficulty: a retrospective 10-year review of the experience of a postgraduate board of examiners. Acad Med. 2013, 88: 111-116. 10.1097/ACM.0b013e3182764cb6.View ArticleGoogle Scholar
- Yates J, James D: Risk factors at medical school for subsequent professional misconduct: multicentre retrospective case–control study. BMJ. 2010, 340: c2040-10.1136/bmj.c2040.View ArticleGoogle Scholar
- Evans DE, Alstead EM, Brown J: Applying your clinical skills to students and trainees in academic difficulty. Clin Teach. 2010, 7: 230-235. 10.1111/j.1743-498X.2010.00411.x.View ArticleGoogle Scholar
- Yao DC, Wright SM: The challenge of problem residents. J Gen Intern Med. 2001, 16: 486-492. 10.1046/j.1525-1497.2001.016007486.x.View ArticleGoogle Scholar
- Ferguson E, James D, Madeley L: Factors associated with success in medical school: systematic review of the literature. BMJ. 2002, 324: 952-957. 10.1136/bmj.324.7343.952.View ArticleGoogle Scholar
- Hamdy H, Prasad K, Anderson MB, Scherpbier A, Willians R, Zwiestra R, Cuddihy H: BEME systematic review: predictive values of measurements obtained in medical schools and future performance in medical practice. Med Teach. 2006, 28: 103-116. 10.1080/01421590600622723.View ArticleGoogle Scholar
- Kenny S, McInnes M, Singh V: Associations between residency selection strategies and doctor performance: a meta-analysis. Med Educ. 2013, 47: 790-800. 10.1111/medu.12234.View ArticleGoogle Scholar
- Harfmann KL, Zirwas J: Can performance in medical school predict performance in residency? A compilation and review of correlative studies. J Am Acad Dermatol. 2010, 65: 1010-1022. 10.1016/j.jaad.2010.07.034.View ArticleGoogle Scholar
- Papadakis MA, Teherani A, Banach MA, Knettler TR, Rattner SL, Stern DT, Veloski JJ, Hodgson CS: Disciplinary action by medical boards and prior behavior in medical school. NEJM. 2005, 353: 2673-2682. 10.1056/NEJMsa052596.View ArticleGoogle Scholar
- Papadakis MA, Hodgson CS, Theherani A, Kohatsu ND: Unprofessional behavior in medical school is associated with subsequent disciplinary action by a state medical board. Acad Med. 2004, 79: 244-249. 10.1097/00001888-200403000-00011.View ArticleGoogle Scholar
- Ringsted C, Hodges B, Scherpbier A: ‘The research compass’: an introduction to research in medical education: AMEE Guide no. 56. Med Teach. 2011, 39: 695-709. 10.3109/0142159X.2011.595436.View ArticleGoogle Scholar
- Kelsey JL, Whittemore AS, Evans AS, Thompson WD: Methods in Observational Epidemiology. 1996, Oxford University Press, New YorkGoogle Scholar
- Hansen DH, O’Neill L, Charles P, Mortensen LS, Norberg K, Christensen MK: Problem Residents and Inadequate Postgraduate Training: Individual, Cultural and Organizational Matters. 2013, Presented at the Association for Medical Education in Europe (AMEE) Conference, Prague, Czech RepublicGoogle Scholar
- The CanMEDS 2005 Physician Competency Framework. Better Standards. Better Physicians. Better Care. 2005, The Royal College of Physicians and Surgeons of Canada, OttawaGoogle Scholar
- Greve J: Håndtering af det uhensigtsmæssige uddannelsesforløb. 2006, Det Regionale Videreuddannelsesråd, Region Nord, ViborgGoogle Scholar
- Kleinbaum DG, Klein M: Logistic Regression. 2002, Springer, New York, 2Google Scholar
- Greenland S, Schwartzbaum JA, Finkle WD: Problems due to small samples and sparse data in conditional logistic regression analysis. Am J Epid. 2000, 151: 531-539. 10.1093/oxfordjournals.aje.a010240.View ArticleGoogle Scholar
- Maher BM, Hynes H, Sweeney C, Khashan AS, O’Rourke M, Doran K, Harris A, O’Flynn S: Medical school attrition-beyond the statistics. A ten year retrospective study. BMC Med Educ. 2013, 13: 13-10.1186/1472-6920-13-13.View ArticleGoogle Scholar
- Roberts NK, Williams RG, Klingensmith M, Sullivan M, Boehler M, Hickson G, Kim MJ, Klamen DL, Leblang T, Schwind C, Titchenal K, Dunnington GL: The case of the entitled resident: a composite case study of a resident performance problem syndrome with interdisciplinary commentary. Med Teach. 2012, 43: 1024-1032. 10.3109/0142159X.2012.719654.View ArticleGoogle Scholar
- Hays RB, Lawson M, Gray C: Problems presented by medical students seeking support: a possible intervention framework. Med Teach. 2011, 33: 161-164. 10.3109/0142159X.2010.509415.View ArticleGoogle Scholar
- Mørcke AM, O’Neill L, Kjeldsen IT, Eika BE: Selected determinants may account for dropout risks among medical students. Dan Med J. 2012, 59: A4493.Google Scholar
- Urup Ruban P, Gammelgaard Petersen M, Møller-Madsen B: More than half of the medical students who apply for a dispensation drop out and need focused counselling. Dan Med J. 2013, 60: 1-5.Google Scholar
- Carraccio C, Wolfsthal SD, Englander E, Ferenz K, Martin C: Shifting paradigms: from Flexner to competencies. Acad Med. 2002, 77: 361-367. 10.1097/00001888-200205000-00003.View ArticleGoogle Scholar
- ten Cate O: Entrustability of professional activities and competency-based training. Med Educ. 2005, 39: 1176-1177. 10.1111/j.1365-2929.2005.02341.x.View ArticleGoogle Scholar
- ten Cate O, Scheele F: Competency-based postgraduate training: Can we bridge the gap between theory and clinical practice?. Acad Med. 2007, 82: 542-547. 10.1097/ACM.0b013e31805559c7.View ArticleGoogle Scholar
- ten Cate O, Snell L, Carraccio C: Medical competence: the interplay between individual ability and the healthcare environment. Med Teach. 2010, 32: 669-675. 10.3109/0142159X.2010.500897.View ArticleGoogle Scholar
- ten Cate O: Nuts and bolts of entrustable professional activities. J Grad Med Educ. 2013, 5: 157-158. 10.4300/JGME-D-12-00380.1.View ArticleGoogle Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.