- Research article
- Open Access
- Open Peer Review
Identification of foundational non-clinical attributes necessary for successful transition to residency: a modified Delphi study with experienced medical educators
BMC Medical Education volume 18, Article number: 150 (2018)
We aimed to identify foundational non-clinical attributes expected of medical school graduates to be successful in residency.
We conducted a three-round modified Delphi study with snowball sampling of experienced medical educators. In Round 1, respondents rated 28 attributes identified from a literature search. Additional attributes were proposed through invited comments. In Round 2, respondents expressed their agreement with advanced attribute definitions and examples. Consensus on final definitions and examples was obtained in Round 3.
Sixty-four percent (105/163) of invited educators participated in Round 1. There was broad representation of educational focus (undergraduate, graduate, and continuing medical education) and field of practice (primary care, sub-specialty, medical, and surgical). Thirteen attributes were advanced to Round 2. Ninety-seven of 105 (92%) respondents participated in Round 2, with greater than 92% agreement for all attributes. Three pairs were consolidated. In Round 3, 88% (85/97) of educators expressed greater than 92% agreement about definitions and representative examples. The final 10 foundational attributes are: communication skills, critical thinking, emotional intelligence, ethical behavior, intellectual curiosity, organizational skills, resilience, self-improvement, teamwork, and vocational commitment.
Through a consensus-building process of medical educators, we identified and defined 10 foundational non-clinical attributes for a medical student’s successful transition to residency.
Calls for reform in medical education emphasize the need for more standardized learning outcomes that foster greater integration across the educational continuum [1, 2]. Achieving this vision of integration between undergraduate and graduate medical education requires a thorough understanding of the attributes and competencies expected of medical school graduates. Clearly defined outcomes can then be used to inform a competency-based approach to curriculum development, teaching, and assessment throughout the entirety of medical school . In particular, accepted attributes and competencies for the successful transition from medical school to residency will better guide curriculum design for the final, and often underutilized, year of medical school which could potentially transform it into a lynchpin for successful integration of the medical education continuum [3,4,5].
National efforts have made great strides towards defining the competencies needed for students’ successful transition to residency. The American Association of Medical Colleges has engaged in two efforts to more thoroughly define competencies appropriate for medical students – the Physician Competency Reference Set (PCRS) and the Core Entrustable Professional Activities for Entering Residency (core EPAs) [6, 7]. While both efforts have the goal of standardizing learning outcomes in medical school, the core EPAs have the added goal of fostering successful student preparation for residency by clearly delineating activities that all graduating medical students should be expected to perform on day one of residency without direct supervision . Initiatives to implement the core EPAs are underway at numerous medical schools around the country as a means of trying to improve medical student preparation for residency [8, 9]. Importantly, both of these competency frameworks rely on foundational attributes, or habits of the mind, which are seen as essential to determining ultimate competence or entrustment. For instance, competencies have been framed as requiring attentiveness, critical curiosity, self-awareness and presence, while EPA’s require trustworthiness and self-awareness .
Studies across multiple specialties repeatedly support the notion that non-clinical attributes are critically important to successful performance in residency [11,12,13,14,15,16,17,18]. Examples of these behaviors include having a commitment to learning, being conscientious, being curious, recognizing one’s limits, exhibiting professional behavior, and having strong interpersonal skills [11,12,13,14]. One study suggests that only 30% of attributes related to successful performance are those traditionally thought of as cognitive (i.e., knowledge and technical skill related) . Another study identifies multiple non-clinical attributes as contributors of difficulty for new interns, including lack of self-reflection skills or poor organizational skills .
Unfortunately, many of these foundational non-clinical attributes are less well studied in medical education, often only indirectly addressed in clinical training, and subsumed into the broader PCRS and core EPAs. So, while these current efforts are necessary to ensure a student’s successful transition to residency, they may insufficiently emphasize non-clinical habits of mind which have also been shown to confer success in residency [11,12,13,14,15,16,17,18]. This success clearly depends on competencies and behaviors that extend beyond the cognitive elements of medical knowledge and patient care [6, 14].
In this study, we set out to systematically identify and define the foundational non-clinical attributes expected of medical school graduates necessary for success in residency training. In doing so, we hope to inform individualized student learning programs, curricular design, and final-year scheduling. Using a modified Delphi method of expert consensus, a snowball sample of experienced educators participated in defining the foundational, non-clinical attributes expected of all medical school graduates.
Three rounds of a modified Delphi procedure, with snowball sampling of faculty with medical education expertise, were used to identify and define non-clinical attributes expected of medical school graduates to be successful in residency training. The Delphi technique is a type of exploratory analysis in which multiple rounds of surveys are used to obtain a reliable consensus from a group of experts [19, 20]. A characteristic hallmark of this method is the use of the participants’ responses from previous rounds as the basis for subsequent rounds. Only those who participate in the previous round are invited to participate in a subsequent round. The success of a Delphi process is dependent on continued participation throughout multiple rounds. Unlike a standard Delphi process, our methodology was modified in that the researchers provided an initial list of attributes for the participants and generated definitions for subsequent rounds for consideration by the Delphi participants. All materials considered by the Delphi participants were developed by the research team through an iterative discussion process based on the participants’ responses from previous rounds. This study was approved by the Colorado Multiple Institutional Review Board.
Subjects and sampling
Participants for this study were drawn from a systematic cross specialty sample of experienced educators at the University of Colorado. All invited participants had a significant educational role in undergraduate, graduate, or continuing medical education as a core educator, mentor, or program leader. We sought to include a diverse group of faculty in terms of the training level of their educational focus (undergraduate, graduate, and continuing medical education) and their field of practice (primary care, sub-specialty, medical and surgical). See Table 1 for the characteristics of study participants. The research team monitored participation to assure the inclusion of varying perspectives across each round of the modified Delphi process.
The research team generated an initial list of 45 faculty felt to have expertise in undergraduate, graduate, and continuing medical education. Snowball sampling then allowed the initial participants to identify additional faculty who they thought could contribute to this process. These additional participants were then added to the initial list and invited to participate in the process. No incentives were provided for participation in the study.
Modified Delphi process
An extensive literature search was performed in 2011 to identify potential undergraduate medical education non-clinical attributes necessary for success in residency. Relevant search terms included: fourth year, undergraduate medical education, competency-based education, professional competence, and graduate medical education. Abstracts were reviewed and relevant articles were read entirely. Reviewing the bibliographies of the initial articles identified additional relevant articles to include. This search, and accompanying discussions among the researchers, was used to create an initial list of non-clinical attributes for use in the first round of the modified Delphi process. In 2012, an online survey was then created for each round of the modified Delphi process. Figure 1 contains the steps of the modified Delphi process.
The purpose of the first round was to identify the non-clinical attributes necessary for success in residency. We asked participants to react to the research team’s generated list of attributes from our review of the literature, rather than generate their own list. An open-ended question, however, allowed the respondents to suggest other attributes not included in the original list. Respondents rated the importance of each of the potential attributes on a six-point Likert scale from “not at all important” to “extremely important.” See Table 2 for list of initial attributes. In addition, respondents were asked to select the five most essential and the five least essential attributes for success in residency training.
Criteria for advancement of attributes to Round 2 were developed, based on iterative discussions among the researchers, with the goal of accurately reflecting the priorities of the Delphi participants. Ratings for each attribute and comments were reviewed. Respondent characteristics were reviewed to assure that sampling was adequately diverse prior to ending Round 1. After discussion, researchers agreed to advance attributes to Round 2 based on a mean importance score of greater than 5, with consideration of percent of respondents naming an attribute as the most and least important.
Definitions and representative examples for all the attributes included in Round 2 were developed through an iterative process in which all the researchers reviewed comments from the participants in the first round and the articles from the literature search. The purpose of the second round was to achieve agreement among proposed definitions and examples. Respondents from the first round were asked to agree, agree with modifications, or disagree with the definition of each attribute. Respondents were also asked if any attributes should be subdivided into multiple other attributes or if multiple attributes should be combined into one. Informed with these responses and comments, the research team engaged in iterative discussions in order to achieve consensus regarding which attributes were to be subdivided or combined before advancing to Round 3. Additionally, attribute definitions and representative examples were refined for clarity.
The purpose of Round 3 was to achieve final agreement on the list of attributes, their definitions, and representative examples. All attributes, either in their original or combined forms, were advanced in this round. Respondents from the second round were asked to indicate their level of agreement with the final attributes, definitions, and representative examples.
The systematic literature search yielded 48 relevant articles that were reviewed by the researchers and an annotated bibliography was created. Based on these articles, we compiled a list of 28 non-clinical attributes relevant to undergraduate and graduate medical education.
After snowball sampling, 163 experienced educators were invited to participate. One hundred and five (105) of the 163 faculty participated (64% response rate) in the first round. The overall mean rating of importance of all attributes was 4.84 out of 6 (standard deviation = 0.89). Based on discussion of the data, 13 attributes were advanced to the second round (see Table 2).
In Round 2, 97 of 105 participants from Round 1 responded (92% response rate). Between 70 and 90% of respondents agreed with the definitions provided and representative examples of each attribute (see Table 2). Based on comments from the respondents about overlaps, three sets of attributes were consolidated (clinical reasoning into critical thinking; empathy into emotional intelligence; time management and prioritization of tasks into organizational skills). The definitions and examples of the remaining attributes were refined, incorporating perspectives from the respondents’ comments during this round.
In Round 3, 85 of 97 experienced educators responded (88% response rate), and there was greater than 92% agreement about the definition and representative examples for all attributes (see Table 2). Based on the results of this final round, we defined 10 non-clinical attributes that are expected of medical school graduates for success in residency training: communication skills, critical thinking, emotional intelligence, ethical behavior, intellectual curiosity, organizational skills, resilience, self-improvement, teamwork, and vocational commitment (see Table 3 for attribute definitions with representative examples).
Based on input from experienced educators in undergraduate, graduate, and continuing medical education, we identified and defined 10 foundational non-clinical attributes necessary for a medical student’s success in residency. Derived from individual educators and across various specialties, these attributes can be considered fundamental to working in almost all clinical settings a new resident might encounter. As such, these attributes could be used to further define and augment medical school curricula and better prepare medical students for residency training.
To date much literature has been published about the competencies and competency domains needed for the successful completion of medical school and graduate medical education [5,6,7,8,9, 21]. One recent article reviewed the literature on competency frameworks and proposed a common taxonomy of competencies for physicians that includes eight domains and 58 competencies in eight domains (i.e., the Physician Competency Reference Set) . Within this taxonomy, non-clinical attributes and competencies are encompassed across multiple domains, emphasizing their importance. All but two of the non-clinical attributes identified in our study can be mapped within the PCRS taxonomy, albeit sometimes indirectly. For example, in our study, emotional intelligence emerged as an independent attribute, while it is subsumed by interpersonal and communication skills in the PCRS. Two of our attributes, vocational commitment and organizational skills, are not explicitly included in the PCRS. The differences between our results and the PCRS may be attributed to the explicit goal of our study to determine the non-clinical attributes for success in residency, rather than a comprehensive list of competencies. This resulted in a more granular list of specific attributes that can be fostered and developed during the fourth year of medical school to support the transition to residency. Regardless, this similarity in results provides confirming evidence for content validity and further supports the findings of our study.
The Core Entrustable Professional Activities for Entering Residency (Core EPAs) is another competency framework related to ours . This framework identifies 13 activities central to all residents that medical students should be able to complete prior to the start of residency and that are designed to holistically assess medical student performance in preparation for residency. Importantly, the core EPAs do not represent all that is expected from medical school graduates.
Our attributes were derived in a way that emphasizes the non-clinical attributes rather than the clinical activities necessary for residency. In addition, there is similarity between the findings of our study and the Accreditation Council for Graduate Medical Education (ACGME) Milestones for residency training [22,23,24]. For example, the ten attributes we identified are included in the 21 pediatric milestones to be reported every 6 months to the ACGME . Similar to our emphasis on non-clinical attributes, of the 21 pediatric milestones, only 6 are related to patient care or medical knowledge, while the other 15 reflect the multitude of non-clinical competencies necessary for a pediatrician . As such, we believe our attributes can be used to supplement efforts to implement the core EPAs and Milestones for both teaching and assessment.
Ultimately, we believe our 10 non-clinical attributes should be emphasized throughout all four years of medical school, not simply during the fourth year. Competency-based medical education involves starting with the end in mind and defining the competencies expected of graduates which can then inform curriculum development . The idea that these attributes should be addressed throughout medical school is supported by a recent study of the core personal competencies considered in medical school admissions processes . The competencies identified in that study are very similar to our findings, which are also similar to the findings from the critical incident studies for residency education, further supporting the importance of this set of skills [11,12,13,14,15,16,17,18].
The attributes identified in our study can be used in future studies to examine prospectively how non-clinical attributes may impact performance in residency education. In addition, the perspectives of other stakeholders, such as students or residents, about the importance of these attributes would further enhance our findings. Finally, efforts to foster and develop these attributes in medical student prior to medical school graduation should be developed and studied.
There were several limitations to our study. First, it was a single institution study, only engaging faculty working within a single culture and single approach to the fourth year of medical school. Faculty at different institutions may prioritize other attributes expected of medical school graduates necessary for success in residency training. Second, we opted to start with a pre-defined list of attributes rather than ask an open-ended question for respondents to list attributes on their own. This modified Delphi process was chosen to base our research on the current literature; however it could have limited the inclusion of additional important attributes. The fact that there were no attributes added to the list after the first round, in which participants were specifically asked if there were other attributes to include, suggests that this method may not have significantly impacted our results.
Through a modified Delphi process of experienced medical educators, we identified and defined 10 foundational non-clinical attributes necessary for graduating medical students to be best prepared for residency training. These attributes can be used to guide curriculum development in the fourth year of medical school, or even earlier.
Accreditation Council for Graduate Medical Education
Entrustable Professional Activity
Physician Competency Reference Set
Cooke M, Irby DM, O'Brien BC. Educating physicians: a call for reform of medical school and residency. Vol 16. San Francisco: Jossey-Bass; 2010.
Weinstein D. chair. Ensuring an Effective Physician Workforce for the United States: Recommendations for Reforming Graduate Med Educ to Meet the Needs of the Publica Paper presented at: Proceedings of a Conference Sponsored by the Josiah Macy Jr. Foundation. New York, NY: Josiah Macy Jr. Foundation 2011.
Wolf SJ, Lockspeiser TM, Gong J, et al. Student perceptions on the fourth year of medical school: a mixed methods analysis. Acad Med. 2014;89:602–7.
O’Brien BC, Niehaus B, Teherani A, et al. Residents’ perspectives on the final year of medical school. International Journal of Medical Education. 2012;3:151–8.
Reddy ST, Chao J, Carter JL, et al. Alliance for clinical education perspective paper: recommendations for redesigning the “final year” of medical school. Teach Learn Med. 2014;26:420–7.
Englander R, Cameron T, Ballard AJ, et al. Toward a common taxonomy of competency domains for health professions and competencies for physicians. Acad Med. 2013;88:1088–94.
Englander R, Flynn T, Carraccio C, et al. Toward defining the foundation of the MD degree: Core Entrustable professional activities for entering residency. Acad Med. 2016;91:1352–58.
Lamba S, Wilson B, Natal B, et al. A suggested emergency medicine boot camp curriculum for medical students based on the mapping of Core Entrustable professional activities to emergency medicine level 1 milestones. Adv Med Educ Pract. 2016;7:115–24.
Vu TR, Angus SV, Aronowitz PB, et al. The internal medicine subinternship – now more important than ever: a joint CDIM-APDIM position paper. J Gen Intern Med. 2015;30:1369–75.
Epstein RM, Hundert EM. Defining and assessing professional competence. JAMA. 2002;287:226–35.
Tarico V, Smith WL, Altmaier E, et al. Critical incident interviewing in evaluation of resident performance. Radiology. 1984;152:327–9.
Altmaier EM, McGuinness G, Wood P, et al. Defining successful performance among pediatric residents. Pediatrics. 1990;85:139–43.
Altmaier EM, Johnson SR, Tarico VS, et al. An empirical specification of residency performance dimensions. Obstet Gynecol. 1988;72:126–30.
Wood PS, Smith WL, Altmaier EM, et al. A prospective study of cognitive and noncognitive selection criteria as predictors of resident performance. Investig Radiol. 1990;25:855–9.
Altmaier EM, From RP, Pearson KS, et al. A prospective study to select and evaluate anesthesiology residents: phase I, the critical incident technique. J Clin Anesth. 1997;9:629–36.
Blouin D, Dagnone JD. Performance criteria for emergency medicine residents: a job analysis. CJEM. 2008;10:539–44.
Edwards JC, Currie ML, Wade TP, et al. Surgery resident selection and evaluation. A critical incident study. Evaluation & the health professions. 1993;16:73–86.
Lyss-Lerman P, Teherani A, Aagaard E, et al. What training is needed in the fourth year of medical school? Views of residency program directors. Acad Med. 2009;84:823–9.
Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41:376–82.
Fink A, Kosecoff J, Chassin M, et al. Consensus methods: characteristics and guidelines for use. Am J Public Health. 1984;74:979–83.
Pereira AG, Harrell HE, Weissman A, et al. Important skills for internship and the fourth-year medical school courses to acquire them: a National Survey of internal medicine residents. Acad Med. 2016;91:821–6.
Englander R, Hicks P, Benson B, Pediatric Milestone Project Working G. Pediatrics milestones: a developmental approach to the competencies. J Pediatr. 2010;157:521–2. 522.e1
Frank JR, Snell LS, Cate OT, et al. Competency-based medical education: theory to practice. Med Teach. 2010;32:638–45.
Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach Sep. 2007;29:648–54.
Koenig TW, Parrish SK, Terregino CA, Williams JP, Dunleavy DM, Volsch JM. Core personal competencies important to entering students’ success in medical school: what are they and how could they be assessed early in the admission process? Acad Med. 2013;88(5):603–13.
The authors would like to acknowledge Morgan Valley and Emma Petroff, for their work on this project and James Martindale PhD for his assistance with our statistical analysis. In addition, the authors would like to thank the Office of Undergraduate Medical Education at the University of Colorado School of Medicine for its support of our effort while conducting this research.
The authors received internal support from the Office of Undergraduate Medical Education at the University of Colorado School of Medicine for their time and effort in completing this study.
Availability of data and materials
The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.
Dr. Wolf, MD, is an Associate Professor and Vice Chair for Academic Affairs in the Department of Emergency Medicine at the University of Virginia School of Medicine, Charlottesville, Virginia. Dr. Lockspeiser, MD, is an Associate Professor in Department of Pediatrics and Director of Electives for the University of Colorado School of Medicine, Aurora, Colorado. Dr. Gong, PhD, is Assistant Professor in the Department of Family Medicine and Assistant Director of Evaluation for the University of Colorado School of Medicine, Aurora, Colorado. Dr. Guiton, PhD, is Associate Professor in the Department of Internal Medicine and Director of Evaluation for the University of Colorado School of Medicine, Aurora, Colorado.
Ethics approval and consent to participate
This study was approved by the Colorado Multiple Institutional Review Board. Consent to participate was gained through completion of the research survey instrument.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Wolf, S.J., Lockspeiser, T.M., Gong, J. et al. Identification of foundational non-clinical attributes necessary for successful transition to residency: a modified Delphi study with experienced medical educators. BMC Med Educ 18, 150 (2018) doi:10.1186/s12909-018-1247-6
- Undergraduate medical education
- Residency transition
- Non-clinical attributes
- Graduate medical education