- Open Access
- Open Peer Review
Interprofessional collaboration milestones: advocating for common assessment criteria in graduate medical education
BMC Medical Education volume 15, Article number: 149 (2015)
Milestone-based assessments of resident physicians inform critical decisions regarding resident competence and advancement. Thus, it is essential that milestone evaluations are based upon strong validity evidence and that consistent evaluation criteria are used across residency programs. A common approach to assessment of interprofessional collaboration milestones is particularly important since standardized measures of individual resident competence in interprofessional collaboration have not been established.
We propose that assessments of interprofessional collaboration in graduate medical education meet common criteria, namely, these assessments should: 1) measure competency of an individual resident, 2) occur in the context of an interprofessional team, 3) be ascertained via direct observation of the resident, 4) be performed in a real-world clinical practice setting (such as a hospital ward, outpatient clinic, or operating room). We present the evidence-based rationale for these criteria and cite examples of published assessment instruments that fulfill one or more of the criteria, however further research is needed to ensure fidelity of assessments.
The proposed criteria may assist residency educators as they endeavor to provide robust and consistent assessments of interprofessional collaboration milestones.
In 2013, the Accreditation Council for Graduate Medical Education introduced the Educational Milestone framework for assessment of residents in the Next Accreditation System . Milestones are a sequence of developmentally-based behaviors that residents are expected to demonstrate throughout training . Milestones are used both to assess the competence of individual residents, as well as to support the accreditation of residency programs . Since milestones may potentially inform high-stakes decisions regarding resident advancement and program accreditation, it is critical that milestone assessments are based on strong validity evidence and are performed according to uniform criteria across residency programs.
Each specialty has published a list of milestones, which implies consensus within specialties regarding what must be assessed. There is no consensus, however, regarding how milestone assessments must be performed. Although there is broad acknowledgement that all assessments should be supported by validity evidence according to established frameworks [3, 4], beyond this, it is left up to individual program directors and teaching faculty to decide how to assess each milestone. In the absence of common criteria for assessment, important decisions may be made using subjective and variable measures.
Lack of a common approach to assessment is particularly problematic for milestones related to interprofessional collaboration and teamwork because, unlike medical knowledge and clinical skills, nationally standardized measures of individual resident competence in interprofessional collaboration have not been established . Although numerous teamwork instruments exist, the most extensively studied instruments with the strongest validity evidence, such as the Safety Attitudes Questionnaire [6–10] and the Team Climate Inventory [11–13], assess clinical teams as a whole rather than individual residents on a team. Yet milestones require assessments of individual residents, including his/her personal contribution to the team function and individual ability to collaborate with various professionals.
Despite these unique assessment challenges, the pressure to rigorously measure collaboration among residents and their interprofessional teams is mounting. An increasing number of authorities cite team-based care as crucial to the practice of medicine and vital to health system reform [14–16]. Effective teamwork can enhance patient safety and improve healthcare quality [17, 18], and a common approach to assessment of interprofessional collaboration is critical to ensure that residents are competent in this essential milestone.
Given the importance of team-based care and the complexities of its assessment, residency educators need direction regarding best practices for assessing interprofessional collaboration milestones among their trainees. Residents and programs would benefit from consistency in assessments, especially since these data are used to make significant decisions about individuals and programs. Therefore, we propose common criteria for the assessment of interprofessional collaboration milestones that, in addition to the established standards for validity evidence [3, 4], include 4 components:
Assessment must measure competency of an individual resident
Assessment must be in the context of an interprofessional team (includes professions other than the resident’s profession)
Assessment must be ascertained via direct observation of the resident
Assessment must occur in a real-world clinical practice setting (such as a hospital ward, outpatient clinic, operating room, emergency room)
These criteria were derived from a comprehensive review of the literature regarding Graduate Medical Education (GME) milestones and interprofessional collaboration and teamwork assessment tools. Database search terms and strategy have been previously published , and were extended to encompass all specialties. The evidence-based rationale for each criterion is presented in the subsequent sections along with recommendations for existing assessment instruments that fulfill one or more of the criteria. Residency educators may choose to implement and/or adapt these instruments to assess interprofessional collaboration milestones.
Interprofessional collaboration assessment of the individual resident
Milestone assessment necessitates attestation of individual resident competence. Unfortunately, among published instruments measuring interprofessional collaboration, the instruments with the most robust validity evidence measure collaboration of the team as a whole rather than at the individual level [6, 9, 10, 12, 13, 19–21]. Competencies for individual interprofessional collaboration have been proposed , but these competencies have not been translated into assessment tools. While the overall functioning of the team in clinical practice is of paramount importance, in residency training, milestone assessment requires measurement of the individual resident’s teamwork abilities and contributions to a team. The Ottawa Global Rating Scale [22–25] (OGRS) is one example of a published instrument that is well-suited for evaluation of an individual on a team. The OGRS is a 7-point descriptively-anchored scale of directly observed teamwork behaviors (problem solving, situational awareness, leadership, resource utilization, and communication) that has been validated among multilevel trainees from various specialties within simulated scenarios . An important limitation of the OGRS is that it has not been studied among residents in real-world clinical practice settings. It also focuses on crisis resource management, so its applicability to individuals that are not in crisis mode is debatable. Validity studies of the OGRS within various inpatient and outpatient clinical settings are a necessary next step to optimize its use for milestone assessment.
Collaboration assessment within an interprofessional team
Effective collaboration among professions enhances patient safety and healthcare quality [17, 18]. In GME, it is important to capture how well residents collaborate with individuals from multiple professions within the workplace. Tools such as the Nurse-Physician Collaboration Scale [26, 27], the ICU Nurse-Physician Questionnaire [28, 29], and the Jefferson Scale of Attitudes Toward Nurse-Physician Collaboration [30–32] have been used among residents to assess attitudes toward collaboration of the nurse-resident physician dyad, yet very few instruments measure resident collaboration with other health professionals such as pharmacists, therapists, social workers, clinical assistants, and administrative staff. Incorporating perspectives from the full spectrum of health professionals would significantly enrich milestone assessment data and provide a more complete picture of resident competence.
Residency educators interested in capturing viewpoints from multiple professions may consider utilizing the Team Climate Inventory (TCI) [11–13], which is a 44 to 65-item instrument that evaluates team participation, support, quality, discussion, clarity of objectives, and teamwork climate. Studies of TCI have included diverse non-physician professionals such as advanced practice providers, nurses, therapists, pharmacists, social workers, dieticians, clerical employees, and psychologists [11–13, 19]. Validity evidence for the TCI includes content validity, internal structure, relationships to other variables, and has been correlated with patient outcomes [12, 13, 20, 21]. The TCI does not, unfortunately, assess individual teamwork behaviors but rather examines teamwork climate as a whole in a self-reported manner. The Teamwork Mini-Clinical Evaluation Exercise (T-MEX) [33, 34] is a second instrument that incorporates observations from multiple health professionals, however it has only been studied within undergraduate medical education. Validity studies among residents are required to determine the utility of the T-MEX within GME.
Interprofessional collaboration assessment via direct observation
Many evaluations of interprofessional collaboration rely upon resident opinion and self-assessment [35–37]. This type of subjective assessment, while valuable to obtain an overall picture of teamwork in many settings [9, 10, 12, 13, 20, 21], requires insight into team functioning and personal performance that some trainees may not possess. The ability to accurately self-assess varies among learners , and the extent to which self-assessment skills can be learned and improved is uncertain [39, 40]. To our knowledge, residents’ ability to accurately assess their own milestone performance has not yet been explored, but research among practicing physicians suggests that they may lack insight into their own ability to work in teams . In addition, milestone assessments are designed to measure observable behaviors and skills. For all of these reasons, direct observation is preferable to self-assessment alone for assessing interprofessional collaboration.
Observations may be performed by supervising faculty, resident peers, nurses, pharmacists, allied health professionals, and others who interact with residents in the workplace. Direct observations can also be obtained from trained raters within simulated settings. Several simulation-based instruments have been developed within general surgery and surgical subspecialties. The Observational Teamwork Assessment for Surgery (OTAS) [42–46] measures communication and collaboration within the operating room using direct observation by raters trained to use this scale. Multiple studies have demonstrated validity evidence for the OTAS including content validity, response process, internal structure, relationships to other variables, and patient outcomes [42–49]. The Nontechnical Skills Evaluation Instrument (NOTECHS) [50–52] is similar to the OTAS in that it also assesses teamwork in the operating room using direct observation, but both are more commonly used for assessing teams or subteams rather than individuals. NOTECHS and OTAS have been simultaneously assessed and showed excellent agreement . An additional tool, Non-Technical Skills for Surgeons (NOTSS), was initially validated in simulated operative settings and uses direct observation of individuals on a team . NOTSS has substantial validity evidence and has helped inform the development of the SCORE (Surgical Council for Resident Education) modules for interprofessional education [54, 55]. Although the OTAS, NOTECHS and NOTSS were designed for use in the operating room, many of the content domains in these instruments, such as leadership, teamwork, communication, problem-solving, decision-making, and situation awareness, are applicable to numerous clinical environments including the hospital ward. Tools designed for use in operative simulation must be further adapted into diverse real-world practice settings to assure validity of these observations.
Interprofessional collaboration assessment in a clinical practice setting
The primary purpose for assessing residents using milestones and other measures of competency is to ensure that, by the end of training, residents are ready for unsupervised practice. For this reason, it is essential that interprofessional collaboration is assessed within real-world clinical practice settings . Many assessments of interprofessional collaboration utilize simulated scenarios [22, 36, 57], and while simulation is an excellent environment for residents to prepare for real clinical encounters, performance within simulation may not always mimic performance with a real patient. It is well known that learner behavior is influenced by environmental factors [58, 59] including workload, work intensity, stress and burn-out [60, 61]. As a result, residents who perform well in the controlled environment of an educational simulation may struggle when faced with the numerous challenges inherent in the practice milieu.
The Multi-disciplinary Team Performance Assessment Tool [62, 63] is an example of a direct observation assessment that has been applied among residents in a real-world clinical context, namely an inpatient oncology unit. This instrument focuses on team communication and captures each individual team member’s contributions to the cancer treatment team. Validity evidence includes content validity, response process, internal structure, and relationships to other variables. The Resident Leadership Scale assesses directly observed team leadership skills among internal medicine residents in the setting of an inpatient hospital ward . Established validity evidence includes content, internal structure, and correlations with other measures of teamwork. Both the Multi-disciplinary Team Performance Assessment Tool and the Resident Leadership Scale have each been studied in a single clinical context (inpatient hospital unit), therefore future studies should explore the utility of these instruments in other clinical settings such as outpatient clinics and operating rooms.
A common approach to the assessment of interprofessional collaboration milestones in GME is imperative. Effective teamwork is necessary to achieve safe, high quality patient care, and milestone evaluations inform crucial decisions within residency programs. Therefore, we suggest that measures of interprofessional collaboration milestones use direct observation of an individual resident on an interprofessional team in a real-world clinical practice setting. While many existing instruments meet one or more of these criteria, these instruments have significant limitations. Future studies should focus on adapting current assessments to more fully encompass these important components within graduate medical education clinical environments.
Nasca TJ, Philibert I, Brigham T, Flynn TC. The next GME accreditation system--rationale and benefits. N Engl J Med. 2012;366(11):1051–6.
Caverzagie KJ, Iobst WF, Aagaard EM, Hood S, Chick DA, Kane GC, et al. The internal medicine reporting milestones and the next accreditation system. Ann Intern Med. 2013;158(7):557–9.
Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ. 2003;37(9):830–7.
Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006;119(2):166. e167-116.
Dietz AS, Pronovost PJ, Benson KN, Mendez-Tellez PA, Dwyer C, Wyskiel R, et al. A systematic review of behavioural marker systems in healthcare: what do we know about their attributes, validity and application? BMJ Qual Saf. 2014;23(12):1031–9.
Sexton JB, Helmreich RL, Neilands TB, Rowan K, Vella K, Boyden J, et al. The Safety Attitudes Questionnaire: psychometric properties, benchmarking data, and emerging research. BMC Health Serv Res. 2006;6:44.
Sexton JB, Makary MA, Tersigni AR, Pryor D, Hendrich A, Thomas EJ, et al. Teamwork in the operating room: frontline perspectives among hospitals and operating room personnel. Anesthesiology. 2006;105(5):877–84.
Poley MJ, van der Starre C, van den Bos A, van Dijk M, Tibboel D. Patient safety culture in a Dutch pediatric surgical intensive care unit: An evaluation using the Safety Attitudes Questionnaire. Pediatr Crit Care Me. 2011;12(6):E310–6.
Davenport DL, Henderson WG, Mosca CL, Khuri SF, Mentzer RM, Residents WCS. Risk-adjusted morbidity in teaching hospitals correlates with reported levels of communication and collaboration on surgical teams but not with scale measures of teamwork climate, safety climate, or working conditions. J Am Coll Surg. 2007;205(6):778–84.
Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AHS, Dellinger EP, et al. Changes in safety attitude and relationship to decreased postoperative morbidity and mortality following implementation of a checklist-based surgical safety intervention. BMJ Qual Saf. 2011;20(1):102–7.
Goh TT, Eccles MP, Steen N. Factors predicting team climate, and its relationship with quality of care in general practice. BMC Health Serv Res. 2009;9:138.
Bower P, Campbell S, Bojke C, Sibbald B. Team structure, team climate and the quality of care in primary care: an observational study. Qual Saf Health Care. 2003;12(4):273–9.
Hann M, Bower P, Campbell S, Marshall M, Reeves D. The association between culture, climate and quality of care in primary health care teams. Fam Pract. 2007;24(4):323–9.
Core Principles & Values of Effective Team-Based Health Care. Institute of Medicine. October 2012. [https://www.accp.com/docs/positions/misc/IOM_TeamBased_Care_Principles_Values.pdf. Accessed 3/10/15]
Making the case for interprofessional education. American Medical Association. June 19, 2014. [http://www.ama-assn.org/ama/pub/ama-wire/ama-wire/post/making-case-interprofessional-education. Accessed 3/10/15]
Core Competencies for Interprofessional Collaborative Practice. Interprofessional Education Collaborative. May 2011. [http://www.aacn.nche.edu/education-resources/ipecreport.pdf. Accessed 3/10/15]
Baker DP GS, Beaubien J, Salas E, Barach P. Medical teamwork and patient safety: the evidence-based relation. Rockville, MD: AHRQ Publication No 05–0053 Agency for Healthcare Research and Quality; 2005.
Leonard M, Graham S, Bonacum D. The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care. 2004;13 Suppl 1:i85–90.
Havyer RD, Wingo MT, Comfere NI, Nelson DR, Halvorsen AJ, McDonald FS, et al. Teamwork assessment in internal medicine: a systematic review of validity evidence and outcomes. J Gen Intern Med. 2014;29(6):894–910.
Proudfoot J, Jayasinghe UW, Holton C, Grimm J, Bubner T, Amoroso C, et al. Team climate for innovation: what difference does it make in general practice? Int J Qual Health C. 2007;19(3):164–9.
Gosling AS, Westbrook JI, Braithwaite J. Clinical team functioning and IT innovation: a study of the diffusion of a point-of-care online evidence system. J Am Med Inform Assoc. 2003;10(3):244–51.
Kim J, Neilipovitz D, Cardinal P, Chiu M, Clinch J. A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: The University of Ottawa Critical Care Medicine, High-Fidelity Simulation, and Crisis Resource Management I Study. Crit Care Med. 2006;34(8):2167–74.
Plant JL, van Schaik SM, Sliwka DC, Boscardin CK, O'Sullivan PS. Validation of a self-efficacy instrument and its relationship to performance of crisis resource management skills. Adv Health Sci Educ Theory Pract. 2011;16(5):579–90.
Kim J, Neilipovitz D, Cardinal P, Chiu M. A comparison of global rating scale and checklist scores in the validation of an evaluation tool to assess performance in the resuscitation of critically ill patients during simulated emergencies (abbreviated as "CRM simulator study IB"). Simul Healthc. 2009;4(1):6–16.
Balki M, Cooke ME, Dunington S, Salman A, Goldszmidt E. Unanticipated difficult airway in obstetric patients: development of a new algorithm for formative assessment in high-fidelity simulation. Anesthesiology. 2012;117(4):883–97.
Ushiro R. Nurse-Physician Collaboration Scale: development and psychometric testing. J Adv Nurs. 2009;65(7):1497–508.
Nair DM, Fitzpatrick JJ, McNulty R, Click ER, Glembocki MM. Frequency of nurse-physician collaborative behaviors in an acute care hospital. J Interprof Care. 2012;26(2):115–20.
Boyle DK, Kochinda C. Enhancing collaborative communication of nurse and physician leadership in two intensive care units. J Nurs Adm. 2004;34(2):60–70.
Hamric AB, Blackhall LJ. Nurse-physician perspectives on the care of dying patients in intensive care units: collaboration, moral distress, and ethical climate. Crit Care Med. 2007;35(2):422–9.
Jones TS, Fitzpatrick JJ. CRNA-physician collaboration in anesthesia. Aana J. 2009;77(6):431–6.
Garber JS, Madigan EA, Click ER, Fitzpatrick JJ. Attitudes towards collaboration and servant leadership among nurses, physicians and residents. J Interprof Care. 2009;23(4):331–40.
Ward J, Schaal M, Sullivan J, Bowen ME, Erdmann JB, Hojat M. The Jefferson Scale of Attitudes toward Physician-Nurse Collaboration: A study with undergraduate nursing students. J Interprof Care. 2008;22(4):375–86.
Olupeliyawa A, Balasooriya C, Hughes C, O'Sullivan A. Educational impact of an assessment of medical students' collaboration in health care teams. Med Educ. 2014;48(2):146–56.
Olupeliyawa AM, O'Sullivan AJ, Hughes C, Balasooriya CD. The Teamwork Mini-Clinical Evaluation Exercise (T-MEX): a workplace-based assessment focusing on collaborative competencies in health care. Acad Med. 2014;89(2):359–65.
Morgan PJ, Pittini R, Regehr G, Marrs C, Haley MF. Evaluating teamwork in a simulated obstetric environment. Anesthesiology. 2007;106(5):907–15.
Gettman MT, Pereira CW, Lipsky K, Wilson T, Arnold JJ, Leibovich BC, et al. Use of high fidelity operating room simulation to assess and teach communication, teamwork and laparoscopic skills: initial experience. J Urol. 2009;181(3):1289–96.
Novy D, Hamid B, Driver L, Koyyalagunta L, Ting J, Perez M, et al. Preliminary evaluation of an educational model for promoting positive team attitudes and functioning among pain medicine fellows. Pain Med. 2010;11(6):841–6.
Hodges B, Regehr G, Martin D. Difficulties in recognizing one's own incompetence: novice physicians who are unskilled and unaware of it. Acad Med. 2001;76(10 Suppl):S87–9.
Ward M, Gruppen L, Regehr G. Measuring self-assessment: current state of the art. Adv Health Sci Educ Theory Pract. 2002;7(1):63–80.
Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med. 2005;80(10 Suppl):S46–54.
Arora S, Miskovic D, Hull L, Moorthy K, Aggarwal R, Johannsson H, et al. Self vs expert assessment of technical and non-technical skills in high fidelity simulation. Am J Surg. 2011;202(4):500–6.
Undre S, Healey AN, Darzi A, Vincent CA. Observational assessment of surgical teamwork: a feasibility study. World J Surg. 2006;30(10):1774–83.
Undre S, Sevdalis N, Healey AN, Darzi A, Vincent CA. Observational teamwork assessment for surgery (OTAS): refinement and application in urological surgery. World J Surg. 2007;31(7):1373–81.
Sevdalis N, Lyons M, Healey AN, Undre S, Darzi A, Vincent CA. Observational teamwork assessment for surgery: construct validation with expert versus novice raters. Ann Surg. 2009;249(6):1047–51.
Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N. Observational teamwork assessment for surgery: content validation and tool refinement. J Am Coll Surg. 2011;212(2):234–43. e231-235.
Healey AN, Undre S, Vincent CA. Developing observational measures of performance in surgical teams. Qual Saf Health Care. 2004;13 Suppl 1:i33–40.
Hull L, Arora S, Kassab E, Kneebone R, Sevdalis N. Assessment of stress and teamwork in the operating room: an exploratory study. Am J Surg. 2011;201(1):24–30.
Russ S, Hull L, Rout S, Vincent C, Darzi A, Sevdalis N. Observational teamwork assessment for surgery: feasibility of clinical and nonclinical assessor calibration with short-term training. Ann Surg. 2012;255(4):804–9.
Wetzel CM, George A, Hanna GB, Athanasiou T, Black SA, Kneebone RL, et al. Stress management training for surgeons-a randomized, controlled, intervention study. Ann Surg. 2011;253(3):488–94.
Mishra A, Catchpole K, McCulloch P. The Oxford NOTECHS System: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. Qual Saf Health Care. 2009;18(2):104–8.
Sevdalis N, Davis R, Koutantji M, Undre S, Darzi A, Vincent CA. Reliability of a revised NOTECHS scale for use in surgical teams. Am J Surg. 2008;196(2):184–90.
Steinemann S, Berg B, DiTullio A, Skinner A, Terada K, Anzelon K, et al. Assessing teamwork in the trauma bay: introduction of a modified "NOTECHS" scale for trauma. Am J Surg. 2012;203(1):69–75.
Yule S, Flin R, Maran N, Rowley D, Youngson G, Paterson-Brown S. Surgeons' non-technical skills in the operating room: reliability testing of the NOTSS behavior rating system. World J Surg. 2008;32(4):548–56.
Yule S, Smink D. Competency-based surgical care: Nontechnical skills in surgery. ACS Surgery: Principles and Practice. 2013. http://www.notss.org/literature/. Accessed 7/22/15.
Surgical Council on Resident Education (SCORE). Interpersonal and Communication Skills NOTSS-based modules. http://surgicalcore.org/modules. Accessed 7/22/15.
Teherani A, Chen HC. The next steps in competency-based medical education: milestones, entrustable professional activities and observable practice activities. J Gen Intern Med. 2014;29(8):1090–2.
Undre S, Koutantji M, Sevdalis N, Gautama S, Selvapatt N, Williams S, et al. Multidisciplinary crisis simulations: the way forward for training surgical teams. World J Surg. 2007;31(9):1843–53.
Hafferty FW, Franks R. The hidden curriculum, ethics teaching, and the structure of medical education. Acad Med. 1994;69(11):861–71.
Hafferty FW. Beyond curriculum reform: confronting medicine's hidden curriculum. Acad Med. 1998;73(4):403–7.
West CP, Shanafelt TD. The influence of personal and environmental factors on professionalism in medical education. BMC Med Educ. 2007;7:29.
Dyrbye LN, Massie Jr FS, Eacker A, Harper W, Power D, Durning SJ, et al. Relationship between burnout and professional conduct and attitudes among US medical students. Jama. 2010;304(11):1173–80.
Lamb BW, Wong HW, Vincent C, Green JS, Sevdalis N. Teamwork and team performance in multidisciplinary cancer teams: development and evaluation of an observational assessment tool. BMJ Qual Saf. 2011;20(10):849–56.
Lamb BW, Sevdalis N, Mostafid H, Vincent C, Green JS. Quality improvement in multidisciplinary cancer teams: an investigation of teamwork and clinical decision-making and cross-validation of assessments. Ann Surg Oncol. 2011;18(13):3535–43.
Orlander JDWJ, Lew RA. Development of a tool to assess the team leadership skills of medical residents. Medical Education Online Vol. 2006;11:1–6.
The authors declare that they have no competing interests.
MTW participated in data gathering and wrote/edited the manuscript. RDAH participated in data gathering and assisted in writing/editing the manuscript. NIC participated in data gathering and assisted in writing/editing the manuscript. DRN participated in data gathering and assisted in writing/editing the manuscript. DAR served as mentor for this project, participated study planning/data gathering, and assisted in writing/editing the manuscript. All authors read and approved the final manuscript.