- Research article
- Open Access
- Open Peer Review
Development and implementation of a mini-Clinical Evaluation Exercise (mini-CEX) program to assess the clinical competencies of internal medicine residents: from faculty development to curriculum evaluation
© Liao et al; licensee BioMed Central Ltd. 2013
- Received: 26 May 2012
- Accepted: 20 February 2013
- Published: 26 February 2013
The mini-CEX is a valid and reliable method to assess the clinical competencies of trainees. Its data could be useful for educators to redesign curriculum as a process of quality improvement. The aim of this study was to evaluate a mini-CEX assessment program in our internal medicine residency training. We investigated the impact of mini-CEX workshops as a faculty development program on the acquisition of cognitive knowledge and the difference of practice behaviors among faculty members used the mini-CEX to assess residents’ performance at work.
We designed an observational, two-phase study. In the faculty development program, we started a mini-CEX workshop for trainers in 2010, and the short-term outcome of the program was evaluated by comparing the pretest and posttest results to demonstrate the improvement in cognitive knowledge on mini-CEX. From September 2010 to August 2011, we implemented a monthly mini-CEX assessment program in our internal medicine residency training. The data of these mini-CEX assessment forms were collected and analyzed.
In the group of 49 mini-CEX workshop attendees, there was a statistically significant improvement in cognitive knowledge by comparing the pretest and posttest results (67.35 ± 15.25 versus 81.22 ± 10.34, p < 0.001). Among the 863 clinical encounters of mini-CEX, which involved 97 residents and 139 evaluators, 229 (26.5%), 326 (37.8%), and 308 (35.7%) evaluations were completed by the first-year, second-year, and third- year residents separately. We found a statistically significant interaction between level of training and score in dimensions of mini-CEX. The scores in all dimensions measured were better for senior residents. Participation in mini-CEX workshops as a faculty development program strengthened the adherence of trainers to the principles of mini-CEX as a formative assessment in regard to provision of feedback. However, a deficiency in engaging residents’ reflection was found.
Faculty development is a prerequisite to train evaluators in order to implement a successful mini-CEX assessment program. We demonstrated the effectiveness of our mini-CEX workshops in terms of knowledge acquisition and enhancement of giving feedback when the faculty members used the tool. Further programs on providing effective feedback should be conducted to increase the impact of the mini-CEX as a formative assessment.
- Mini clinical evaluation exercise
- Faculty development
Training competent physicians to improve the quality of health care and to meet societal expectations are the primary goals of current medical education . It has been asserted that educators should design a competency-based training program guided by the principles and structures of outcome-based education . In 1998, the Accreditation Council of Graduate Medical Education (ACGME) began an initiative, called the Outcome Project, which fostered residency training with a focus on development and assessment of the six competencies, including medical knowledge, patient care, interpersonal and communication skills, systems-based practice, professionalism, and practice-based learning and improvement . Among the assessment tools targeted on various competencies evolving for years, the direct observation at workplace has played an important role in the process of these educational reforms [4, 5]. In addition to serving as the guidance for feedback on learners’ performance, the results of various assessments could be useful for educators to modify their training programs as a process of quality improvement [6, 7].
The American Board of Internal Medicine (ABIM) developed a quality assessment tool, the mini-Clinical Evaluation Exercise (mini-CEX), which has been applied internationally to assess the clinical competencies of trainees at different subspecialties in a wide variety of clinical settings [8–11]. For each mini-CEX encounter, one evaluator observes the resident conduct a focused interview, physical examination, or therapeutic counseling at a selected workplace, followed by provision of immediate feedback and completion of the rating form. With its feature to provide feedback following a clinical assessment, the mini-CEX also serves as a formative method to guide trainees’ professional development. Previous research on the mini-CEX focused on its validity, reliability, and feasibility to assess the clinical skills of residents, and the educational impact of effective feedback to foster their future learning and improvement [12–14].
Since 2006, the Taiwan Association of Medical Education (TAME) facilitated several medical centers to establish the “General Medicine Training Demonstration Center” with support from the Department of Health, aiming to promote the postgraduate training and provide a holistic patient-centered quality care in Taiwan. An additional project named “The Faculty Development Program for Postgraduate Training” started in 2009 with objectives to equip the clinical teachers with the capabilities of teaching and assessing the ACGME six competencies of trainees . Our hospital was one of the certified institutions to implement both “General Medicine Training Demonstration Center” and “The Faculty Development Program for Postgraduate Training” projects. We conducted a two-hour mini-CEX workshop to train these clinical teachers to acquire the cognitive knowledge on mini-CEX and its clinical applications. In September 2010, we proposed a monthly mini-CEX assessment program to the Department of Internal Medicine and each resident was required to be assessed during the course of training.
The aim of this study was to evaluate a mini-CEX assessment program in our internal medicine residency training. We specifically investigated the impact of mini-CEX workshops as a faculty development program on the acquisition of cognitive knowledge and the difference of practice behaviors among evaluators used the mini-CEX to assess residents’ performance at work.
We designed this observational, two-phase study to evaluate a mini-CEX assessment program in our internal medicine residency training by reporting on the outcomes of mini-CEX workshops as a faculty development program and identifying potential opportunities for curricular improvement. Ethical approval was obtained from the Institutional Review Board of Chang Gung Memorial Hospital.
The faculty development program
We organized a two-hour mini-CEX workshop as one of the assessment curricula within the project of Faculty Development Program of Postgraduate Training. The workshop comprising a pretest, mini-lecture, video-clip rating exercise, small group discussion, and a posttest. A multiple-choice-question test on mini-CEX was designed as the pretest and posttest (Additional file 1). Its contents included the cognitive knowledge on assessment of clinical competence, the application of the mini-CEX as a workplace-based assessment tool, and the principles of providing effective feedback. Improvement in cognitive knowledge was assessed by comparing the results between the pretest and posttest. A total of 49 participants attended the mini-CEX workshops were enrolled.
The monthly mini-CEX assessment program
In September 2010, we started a monthly mini-CEX assessment program for internal medicine residents. A modified ABIM mini-CEX assessment form was adopted, including assessing six dimensions of performance and one global rating on overall clinical competence. The six dimensions of performance were medical interviewing skills, physical examination skills, counseling skills, clinical judgment, humanistic qualities / professionalism, organization and efficiency. We specifically listed a descriptor of each dimension in the mini-CEX assessment form following each rating for reference. Additionally, we divided the space for comment into two categories, resident’s reflection and evaluator’s feedback respectively. As part of the implementation of the mini-CEX program, we provided a grand round didactic lecture titled “ The mini-CEX as a workplace-based assessment” for all the residents and faculty members of internal medicine, addressing the issues of in-training evaluation and the procedural aspects of the mini-CEX as a formative assessment. The faculty members were asked to assess residents’ performance monthly and handed the completed mini-CEX assessment forms to the office of Internal Medicine for statistical analysis.
To exam the quality of feedback, two authors (K.C.L. and S.J.P.) reviewed the assessment forms gathered and classified the contents of feedback and reflection. We found that several contents of the written feedback were not easy to be categorized into the dimensions measured by the mini-CEX. That is, most evaluators provided and documented comments in a more customary way rather than organized feedback according to the order of the competencies measured on the mini-CEX form. Similar findings have been described in literature . In order to demonstrate authenticity and avoid misinterpretation, we decided to categorize the contents of written feedback into five specific aspects after discussion, including history taking, physical examination, counseling and communication skills, attitudes/professionalism, and clinical reasoning. Previous literature also addressed that engagement of resident’s reflection by evaluators was regarded as an indicator of effective feedback . The contents of reflection were conventionally grouped into medical knowledge, clinical skills, and attitude/professionalism (Additional file 2).
The continuous variables were expressed as means and standard deviations and compared using Student’s t-tests. The categorical variables were summarized as proportions and compared using Chi-square tests or Fisher’s exact tests. A p-value of less than 0.05 was taken as statistically significant. A repeated measures analysis of variance (RM-ANOVA) was conducted to evaluate the interaction between the score in dimensions of mini-CEX (within-subjects factor) and level of training (between-subjects factor). All data were analyzed using the Statistical Package for the Social Sciences (SPSS) 17.0 for Windows.(Chicago, IL, USA).
The mini-CEX workshops as a faculty development program
Descriptive statistics for 49 trainers attended the mini-CEX workshops in 2010
Others : Psychiatry, Family Medicine, Emergency Medicine
The monthly mini-CEX assessment program
Descriptive statistics in the mini-CEX assessment forms collected from September 2010 to August 2011
Type of visit
Follow up patient
Focus of evaluation
Mean ± SD
Attending physician (workshop attendees)
Attending physician (non- workshop attendees)
Evaluation of the interaction between level of training and score in dimensions of mini-CEX by repeated measures analysis of variance (RM-ANOVA)
Sum of squares
Level of training
Score in dimensions of mini-CEX
Level of training x Dimensions of mini-CEX
Categories of feedback and reflection in the mini-CEX assessment
Categories of feedback
Counseling / communication skills
Attitudes / Professionalism
Without engagement of resident’s reflection
With engagement of resident’s reflection
Categories of reflection
Attitudes / Professionalism
Comparison of mini-CEX encounters assessed by different evaluators
Engaging resident reflection
8.23 ± 0.66
8.14 ± 0.70
8.24 ± 0.88
8.17 ± 0.73
21.34 ± 12.75
17.64 ± 11.82
12.97 ± 6.96
15.46 ± 10.90
To increase the impact of the mini-CEX as a formative assessment, a faculty development program or rater training program plays an important role. A variety of faculty development programs have been designed and implemented successfully in worldwide institutions [18–20]. It is also the obligations of educators to choose proper assessment methods to evaluate the outcomes of education linking with the objectives and goals of their curricula. In our study, even though we successfully demonstrated the short-term outcome of mini-CEX workshops as a faculty development program by an improvement in cognitive knowledge, we still questioned the sustainability of these gains and effects on trainers’ practice behaviors of teaching at workplaces. Analyzing the collected data of mini-CEX, we found that the evaluators with participation in the faculty development program tended to adhere to the principles of mini-CEX as a formative assessment by providing feedback to trainees more frequently. These results indirectly demonstrated the sustained impact of our faculty development program. However, there was a deficiency in engaging trainee’s reflection found in the group of workshop attendees. According to Archer’s model for effective feedback , which addressed self-monitoring (reflection on action) supported by external feedback and linkage with personal goals (action plan) in a coherent process, the result in our study points out future improvement for faculty development program through the process of curriculum evaluation.
We achieved the goal to assess residents’ performance confronting with patients of different complexities under various circumstances in our monthly mini-CEX assessment program. These mini-CEX results were regarded as components of a continuous learning curve for each resident and offered a clear guidance for both residents and educators to bring about personal and curricular improvement. In the meanwhile, the assessment process provided an interaction between residents and evaluators, either in communicating contents of feedback and future improvement strategies with each other, or in evaluators’ role-modeling approaches to patients during or after the exercise as a teaching method . A previous qualitative study showed that the residents perceived the mini-CEX as anxiety-provoking because of its dual roles of assessment and education . To implement a faculty development and longitudinal assessment program throughout the continuum of education, recruiting both residents and educators could be a solution to relieve mutual tensions and assume a constructive attitude toward clinical assessments .
Our study also demonstrated that the mini-CEX was a feasible tool to evaluate professional development of residents since level of training was significantly attributed to higher scores in all dimensions measured. When used alone, the mini-CEX may be insufficient to reflect trainees’ mastery of each competency. Previous studies showed the validity of the mini-CEX was supported by the strong correlations in scores between the mini-CEX and other assessments [5, 24]. Residents’ performance mature gradually at each level and their learning should be facilitated by means of providing effective feedback at each teachable moment. Compared with the results in feedback analysis, attitude/professionalism was the commonest category of feedback; on the contrary, this was the least frequent category of resident’s reflection in our study. Formative assessment as the mini-CEX has its strength in real-time observation on a trainee’s attitude toward patients during encounters at different workplaces. To maximize the effect of feedback, we must nurture trainee reflection-in-action rather than a trainer-driven, but a two-way process in which trainers provide comments and at the same time encourage trainees to self-reflect on their performance . The cultivation of professionalism in residency program requires organizational approaches since the social and educational environments of training institutions have a profound influence on the hidden curriculum. It has been challenging to demonstrate this competency learnt and internalized by residents even though there are evolving assessment tools specifically focusing on professionalism [25–27]. The mini-CEX was a practical tool for evaluators to observe contexts of professionalism expressed in a resident’s interaction with patients. Promoting a culture of professionalism and providing opportunities for self-reflection must be built into our current program to foster both cognitive and behavioral changes .
Although more of the participants attending our mini-CEX workshops provided recommendations as feedback, those without participation spent more time giving feedback and engaged residents’ reflection more often. Further qualitative studies should be conducted to investigate this contradictory finding, such as the perceptions from evaluators and trainees, specificity of feedback contents, and process of delivering feedback. Provision of feedback from the evaluators to residents after observing resident-patient interaction is a complex and dynamic process and is influenced by many variables, including faculty members’ tensions in balancing positive and negative feedback, their own perceived self-efficacy, their perceptions of the resident’s insight, skill, and potential, the faculty member-resident relationship and contextual factors . Interventions like workshops on provision of effective feedback or modification of the assessment instrument could be considered in future programs to facilitate interactive and high-quality feedback in the mini-CEX [17, 29].
Our study has several limitations. First, this was a single specialty and institution study and the results could not be generalizable to residents in other settings or contexts. The institutional environment is an important determinant for implementing a long-term formative assessment program and developing support of a feedback culture. Second, the retrospective analysis on the written feedback might preclude the actual feedback contents and also lead to underestimation of feedback quantity. An approach of video-taped sessions in the mini-CEX might be an alternative method to overcome this limitation. Nevertheless, videotaping is both time and labor-consuming and even difficult to conduct in a long-term continuous assessment program. Third, when considering workshop effectiveness, the faculty should be evaluated using models that evaluate their ability as raters, such as generalizability analysis or variance components analysis. These statistical models would be helpful to determine how much variance in scores is due to the raters. The multiple-choice-question pretest/ posttest we used in the study could only demonstrate the short-term outcome of mini-CEX workshops in knowledge acquisition. Finally, a substantial amount of missing data in each mini-CEX assessment form would bias the interpretation of our results.
We demonstrated the outcomes of mini-CEX workshops as a faculty development program by providing the evidence of improvement in cognitive knowledge and its sustained impact on the practice behaviors of workshop attendees when they used mini-CEX to assess residents’ performance in terms of providing feedback. Using the data from the mini-CEX assessment program, we also demonstrated the feasibility of this instrument to monitor the professional development of internal medicine residents. Future faculty development program specifically on giving effective feedback should be provided to facilitate residents’ learning as a process of quality improvement.
This study was approved by the ethical review board of Chang Gung Memorial Hospital. The faculty members and residents of internal medicine were informed of the purpose of the study and assured of confidentiality. Written informed consents were obtained from the participants enrolled for publication of this report.
This work was supported in part by an educational grant from the Department of Health in Taiwan (099-VMRPG39004). Part of the results in this project (The monthly mini-CEX assessment program) was drafted in a poster and displayed at the AMEE Annual Meeting in Lyon, France, 25-29 August 2012.
- Frank JR, Danoff D: The CanMEDS initiative: implementing an outcomes-based framework of physician competencies. Med Teach. 2007, 29 (7): 642-647. 10.1080/01421590701746983.View ArticleGoogle Scholar
- Harden RM: Outcome-Based Education: the future is today. Med Teach. 2007, 29 (7): 625-629. 10.1080/01421590701729930.View ArticleGoogle Scholar
- Swing SR: The ACGME outcome project: retrospective and prospective. Med Teach. 2007, 29 (7): 648-654. 10.1080/01421590701392903.View ArticleGoogle Scholar
- Kogan JR, Holmboe ES, Hauer KE: Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009, 302 (12): 1316-1326. 10.1001/jama.2009.1365.View ArticleGoogle Scholar
- Pelgrim EA, Kramer AW, Mokkink HG, van den Elsen L, Grol RP, van der Vleuten CP: In-training assessment using direct observation of single-patient encounters: a literature review. Adv Health Sci Educ Theory Pract. 2011, 16 (1): 131-142. 10.1007/s10459-010-9235-6.View ArticleGoogle Scholar
- Gozu A, Windish DM, Knight AM, Thomas PA, Bass EB, Kern DE: Long term impact of a longitudinal faculty program in curriculum development: A case control study. J Gen Intern Med. 2006, 21: 81-81.Google Scholar
- Gozu A, Windish DM, Knight AM, Thomas PA, Kolodner K, Bass EB, Sisson SD, Kern DE: Long-term follow-up of a 10-month programme in curriculum development for medical educators: a cohort study. Med Educ. 2008, 42 (7): 684-692. 10.1111/j.1365-2923.2008.03090.x.View ArticleGoogle Scholar
- Norcini JJ, Blank LL, Arnold GK, Kimball HR: The mini-CEX (clinical evaluation exercise): a preliminary investigation. Ann Intern Med. 1995, 123 (10): 795-799.View ArticleGoogle Scholar
- Norcini JJ, Blank LL, Duffy FD, Fortna GS: The mini-CEX: a method for assessing clinical skills. Ann Intern Med. 2003, 138 (6): 476-481.View ArticleGoogle Scholar
- Kogan JR, Bellini LM, Shea JA: Implementation of the mini-CEX to evaluate medical students’ clinical skills. Acad Med. 2002, 77 (11): 1156-1157.View ArticleGoogle Scholar
- Weller JM, Jolly B, Misur MP, Merry AF, Jones A, Crossley JG, Pedersen K, Smith K: Mini-clinical evaluation exercise in anaesthesia training. Br J Anaesth. 2009, 102 (5): 633-641. 10.1093/bja/aep055.View ArticleGoogle Scholar
- Cook DA, Beckman TJ, Mandrekar JN, Pankratz VS: Internal structure of mini-CEX scores for internal medicine residents: factor analysis and generalizability. Adv Health Sci Educ Theory Pract. 2010, 15 (5): 633-645. 10.1007/s10459-010-9224-9.View ArticleGoogle Scholar
- Singh T, Sharma M: Mini-clinical examination (CEX) as a tool for formative assessment. Natl Med J India. 2010, 23 (2): 100-102.Google Scholar
- Sidhu RS, Hatala R, Barron S, Broudo M, Pachev G, Page G: Reliability and acceptance of the mini-clinical evaluation exercise as a performance assessment of practicing physicians. Acad Med. 2009, 84 (10 Suppl): S113-S115.View ArticleGoogle Scholar
- Lee FY, Yang YY, Hsu HC, Chuang CL, Lee WS, Chang CC, Huang CC, Chen JW, Cheng HM, Jap TS: Clinical instructors’ perception of a faculty development programme promoting postgraduate year-1 (PGY1) residents’ ACGME six core competencies: a 2-year study. BMJ open. 2011, 1 (2): e000200-10.1136/bmjopen-2011-000200.Google Scholar
- Kogan JR, Conforti LN, Bernabeo EC, Durning SJ, Hauer KE, Holmboe ES: Faculty staff perceptions of feedback to residents after direct observation of clinical skills. Med Educ. 2012, 46 (2): 201-215. 10.1111/j.1365-2923.2011.04137.x.View ArticleGoogle Scholar
- Holmboe ES, Yepes M, Williams F, Huot SJ: Feedback and the mini clinical evaluation exercise. J Gen Intern Med. 2004, 19 (5 Pt 2): 558-561.View ArticleGoogle Scholar
- Cole KA, Barker LR, Kolodner K, Williamson P, Wright SM, Kern DE: Faculty development in teaching skills: an intensive longitudinal model. Acad Med. 2004, 79 (5): 469-480. 10.1097/00001888-200405000-00019.View ArticleGoogle Scholar
- Steinert Y, Mann K, Centeno A, Dolmans D, Spencer J, Gelula M, Prideaux D: A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach. 2006, 28 (6): 497-526. 10.1080/01421590600902976.View ArticleGoogle Scholar
- Skeff KM, Stratos GA, Bergen MR, Regula DP: A pilot study of faculty development for basic science teachers. Acad Med. 1998, 73 (6): 701-704. 10.1097/00001888-199806000-00018.View ArticleGoogle Scholar
- Archer JC: State of the science in health professional education: effective feedback. Med Educ. 2010, 44 (1): 101-108. 10.1111/j.1365-2923.2009.03546.x.View ArticleGoogle Scholar
- Weller JM, Jones A, Merry AF, Jolly B, Saunders D: Investigation of trainee and specialist reactions to the mini-Clinical Evaluation Exercise in anaesthesia: implications for implementation. Br J Anaesth. 2009, 103 (4): 524-530. 10.1093/bja/aep211.View ArticleGoogle Scholar
- Malhotra S, Hatala R, Courneya CA: Internal medicine residents’ perceptions of the Mini-Clinical Evaluation Exercise. Med Teach. 2008, 30 (4): 414-419. 10.1080/01421590801946962.View ArticleGoogle Scholar
- Durning SJ, Cation LJ, Markert RJ, Pangaro LN: Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Acad Med. 2002, 77 (9): 900-904. 10.1097/00001888-200209000-00020.View ArticleGoogle Scholar
- Epstein RM, Hundert EM: Defining and assessing professional competence. JAMA. 2002, 287 (2): 226-235. 10.1001/jama.287.2.226.View ArticleGoogle Scholar
- Branch WT: Use of critical incident reports in medical education A perspective. J general Intern Med. 2005, 20 (11): 1063-1067. 10.1111/j.1525-1497.2005.00231.x.View ArticleGoogle Scholar
- Cruess R, McIlroy JH, Cruess S, Ginsburg S, Steinert Y: The Professionalism Mini-evaluation Exercise: a preliminary investigation. Acad Med. 2006, 81 (10 Suppl): S74-S78.View ArticleGoogle Scholar
- Stern DT, Papadakis M: The developing physician–becoming a professional. N Engl J Med. 2006, 355 (17): 1794-1799. 10.1056/NEJMra054783.View ArticleGoogle Scholar
- Fernando N, Cleland J, McKenzie H, Cassar K: Identifying the factors that determine feedback given to undergraduate medical students following formative mini-CEX assessments. Med Educ. 2008, 42 (1): 89-95.Google Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/13/31/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.