This article has Open Peer Review reports available.
Using a formative simulated patient exercise for curriculum evaluation
© Solomon et al; licensee BioMed Central Ltd. 2004
Received: 01 January 2004
Accepted: 12 May 2004
Published: 12 May 2004
It is not clear that teaching specific history taking, physical examination and patient teaching techniques to medical students results in durable behavioural changes. We used a quasi-experimental design that approximated a randomized double blinded trial to examine whether a Participatory Decision-Making (PDM) educational module taught in a clerkship improves performance on a Simulated Patient Exercise (SPE) in another clerkship, and how this is influenced by the time between training and assessment.
Third year medical students in an internal medicine clerkship were assessed on their use of PDM skills in an SPE conducted in the second week of the clerkship. The rotational structure of the third year clerkships formed a pseudo-randomized design where students had 1) completed the family practice clerkship containing a training module on PDM skills approximately four weeks prior to the SPE, 2) completed the family medicine clerkship and the training module approximately 12 weeks prior to the SPE or 3) had not completed the family medicine clerkship and the PDM training module at the time they were assessed via the SPE.
Based on limited pilot data there were statistically significant differences between students who received PDM training approximately four weeks prior to the SPE and students who received training approximately 12 weeks prior to the SPE. Students who received training 12 weeks prior to the SPE performed better than those who received training four weeks prior to the SPE. In a second comparison students who received training four weeks prior to the SPE performed better than those who did not receive training but the differences narrowly missed statistical significance (P < 0.05).
This pilot study demonstrated the feasibility of a methodology for conducting rigorous curricular evaluations using natural experiments based on the structure of clinical rotations. In addition, it provided preliminary data suggesting targeted educational interventions can result in marked improvements in the clinical skills spontaneously exhibited by physician trainees in a setting different from which the skills were taught.
Medical decision-making is often complex, requiring patients and their physicians to negotiate an acceptable treatment plan. Because of this, interest in physicians' use of participatory decision-making (PDM) has increased . The use of PDM, its impact on patient satisfaction, and the effect on clinical outcomes has been studied in a number of clinical contexts including general medical and surgical care [2, 3], diabetes self-management  and asthma [5, 6].
Decision making within the patient/clinician relationship where the patient participates as an informed partner with their physician in jointly choosing a course of clinical care has been labelled and defined in a number of ways. These include participatory decision making , informed decision making  and shared decision making  to name a few. Over the last several decades, there has been a growing interest in this approach to clinical decision making . While there is little evidence that PDM improves outcomes [2, 4], the use of PDM has been advocated on both ethical  and educational grounds .
Given the interest in PDM and a general consensus that patients should be informed and actively involved in making decisions about their health care, PDM skills need to be incorporated into clinical training programs. Keefe et al  recently described their experience implementing a curricular module addressing PDM and clinical preventive services in a third year family practice clerkship at our institution.
There is little research on the extent students spontaneously use PDM skills in routine patient care activities remote from the educational context in which they were taught. To assess students' unprompted use of skills taught in Keefe et al's curriculum, we have developed a formative, PDM-based Simulated Patient Experience (SPE) administered as part of our third year internal medicine clerkship. The following report describes the study and presents data from its pilot implementation.
Third year internal medicine clerkship students at Michigan State University (MSU) College of Human Medicine (CHM) were offered an opportunity to participate in the study. The pilot data presented in this report were based on students in the second and third rotation of the clerkship during the 2003–2004 academic year.
Discussion of the patient's role in decision making
Discussion of the clinical issue or nature of the decision
Discussion of the alternatives
Discussion of the pros and cons of the alternatives
Discussion of uncertainties associated with the decision
Assessment of the patient's understanding
Exploration of patient preference
Specific decisions are also categorized into "basic", "intermediate", and "complex".
PDM is presented and modelled by faculty, discussed in small groups, and practiced using web-based case simulations. Students must pass an evaluative SPE on shared decision-making and preventive services prior to completion of the clerkship.
During the third-year internal medicine clerkship, students participate in an all day skills workshop conducted during the second week of the eight-week clerkship. The SPE station was one of five small group sessions included in the workshop. Students rotate through the small group sessions and three whole group sessions over the course of the day. The SPE station was intended to provide feedback, and it was made clear to the students that their performance would not impact on their clerkship grade.
Rating scale used by faculty in evaluating the students
Appropriately greets patient
Establishes the purpose of the encounter
Clarifies roles of the patient and physician in decision making and gains the patient's permission to continue the discussion
Presents the issues in terms the patient can comprehend
Reviews available test results
Discusses alternatives (treatment versus no treatment) and their pros and cons
Discusses safety issues related to new seizures
Counsels re: driving restriction under Michigan state law
Explores patient's preferences
Ensures patient understands the issues
Responds appropriately to affect
At CHM, the three required primary care clerkships (pediatrics, family practice and internal medicine) are taken during the first three eight-week rotations of the third year of medical school. This rotational structure forms a natural experimental design. The order in which students rotate through the clerkships is arbitrary, and we believe, unlikely to bias the results of this study in any systematic way. For practical purposes, we believe it approximates random assignment.
During the second rotation of the academic year a portion of the students taking the internal medicine clerkship have completed the family practice clerkship and the PDM module during the first rotation, while the rest completed the pediatrics clerkship. During the third rotation, all students have completed the family practice clerkship; however, a portion completed it during the first rotation, approximately 12 weeks prior to the SPE, while the others completed it during the second rotation, or approximately four weeks prior to the exercise.
Study design formed by the clerkship rotational schedule*
1. Did not complete PDM Module
2. Did not Complete PDM Module
4. Completed the PDM module during the first rotation
3. Completed the PDM module during the first rotation
5. Completed the PDM module during the second rotation
Two scales were formed from the individual items on the rating sheet. The first was a total score or sum of all 11 items. A second "PDM" score was a sum of the items that specifically reflected the elements of the Braddock model; items 1–4, 6 and 9 from Table 1.
The students' performance was rated as "poor/not attempted," "adequate," or "excellent" on a three point scale with "excellent" rated as a "3". Given the small sample size of this pilot study, we chose to test for statistically significant differences between the groups using non-parametric tests. Both the Kruskal-Wallis test for difference in ranks and the "median test" for differences in medians were used to test for differences among the four groups. When differences were found, a post-hoc analysis was performed. The tests were repeated on subsets of the data to test for differences among pairs of the groups that addressed the key research questions, (cell 2 versus cell 3 and cell 4 versus cell 5 of Table 2) as suggested by Conover .
Twenty-two students from the second rotation of the internal medicine clerkship and 25 students from the third rotation of the internal medicine clerkship agreed to participate in the study. Five of the students from the second rotation of internal medicine had completed the family medicine clerkship during the first rotation of the year and 17 had the pediatrics clerkship. Eighteen of the students from the third rotation of internal medicine had completed the family practice clerkship during their first rotation and seven had completed the family medicine clerkship during their second rotation.
Student ratings by groups
M. Rank ‡
The differences among the groups were just statistically significant on the Kruskal-Wallis test (p = 0.05) and clearly statistically significant on the median test (p = 0.008) for the total score. Differences among the groups for the PDM items were not statistically significant (p < 0.05) for either statistical test.
The total score on the rating sheet of the students in the second rotation of the internal medicine clerkship who completed the family practice clerkship during the first rotation was compared with the students from the second rotation of the internal medicine clerkship who completed the pediatric clerkship during the first rotation (cell 2 versus cell 3) using both the Kruskal-Wallis and median test. Both tests were not statistically significant though the difference approached statistical significance for ranks (Kruskal-Wallis test) (p < 0.07).
The students in the third rotation of the internal medicine clerkship who completed the family practice clerkship during the first rotation were compared with those who completed the family medicine rotation during the second rotation (cell 4 versus cell 5) in terms of their total score on the rating sheet. Both the Kruskal-Wallis and median test were statistically significant (p < 0.044 and p < 0.016 respectively).
Unfortunately, the students in two of CHM's six community campuses rotated through the primary care clerkships as a group, e.g. those who took family practice the first rotation all rotated through pediatrics the second rotation and internal medicine the third rotation, which resulted in an uneven distribution of students in the different cells of the design. This further reduced the power of the design that was already limited due to the modest number of students in the clerkship. The students in the second rotation of the internal medicine clerkship who completed the family practice clerkship and the PDM module performed substantially better than the students who had not completed the module. These results were expected though they failed to achieve statistical significance with the very small sample size available for conducting the inferential statistical tests.
We were somewhat surprised that the students from the third rotation of the internal medicine clerkship who completed the family practice clerkship and the PDM module during the first rotation performed substantially better than those who completed the family practice clerkship and the PDM module during the second rotation. One might expect students would be more likely to exhibit skills they were taught four weeks earlier as compared with students who where taught the same skills 12 weeks earlier. One possibility is that the students who took family practice during the first rotation used the PDM skills they were taught during the intervening pediatrics clerkship and this practice enhanced their performance on the SPE case over the performance of students who completed family practice clerkship and the PDM module during the second rotation and did not have the opportunity to practice the skills. It will be interesting to see if this finding replicates in the data that will be collected during the 2004–2005 academic year.
At this point, due to the limited amount of data, we have not formally examined the differences among the groups in terms of the students' performance on the individual rating items. With an additional year of data, these comparisons may well provide some interesting insights into the impact of the PDM module on specific aspects of how the students interact with a patient while working though a treatment decision.
We feel this approach to assessing the impact of educational interventions in clinical training has many potential uses and can provide valuable insights that would be difficult to achieve with more traditional evaluation designs. It also provides a means by which colleagues in different clinical disciplines can collaborate to assess the extent clinical skills taught in one clerkship are exhibited in another clerkship using a fairly rigorous quasi-experimental design.
While this in itself is a valuable and relatively unique contribution, we believe the approach also goes a step further. As noted by George Miller , there are qualitatively different levels of assessing clinical competency. He uses a metaphor of a pyramid where the demonstration of rote knowledge is at the bottom and the actual performance of physicians or physician trainees in real life settings is at the top. While this design does not directly measure what students will do in actual practice, assessing the extent patient counseling skills are used appropriately in a very different context weeks or months after they were taught in our view provides convincing evidence students are likely to use these skills in real clinical situations.
The rotational structure of the third year required clerkships at CHM is quite unique. In fact we know of no other medical school that conducts the required primary care clerkships in the first half of the third year and the required specialty clerkships in the second half of the third year. Having only three clerkships in the rotational structure provided this simplified and elegant design for conducting this study. The six clerkship rotational structure that is used in most medical schools in the USA creates a much more complex situation. We believe however that it is possible to develop a generalized quasi-experimental research design and analysis strategy for conducting this type of study in a traditional six clerkship rotational structure. We are in the process of working out the mathematics and logistical issues of how that might be done and hope to publish this work in the near future.
This study was funded in part by grant number 1 D16 HP 00119 01 from the Division of Medicine, Bureau of Health Professions, Health Services and Resources Administration.
- Sheridan SL, Harris RP, Woolf SH: Shared decision making about screening and chemoprevention. A suggested approach from the U.S. Preventive Services Task Force. Am J Prev Med. 2004, 26 (1): 67-80. 10.1016/j.amepre.2003.09.011.View ArticleGoogle Scholar
- Braddock CH, Edwards KA, Hasenberg NM, Laidley TL, Levinson W: Informed decision making in the outpatient practice. Time to get back to the basics. JAMA. 1999, 282: 2313-2320.View ArticleGoogle Scholar
- Gotler RS, Flocke SA, Goodwin MA, Zyzanski SJ, Murray TH, Strange KC: Facilitating participatory decision-making: what happens in real-world community practice?. Med Care. 2000, 38 (12): 1200-1209.View ArticleGoogle Scholar
- Heisler M, Bouknight RR, Hayward RA, Smith DM, Kerr EA: The relative importance of physician communication, participatory decision making, and patient understanding in diabetes self management. J Gen Intern Med. 2002, 17 (4): 243-252. 10.1046/j.1525-1497.2002.10905.x.View ArticleGoogle Scholar
- Adams RJ, Smith BJ, Ruffin RE: Patient preferences for autonomy in decision making in asthma management. Thorax. 2001, 56 (2): 126-132. 10.1136/thorax.56.2.126.View ArticleGoogle Scholar
- Adams RJ, Smith BJ, Ruffin RE: Impact of the physician's participatory style in asthma outcomes and patient satisfaction. Ann Allergy Asthma Immunol. 2001, 86 (3): 263-271.View ArticleGoogle Scholar
- Kaplan SH, Greenfield S, Ware JE: Assessing the effects of physician-patient interactions on the outcomes of chronic disease. Med Care. 1989, 27 (3 Suppl): S110-27.View ArticleGoogle Scholar
- O'Conner AM, Fiset V, DeGrasse C, et al: Decision aids for patients considering options affecting cancer outcomes: evidence of efficacy and policy implications. J Natl Cancer Inst Monogr. 1999, 67-80.Google Scholar
- Keefe CW, Thompson ME, Noel MM: Medical students, clinical preventive services, and shared decision making. Acad Med. 2002, 77 (11): 1160-1161.View ArticleGoogle Scholar
- Conover WJ: Practical Nonparametric Statistics. 1971, New York: John Wiley & SonsGoogle Scholar
- Miller GE: The assessment of clinical skills/competence/performance. Acad Med. 1990, 65: S63-S67.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/4/8/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article: verbatim copying and redistribution of this article are permitted in all media for any purpose, provided this notice is preserved along with the article's original URL.