Using a formative simulated patient exercise for curriculum evaluation
BMC Medical Education volume 4, Article number: 8 (2004)
It is not clear that teaching specific history taking, physical examination and patient teaching techniques to medical students results in durable behavioural changes. We used a quasi-experimental design that approximated a randomized double blinded trial to examine whether a Participatory Decision-Making (PDM) educational module taught in a clerkship improves performance on a Simulated Patient Exercise (SPE) in another clerkship, and how this is influenced by the time between training and assessment.
Third year medical students in an internal medicine clerkship were assessed on their use of PDM skills in an SPE conducted in the second week of the clerkship. The rotational structure of the third year clerkships formed a pseudo-randomized design where students had 1) completed the family practice clerkship containing a training module on PDM skills approximately four weeks prior to the SPE, 2) completed the family medicine clerkship and the training module approximately 12 weeks prior to the SPE or 3) had not completed the family medicine clerkship and the PDM training module at the time they were assessed via the SPE.
Based on limited pilot data there were statistically significant differences between students who received PDM training approximately four weeks prior to the SPE and students who received training approximately 12 weeks prior to the SPE. Students who received training 12 weeks prior to the SPE performed better than those who received training four weeks prior to the SPE. In a second comparison students who received training four weeks prior to the SPE performed better than those who did not receive training but the differences narrowly missed statistical significance (P < 0.05).
This pilot study demonstrated the feasibility of a methodology for conducting rigorous curricular evaluations using natural experiments based on the structure of clinical rotations. In addition, it provided preliminary data suggesting targeted educational interventions can result in marked improvements in the clinical skills spontaneously exhibited by physician trainees in a setting different from which the skills were taught.
Medical decision-making is often complex, requiring patients and their physicians to negotiate an acceptable treatment plan. Because of this, interest in physicians' use of participatory decision-making (PDM) has increased . The use of PDM, its impact on patient satisfaction, and the effect on clinical outcomes has been studied in a number of clinical contexts including general medical and surgical care [2, 3], diabetes self-management  and asthma [5, 6].
Decision making within the patient/clinician relationship where the patient participates as an informed partner with their physician in jointly choosing a course of clinical care has been labelled and defined in a number of ways. These include participatory decision making , informed decision making  and shared decision making  to name a few. Over the last several decades, there has been a growing interest in this approach to clinical decision making . While there is little evidence that PDM improves outcomes [2, 4], the use of PDM has been advocated on both ethical  and educational grounds .
Given the interest in PDM and a general consensus that patients should be informed and actively involved in making decisions about their health care, PDM skills need to be incorporated into clinical training programs. Keefe et al  recently described their experience implementing a curricular module addressing PDM and clinical preventive services in a third year family practice clerkship at our institution.
There is little research on the extent students spontaneously use PDM skills in routine patient care activities remote from the educational context in which they were taught. To assess students' unprompted use of skills taught in Keefe et al's curriculum, we have developed a formative, PDM-based Simulated Patient Experience (SPE) administered as part of our third year internal medicine clerkship. The following report describes the study and presents data from its pilot implementation.
Third year internal medicine clerkship students at Michigan State University (MSU) College of Human Medicine (CHM) were offered an opportunity to participate in the study. The pilot data presented in this report were based on students in the second and third rotation of the clerkship during the 2003–2004 academic year.
The family practice clerkship at CHM implemented an educational module to enhance skills in common preventive services and teach students how to inform and involve patients concerning those services. Students are trained to use a variation of a model developed by Braddock and colleagues . Braddock's model includes seven elements of informed decision making:
Discussion of the patient's role in decision making
Discussion of the clinical issue or nature of the decision
Discussion of the alternatives
Discussion of the pros and cons of the alternatives
Discussion of uncertainties associated with the decision
Assessment of the patient's understanding
Exploration of patient preference
Specific decisions are also categorized into "basic", "intermediate", and "complex".
PDM is presented and modelled by faculty, discussed in small groups, and practiced using web-based case simulations. Students must pass an evaluative SPE on shared decision-making and preventive services prior to completion of the clerkship.
During the third-year internal medicine clerkship, students participate in an all day skills workshop conducted during the second week of the eight-week clerkship. The SPE station was one of five small group sessions included in the workshop. Students rotate through the small group sessions and three whole group sessions over the course of the day. The SPE station was intended to provide feedback, and it was made clear to the students that their performance would not impact on their clerkship grade.
The SPE station is structured as an office appointment for a man in his early twenties with a first time seizure. Before meeting the SP, students are given a "patient file" providing the results of a history and physical examination. To ensure adequate knowledge on the part of the student, they are provided with a fact sheet about seizures. Students are given approximately five minutes to review the material. Each student then spent up to 20 minutes discussing treatment options and the implications of the seizure with the SP while being observed by a faculty member. During the last five minutes of the session, the faculty member along with the SP provided verbal feedback to the student. The faculty member also completed a rating form based on key skills taught in the family practice module and other desirable case-specific behaviors. The rating items are listed in Table 1. Performance was rated as "poor/not attempted," "adequate," or "excellent." There were three replications of the case that were necessary to allow all the students in the workshop to complete the station. The authors as well as other faculty rated the students. The faculty raters discussed the rating criteria, however, no formal training was given in how to rate the students. It was felt that the rating criteria were clear and no additional training was needed.
At CHM, the three required primary care clerkships (pediatrics, family practice and internal medicine) are taken during the first three eight-week rotations of the third year of medical school. This rotational structure forms a natural experimental design. The order in which students rotate through the clerkships is arbitrary, and we believe, unlikely to bias the results of this study in any systematic way. For practical purposes, we believe it approximates random assignment.
During the second rotation of the academic year a portion of the students taking the internal medicine clerkship have completed the family practice clerkship and the PDM module during the first rotation, while the rest completed the pediatrics clerkship. During the third rotation, all students have completed the family practice clerkship; however, a portion completed it during the first rotation, approximately 12 weeks prior to the SPE, while the others completed it during the second rotation, or approximately four weeks prior to the exercise.
The rotational structure forms the study design shown in Table 2. The contrast in performance between students in cells 2 and 3 of the design (the students in the second rotation of the internal medicine clerkship) forms a pseudo-randomized trial of the impact of completing the family practice clerkship including the PDM training module as compared with completing the pediatrics clerkship without the PDM module on the students' use of PDM techniques in counseling a patient in the SPE. The contrast between cells 4 and 5 in the design (students completing the third rotation of the internal medicine clerkship) forms a parallel design to that between cells 2 and 3. The comparison, however, assesses the impact of completing the family practice clerkship and the PDM module approximately four weeks versus approximately 12 weeks prior to the SPE. Since the faculty members rating the students did not know a particular student's rotation schedule, and the students were not informed at the time of the SPE of the specifics of the study, the design is also essentially double-blinded.
Two scales were formed from the individual items on the rating sheet. The first was a total score or sum of all 11 items. A second "PDM" score was a sum of the items that specifically reflected the elements of the Braddock model; items 1–4, 6 and 9 from Table 1.
The students' performance was rated as "poor/not attempted," "adequate," or "excellent" on a three point scale with "excellent" rated as a "3". Given the small sample size of this pilot study, we chose to test for statistically significant differences between the groups using non-parametric tests. Both the Kruskal-Wallis test for difference in ranks and the "median test" for differences in medians were used to test for differences among the four groups. When differences were found, a post-hoc analysis was performed. The tests were repeated on subsets of the data to test for differences among pairs of the groups that addressed the key research questions, (cell 2 versus cell 3 and cell 4 versus cell 5 of Table 2) as suggested by Conover .
Twenty-two students from the second rotation of the internal medicine clerkship and 25 students from the third rotation of the internal medicine clerkship agreed to participate in the study. Five of the students from the second rotation of internal medicine had completed the family medicine clerkship during the first rotation of the year and 17 had the pediatrics clerkship. Eighteen of the students from the third rotation of internal medicine had completed the family practice clerkship during their first rotation and seven had completed the family medicine clerkship during their second rotation.
Table 3 presents the number of observations, means, standard deviations and average ranks for the total of the rating items and the subset of the items directly addressing PDM skills for each of the four groups (cells 2 through 5 of Table 2).
The differences among the groups were just statistically significant on the Kruskal-Wallis test (p = 0.05) and clearly statistically significant on the median test (p = 0.008) for the total score. Differences among the groups for the PDM items were not statistically significant (p < 0.05) for either statistical test.
The total score on the rating sheet of the students in the second rotation of the internal medicine clerkship who completed the family practice clerkship during the first rotation was compared with the students from the second rotation of the internal medicine clerkship who completed the pediatric clerkship during the first rotation (cell 2 versus cell 3) using both the Kruskal-Wallis and median test. Both tests were not statistically significant though the difference approached statistical significance for ranks (Kruskal-Wallis test) (p < 0.07).
The students in the third rotation of the internal medicine clerkship who completed the family practice clerkship during the first rotation were compared with those who completed the family medicine rotation during the second rotation (cell 4 versus cell 5) in terms of their total score on the rating sheet. Both the Kruskal-Wallis and median test were statistically significant (p < 0.044 and p < 0.016 respectively).
Unfortunately, the students in two of CHM's six community campuses rotated through the primary care clerkships as a group, e.g. those who took family practice the first rotation all rotated through pediatrics the second rotation and internal medicine the third rotation, which resulted in an uneven distribution of students in the different cells of the design. This further reduced the power of the design that was already limited due to the modest number of students in the clerkship. The students in the second rotation of the internal medicine clerkship who completed the family practice clerkship and the PDM module performed substantially better than the students who had not completed the module. These results were expected though they failed to achieve statistical significance with the very small sample size available for conducting the inferential statistical tests.
We were somewhat surprised that the students from the third rotation of the internal medicine clerkship who completed the family practice clerkship and the PDM module during the first rotation performed substantially better than those who completed the family practice clerkship and the PDM module during the second rotation. One might expect students would be more likely to exhibit skills they were taught four weeks earlier as compared with students who where taught the same skills 12 weeks earlier. One possibility is that the students who took family practice during the first rotation used the PDM skills they were taught during the intervening pediatrics clerkship and this practice enhanced their performance on the SPE case over the performance of students who completed family practice clerkship and the PDM module during the second rotation and did not have the opportunity to practice the skills. It will be interesting to see if this finding replicates in the data that will be collected during the 2004–2005 academic year.
At this point, due to the limited amount of data, we have not formally examined the differences among the groups in terms of the students' performance on the individual rating items. With an additional year of data, these comparisons may well provide some interesting insights into the impact of the PDM module on specific aspects of how the students interact with a patient while working though a treatment decision.
We feel this approach to assessing the impact of educational interventions in clinical training has many potential uses and can provide valuable insights that would be difficult to achieve with more traditional evaluation designs. It also provides a means by which colleagues in different clinical disciplines can collaborate to assess the extent clinical skills taught in one clerkship are exhibited in another clerkship using a fairly rigorous quasi-experimental design.
While this in itself is a valuable and relatively unique contribution, we believe the approach also goes a step further. As noted by George Miller , there are qualitatively different levels of assessing clinical competency. He uses a metaphor of a pyramid where the demonstration of rote knowledge is at the bottom and the actual performance of physicians or physician trainees in real life settings is at the top. While this design does not directly measure what students will do in actual practice, assessing the extent patient counseling skills are used appropriately in a very different context weeks or months after they were taught in our view provides convincing evidence students are likely to use these skills in real clinical situations.
The rotational structure of the third year required clerkships at CHM is quite unique. In fact we know of no other medical school that conducts the required primary care clerkships in the first half of the third year and the required specialty clerkships in the second half of the third year. Having only three clerkships in the rotational structure provided this simplified and elegant design for conducting this study. The six clerkship rotational structure that is used in most medical schools in the USA creates a much more complex situation. We believe however that it is possible to develop a generalized quasi-experimental research design and analysis strategy for conducting this type of study in a traditional six clerkship rotational structure. We are in the process of working out the mathematics and logistical issues of how that might be done and hope to publish this work in the near future.
Sheridan SL, Harris RP, Woolf SH: Shared decision making about screening and chemoprevention. A suggested approach from the U.S. Preventive Services Task Force. Am J Prev Med. 2004, 26 (1): 67-80. 10.1016/j.amepre.2003.09.011.
Braddock CH, Edwards KA, Hasenberg NM, Laidley TL, Levinson W: Informed decision making in the outpatient practice. Time to get back to the basics. JAMA. 1999, 282: 2313-2320.
Gotler RS, Flocke SA, Goodwin MA, Zyzanski SJ, Murray TH, Strange KC: Facilitating participatory decision-making: what happens in real-world community practice?. Med Care. 2000, 38 (12): 1200-1209.
Heisler M, Bouknight RR, Hayward RA, Smith DM, Kerr EA: The relative importance of physician communication, participatory decision making, and patient understanding in diabetes self management. J Gen Intern Med. 2002, 17 (4): 243-252. 10.1046/j.1525-1497.2002.10905.x.
Adams RJ, Smith BJ, Ruffin RE: Patient preferences for autonomy in decision making in asthma management. Thorax. 2001, 56 (2): 126-132. 10.1136/thorax.56.2.126.
Adams RJ, Smith BJ, Ruffin RE: Impact of the physician's participatory style in asthma outcomes and patient satisfaction. Ann Allergy Asthma Immunol. 2001, 86 (3): 263-271.
Kaplan SH, Greenfield S, Ware JE: Assessing the effects of physician-patient interactions on the outcomes of chronic disease. Med Care. 1989, 27 (3 Suppl): S110-27.
O'Conner AM, Fiset V, DeGrasse C, et al: Decision aids for patients considering options affecting cancer outcomes: evidence of efficacy and policy implications. J Natl Cancer Inst Monogr. 1999, 67-80.
Keefe CW, Thompson ME, Noel MM: Medical students, clinical preventive services, and shared decision making. Acad Med. 2002, 77 (11): 1160-1161.
Conover WJ: Practical Nonparametric Statistics. 1971, New York: John Wiley & Sons
Miller GE: The assessment of clinical skills/competence/performance. Acad Med. 1990, 65: S63-S67.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/4/8/prepub
This study was funded in part by grant number 1 D16 HP 00119 01 from the Division of Medicine, Bureau of Health Professions, Health Services and Resources Administration.
DJS developed the research design and directed the project. He also performed the data collection and management, conducted the statistical analyses, rated students in the SPE and wrote a significant portion of the manuscript. HSL acted as the medical consultant for designing the SPE, wrote the SPE case and materials, rated students in the SPE, and wrote a significant portion of the manuscript. CWK helped develop and conduct the PDM training module, and reviewed and helped edit the manuscript. MET directs the family practice clerkship, helped develop and conduct the PDM training module, and reviewed and helped edit the manuscript. MMN helped develop and conduct the PDM training module, rated students in the SPE, and reviewed and helped edit the manuscript. All authors have read and approved the final manuscript.
About this article
Cite this article
Solomon, D.J., Laird-Fick, H.S., Keefe, C.W. et al. Using a formative simulated patient exercise for curriculum evaluation. BMC Med Educ 4, 8 (2004). https://doi.org/10.1186/1472-6920-4-8