We utilized resources and personnel from the Center for Competency Development and Assessment (CCDA) at UC to develop three SP cases used for our mock oral exam. These three cases were adapted from patients seen in clinical practice at the UC. Cases were designed with an increasing level of complexity for each level of training. For example, the PGY-2 case was a straightforward case of a right hemisphere stroke (obvious clinical findings), the PGY-3 case was a frontal brain tumor who had presented with a seizure (subtle exam findings), and the PGY-4 case was the most complex with a diagnosis of Devic's disease (complicated history and exam findings indicating multiple lesions in the central nervous system). The final case "scenarios" are included as Appendices 1–3. [see Additional file 1] [see Additional file 2] [see Additional file 3] A global fee was assessed for the utilization of the CCDA facilities and staff time.
Each case was assigned to one of three SPs contracted with the University of Cincinnati College of Medicine for CCDA activities. Each had volunteered for SP duties previously, and 3 of the best performers were selected for this exercise. They were paid on an hourly basis commensurate with CCDA policy. None of the SP participants had a pre-existing neurologic diagnosis.
Each individual SP met with two of the authors (BK, SH) for four training sessions of 1–1.5 hours each, in which the history and exam findings for their case were first taught and then practiced. Videos of physical findings, (such as hemiparesis, spasticity, deep tendon reflex hyper or hypoactivity, visuospatial neglect, facial droop, and appropriate emotional affect) taken from CD-ROM textbooks where available were used as examples to help the SP understand the physical findings to be simulated. Each SP finally underwent a "dress rehearsal" in which another faculty neurologist (DK or AS) examined them and provided feedback on performance; no retraining was considered necessary. All neurology faculty donated their time.
Each resident was asked to sign an Honor Code Agreement prior to the examination, in which they promised not to reveal any details of the exercise to their colleagues either before or after their exam. In this way, cases can be reused in future years.
The mock oral board exercises for UC residents (n = 3 for PGY-2, n = 3 for PGY-3, and n = 2 for PGY-4) were performed at the UC Center for Competency Development and Assessment (CCDA) on June 19, 2002. Three PGY-4 residents from Indiana University were later tested on October 2, 2002, increasing the number of PGY-4 residents tested to five. The exercise was required for trainees at the University of Cincinnati, taking the place of the yearly "mock oral boards". The exercise was voluntary for trainees from Indiana University. Both exam sessions were conducted in one afternoon each, where each of the SPs was examined by 2 or 3 residents from the given PGY level of training specific to their case. All resident cases will hereafter be described by level of training.
Each resident was directly observed by one or two neurology faculty who were present in the room in a manner similar to the ABPN examination. Each resident was scored in a standardized fashion, using an assessment form similar to those used by the ABPN for scoring the live patient hour. Each part of the history and examination is listed on the form, and the resident's performance on each was numerically graded as unsatisfactory (1), borderline (2), or satisfactory (3). Additional general areas of assessment, using the same 1–3 scale included communication skills, respect for the patient, and concern for patient safety. For assessment forms completed by faculty, the maximum possible score was 75 regardless of the year of training. A space was also provided for note-taking and for faculty to provide written comments. The attending physicians were asked to make a subjective final pass/fail determination for each resident exam, using criteria similar to the ABPN examination. A sample assessment sheet is included as Appendix 4 [see Additional file 4].
The standardized patient was given an assessment sheet with case-specific details regarding the history and examination. The SP marked the sheet if the resident obtained the relevant history or exam component. The maximum possible scores varied by case due to differing complexities in history or exam. The SP was given explicit details regarding performance of each part of the history and exam, but was allowed to divulge historical information in a way that felt natural to them. The SP answered several yes/no questions at the end of the assessment form, such as "Did the resident make you feel comfortable" and "would you be comfortable seeing this doctor again". The SP assessment sheet for each case is included with the case scenario in Appendices 1–3. [see Additional file 1] [see Additional file 2] [see Additional file 3]
The PGY-3 case with a brain tumor had an additional twist. It was expected that the resident would ultimately consider brain tumor as a potential cause for the patient's symptoms, and would consider brain imaging as part of the patient's workup. After the discussion of the resident's thought process regarding the case had been completed, the resident was informed that imaging studies had been performed and a brain tumor had been found. The resident was then asked to go back into the room and "break the bad news" to the SP. This is not in keeping with ABPN exam format, but provides for assessment of the resident in end of life care issues. Specific instructions were given to the SP for this portion of the encounter (see Appendix 2) [see Additional file 1]. A separate faculty assessment form and SP checklist was provided for this exercise (see Appendix 5). [see Additional file 5]
After all residents had completed the exam, faculty members and residents were asked to complete a survey about their experience. Not all participants completed this survey, although those who did so answered every question. Each survey question utilized a 1–5 Likert scale (1 = strongly agree, 5 = strongly disagree).
Statistical Analyses
This analysis of data gathered for program and resident evaluation was classified as exempt from review by the Institutional Review Board under category 1 (research on normal education practices) and category 2 (research involving educational tests and survey procedures), as defined in 45 CFR 46.101(b). Summed scores from each physician and SP were calculated for each resident. For residents who had two examiners, inter-observer agreement for summed scores was evaluated using the mean difference in summed scores, the correlation between summed scores using Pearson's correlation coefficient, and Lin's concordance correlation coefficient. The mean of the two scores was used as the score for that resident for that SP. Within each year of training, a mean and standard deviation score for all residents was calculated to measure performance on the test.
For each question on the survey, mean responses were calculated for residents and attendings independently. Differences between faculty and residents on each survey item were tested using the student's t-test. The t-test is relatively robust to non-normality and comparisons were checked using non-parametric statistics (Mann-Whitney U test); t-tests are reported here to be consistent with reporting of means and standard deviations.