Participants and Setting
In May 2003 and May 2004, all categorical PGY-1 (postgraduate year 1) residents from two successive classes in the University of Wisconsin-Madison Internal Medicine Residency program completed an EBM knowledge and skills pre-test (Figure 1). Approximately half of the residents in each class were then randomized by computer-generated random numbers either to a treatment group (12 in 2003, 11 in 2004) where they participated in an EBM workshop during the fall of their PGY-2 year, or to a control group where they did not attend the workshop (14 in 2003, 11 in 2004). Six and 18 months later, in May of their PGY-2 and PGY-3 years, residents again completed EBM knowledge tests. This study protocol was approved by the University of Wisconsin Health Sciences Institutional Review Board. All residents received Institutional Review Board approved study information sheets prior to participation and provided implied consent by completing the EBM tests and surveys.
Formal EBM Training
EBM Workshop
Residents in the treatment group participated in an interactive 4-hour EBM workshop, co-conducted by one faculty member (DF) and one librarian. The workshop, which took place in a computer lab, emphasized four steps in the EBM process: a) developing an answerable question, b) finding the best available evidence, c) evaluating the evidence, and d) applying the evidence to a patient care decision. The case-based teaching covered both therapeutic and diagnostic patient care decisions. Individual and group exercises focused on developing and refining clinical questions in the PICO (Patient, Intervention, Comparison, Outcome) format [12]; individually searching multiple evidence databases; evaluating individual articles for validity; calculating absolute risk reductions (ARR), numbers needed to treat (NNT), relative risk reductions (RRR) and likelihood ratios; and applying evidence to a patient care decision.
The only other formal EBM training that residents received during the study period was an EBM journal club. Each resident presented at journal club once during their PGY-3 year. They met with an advisor to develop a clinical question, and to search for and critically appraise the appropriate evidence. They presented the case and findings to other residents and internal medicine faculty during journal club.
Outcome Measures
EBM Test
Residents were given 40 minutes to complete an EBM test of knowledge and skills that consisted of 25 multiple choice questions covering seven EBM focus areas: a) asking clinical questions, b) searching, c) EBM resources, d) critical appraisal of therapeutic and diagnostic evidence, e) calculating ARR, NNT, and RRR, f) interpreting diagnostic test results, and g) interpreting confidence intervals. The test items required application of EBM concepts. Each item was scored as either correct (1 point) or incorrect (0 points) for a maximum possible score of 25. The same test was used each year; all copies were collected immediately after completion and no feedback was provided to residents.
Our EBM test was developed by the first author (DF) in collaboration with a local EBM expert for a prior project with internal medicine residents [13]. It was revised based on item analysis to include five fewer questions and minor question rewording. Post-hoc construct validity was demonstrated by a one-way analysis of variance comparing the total EBM test scores of 10 first year medical students who had no previous exposure to EBM (M = 10.9, SD = 2.8) with the total EBM test scores of the 48 PGY-1 residents who participated in this study (M = 14.5, SD = 3.6) and 9 EBM experts (M = 22.9, SD = 2.1) who had served as teachers either in a national week-long EBM workshop or in a local EBM faculty development program. EBM experts earned significantly higher EBM test scores than PGY-1 residents (p < 0.001), who in turn, earned significantly higher scores than first year medical students (p = 0.004). Responsiveness of the test was also demonstrated with 16 practicing clinicians during a faculty development fellowship that included EBM training. Mean difference in fellows' pre-test to post-test EBM scores was 5.8 points (95% CI, 4.2, 7.4) [14].
Literature Searches and Resource Usage
In May of each year, at the time of EBM test administration, residents were asked to complete a brief questionnaire self-reporting the number of literature searches they performed during the past week and the number of times they used each of five evidence-based resources in the past month (irrespective of which service they were on): UpToDate, MEDLINE, ACP Journal Club, Cochrane Database of Systematic Reviews (CDSR), and Database of Abstracts of Reviews of Effects (DARE). These resources were freely available to residents at the time the study was initiated. The number of self-reported literature searches was categorized as 0, 1-2, 3-5, 6-10, or more than 10 times per week. The use of evidence-based resources was categorized as 0, 1-2, 3-5, 6-10, or more than 10 times per month.
Self-Efficacy and Workshop Evaluation
Residents who participated in the EBM workshop completed an anonymous 10-item self-assessment of their understanding and ability to practice EBM for therapeutic and diagnostic decision-making. They also responded to a single item evaluating the overall quality of the EBM workshop, rated on a 5-point Likert scale from strongly disagree (1) to strongly agree (5).
Data analysis
Analysis of covariance (ANCOVA) was used to test the effect of treatment group on residents' EBM test scores in the PGY-2 and PGY-3 years, while controlling for baseline (PGY-1) EBM test scores as a covariate. T-tests were conducted to test for group differences in the change of mean test scores between time points. T-tests and Cohen's d were calculated to estimate the changes and effect sizes of EBM test scores for all residents over residency years. The median number of self-reported searches and resources used were computed for each group, and Mann-Whitney U tests were used to assess for differences between groups at each time point, as well as to test for differences between residency years. Descriptive analyses convey residents' self-assessments of knowledge gained following the EBM workshop. Data were analyzed using SPSS for Windows version 16 (SPSS Inc., Chicago, IL, USA).