Evidence-based medicine training during residency: a randomized controlled trial of efficacy

Competent clinical decision-making is a complex and critical process[1] and leaders in graduate medical education have long sought to hone residents' decision-making skills [2]. The introduction of evidence-based medicine (EBM) in 1992[3] and its subsequent inclusion by the Accreditation Council for Graduate Medical Education (ACGME) as a core component of practice-based learning and improvement[4] served as a catalyst for residency programs to incorporate evidence-based practice concepts into their curricula [5].

After two decades and a multitude of systematic reviews, non-randomized controlled studies, and pre- and post-intervention studies, the impact of formal EBM training on resident knowledge, skills, attitudes, and behavior remains unclear [6]. Challenges remain for translating EBM knowledge into clinical practice [7, 8], and barriers to successful implementation of EBM skills have not yet been resolved [9]. Determination of the best methods for teaching clinical decision-making has been made difficult by the lack of well-validated evaluation tools and the absence of randomized controlled trials evaluating the impact of EBM educational interventions [10, 11]. This paper reports results from a long-term, randomized controlled trial designed to test the hypothesis that participation in a brief interactive EBM workshop leads to increases in residents' EBM knowledge, literature searching, and self-reported use of evidence-based resources. Improvements in EBM competency are assessed across residency training in an effort to elucidate how best to prepare residents for effective clinical decision-making.

This randomized controlled trial assessed the impact of an EBM workshop on residents' EBM knowledge and skills within the context of residency training. Unlike previous short-term randomized [15] and non-randomized trials [12, 16, 17], the results failed to indicate an independent effect of our 4-hr interactive EBM workshop on residents' EBM test performance. Although residents rated the EBM workshop very positively and reported perceived gains in EBM knowledge, their perceptions were not associated with increases in EBM skills.

Several limitations could have prevented us from documenting an effect of our EBM workshop on residents' EBM test scores. First, although the difference in change in PGY-1 to PGY-3 test scores between the treatment and control groups was small (0.41), the confidence intervals around this change included a possible medium effect size, and the small sample size may have limited our ability to detect differences that might be considered clinically or educationally significant. Second, knowledge is a dynamic construct that changes over time [18–20]. Our assessments were conducted 6 and 18 months following the intervention, so short-term increases in EBM knowledge could have been missed, or masked by shared learning if workshop participants effectively translated their EBM knowledge to non-workshop participants through journal club, critical reading projects, and close interaction during clinical care. Third, although we did not assess residents' prior EBM training in medical school, we noted that our residents' baseline EBM test scores (58%) were actually much closer to the post-test scores reported by Smith et al.[21] (58-64%) and Fritsche et al.[16] (65%) than they were to the baseline scores reported in those studies (~40%). Although different tests were used in each study, it is possible that our residents' initially higher levels of EBM competency may have reduced their opportunities to demonstrate improvement.

Our results did, however, indicate significant improvement in residents' EBM knowledge, numbers of literature searches, and use of secondary evidence resources over the course of residency training. In addition, PGY-2 residents who participated in the workshop exhibited an increased use of CDSR and DARE compared with the control group. This is consistent with other randomized controlled trials demonstrating an increase in residents' use of evidence-based resources following EBM training [15] particularly when training focused on using secondary evidence resources instead of MEDLINE.

This study is one of only a few randomized controlled trials evaluating the impact of an EBM curriculum over the course of residency training. The fact that our results are inconsistent with some previous randomized controlled trials, for example, Kim et al. (2008), highlights the need for more rigorous, long-term trials of EBM education to provide guidance for implementing effective EBM programs. More intense doses of EBM education may be required to produce measurable improvement in residents' EBM skills [22].

Research also suggests that integration of EBM training into clinical practice may provide better results in improving EBM skills [10], although this has not been confirmed in randomized controlled trials. Furthermore, the true goal of EBM education is to improve resident behavior and patient outcomes. This will require a combination of educational interventions to teach basic skills, real time evidence access and incorporation of evidence into clinical care. Our data were collected from 2003-2006 and the change in the quality and availability of secondary resources since that time may require different methods of teaching and integration of EBM practices into clinical care.

Based on these results, our residency program has built a broader-based EBM curriculum that enhances faculty competencies, stimulates resident scholarship, and guides residents in their application of EBM at the point of care. We now provide separate interactive workshops for our PGY-1 and PGY-2 residents to allow time for reinforcement and consolidation of EBM skills. We are also integrating our EBM training into clinical care through piloting a new evidence-based "educational prescription" tool designed to walk residents through the steps of EBM while answering clinical questions at the point of care [23]. Validation of our evidence-based prescription tool as part of a multi-institutional trial will allow us to assess the generalizability of different types of EBM educational interventions and avoid issues of contamination that are common in single center studies [24].

This randomized controlled trial was not able to detect a significant effect of a 4-hour EBM workshop on residents' EBM knowledge and skills, although residents did demonstrate an increase in EBM skills, literature searching, and resource usage over the course of residency training. Future multi-institutional randomized controlled studies are warranted to determine the best methods for improving residents' EBM skills as well as for enhancing their abilities to apply evidence during clinical practice.