This article has Open Peer Review reports available.
Description and evaluation of an EBM curriculum using a block rotation
© Thom et al; licensee BioMed Central Ltd. 2004
Received: 26 April 2004
Accepted: 11 October 2004
Published: 11 October 2004
While previous authors have emphasized the importance of integrating and reinforcing evidence-based medicine (EBM) skills in residency, there are few published examples of such curricula. We designed an EBM curriculum to train family practice interns in essential EBM skills for information mastery using clinical questions generated by the family practice inpatient service. We sought to evaluate the impact of this curriculum on interns, residents, and faculty.
Interns (n = 13) were asked to self-assess their level of confidence in basic EBM skills before and after their 2-week EBM rotation. Residents (n = 21) and faculty (n = 12) were asked to assess how often the answers provided by the EBM intern to the inpatient service changed medical care. In addition, residents were asked to report how often they used their EBM skills and how often EBM concepts and tools were used in teaching by senior residents and faculty. Faculty were asked if the EBM curriculum had increased their use of EBM in practice and in teaching.
Interns significantly increased their confidence over the course of the rotation. Residents and faculty felt that the answers provided by the EBM intern provided useful information and led to changes in patient care. Faculty reported incorporating EBM into their teaching (92%) and practice (75%). Residents reported applying the EBM skills they learned to patient care (86%) and that these skills were reinforced in the teaching they received outside of the rotation (81%). All residents and 11 of 12 faculty felt that the EBM curriculum had improved patient care.
To our knowledge, this is the first published EBM curriculum using an individual block rotation format. As such, it may provide an alternative model for teaching and incorporating EBM into a residency program.
Evidence-based medicine (EBM) strives to provide a systematic approach to integrating the best research evidence with clinician expertise and patient preferences to provide better patient care . While the potential for answering clinical questions using online resources is high , a recent study found resident physicians rarely used web-based or other evidence-based sources to answer clinical questions, preferring instead another person or a pocket reference . Many authors have argued for the importance of teaching EBM skills during residency training, and several have cited evidence to support the desirability of integrating EBM training with other aspects of the residency program [4–8]. Such a curriculum presents several challenges, including finding sufficient time to teach EBM skills to interns and developing ways to integrate and reinforce EBM among residents and faculty.
While there have been several studies of residency EBM curricula [4, 9–11], none, to our knowledge, has operated in the framework of an intern block rotation. In this paper we describe an EBM curriculum based on an individual block rotation and designed to integrate and reinforce EBM skills throughout the residency program; we also report on its evaluation by interns, residents and faculty.
The UCSF family practice residency program is based at San Francisco General Hospital (SFGH), a large county hospital serving the urban poor. The program runs a busy family practice inpatient service (at SFGH) as well as the Family Health Center, an outpatient clinic that includes both continuity practices and acute care services. Within this setting, we formulated 3 primary goals for our EBM curriculum: (1) to teach interns basic EBM concepts and skills; (2) to disseminate and reinforce EBM skills to second and third year residents and faculty; and (3) to apply EBM to the care of patients. This study was approved by the UCSF Committee on Human Research.
The EBM curriculum for UCSF family practice residents began, in its current form, summer of 2001. The core of the curriculum is a 2-week, individual EBM rotation for interns. In contrast to the usual format of multiple lectures or learning modules scattered throughout residency, the two-week individual block format provides for a concentrated time in which to learn EBM. It also allows tailoring of the rotation to fit the residents' backgrounds and interests. During each of the two weeks, residents have 3 half-day clinics, with the remainder of the time available for EBM. The EBM portion of the rotation fulfills the ACGME requirements for resident research and scholarly activity.
The rotation begins with a meeting with the EBM faculty Director (DT) during which the structure and goals of the rotation are communicated, the intern's knowledge and experience related to EBM are assessed by review of his or her experience and by a pre-rotation test of EBM skills and knowledge (described below). Interns receive reference materials [1, 12] and a binder including detailed instructions for the rotation and key articles. The intern, together with the rotation faculty director (DT) and the medical librarian co-director (JH), attends family practice inpatient service rounds to obtain one or two clinical questions directly bearing on the care of one or two patients. With faculty guidance, questions are then formatted into the standard EBM structure identified by the acronym PICO (for patient/problem, intervention, comparison, and outcome). The entire process generally takes 5 to 10 minutes and includes modeling of how to formulate an appropriate 'answerable' question by senior residents and the inpatient attending faculty. The question(s) generated are then used by the librarian Co-director as material for a tutorial on developing search strategies and using high-quality web-based EBM resources. The emphasis of the tutorial is to introduce the intern to the concept of information mastery . An initial assessment of the intern's searching experiences and use of EBM resources helps to tailor the tutorial session. In the tutorial, the intern is introduced to essential EBM concepts, including clinical question development, levels of evidence search strategies, and appraisal techniques , and learns to translate the 'answerable' clinical question into a 'searchable' one. Based on the type of clinical question, the intern reviews the possible levels of evidence and study designs and formulates a valid search strategy. As the intern searches for the evidence, the medical librarian provides input into alternative search strategies and information resources. The results of this and any subsequent searches are discussed with the faculty EBM Director and a 1-page response is developed in the form of a critically appraised topic or CAT . Each CAT is structured to include the 'clinical bottom line' answer to the question; the clinical scenario that generated the questions; a summary of the evidence; a critical appraisal of the evidence; and citations. Results are presented to and discussed with the inpatient team. CATs are stored at the EBM website for future reference by residents and faculty .
During the first week, the intern completes a web-based EBM tutorial  which covers critical evaluation of articles about diagnosis, therapy, prognosis, meta-analysis and decision making, and how to ask clinical questions. In the second week the intern receives another 1 or 2 clinical questions from the inpatient service which are searched and answered as above. The intern also prepares and presents a journal club, which is attended by faculty, residents and medical students. A research article is selected by the intern (with guidance from the EBM Director), which could potentially change a primary care practice around a clinical question. The article is critically reviewed using an EBM approach [1, 12], presented and discussed by the group.
During the initial 3 months of the curriculum, interns completed a written test of their EBM skills and knowledge before and immediately following the rotation. The tests were adapted, with permission, from a similar instrument developed by Sean Schafer, MD and Katie Ramos, PhD at the University of California Fresno Family Practice Residency Program . Scores improved post-rotation in all 3 areas tested: EBM terms and concepts 81% to 97%; quantitative skills 51% to 80%; question formulation and searching 71% to 92%, with the total score increasing from 63% to 87%.
As others have noted, progress in the use of EBM depends on the availability of information support services to resident and faculty at the point of patient care . In conjunction with our EBM rotation we have also improved support for residents and faculty searching for evidence-based answers to clinical questions, through the development of our EBM-filtered information support website , access to a librarian information specialist (JH), and provision of PDAs to our interns who did not have their own.
We estimate that training 13 interns per year requires approximately 70 hours of librarian time and 200 hours of the faculty Director's time. In the event that the medical librarian Co-Director (JH) is not available, the faculty Director (DT) provides coverage for this portion of the rotation. Absence of the faculty Director for one or two days can usually be covered by schedule modifications and communication by telephone and e-mail. An extended absence of the faculty Director requires another faculty member assuming supervisory responsibility.
We evaluated our EBM curriculum with respect to our 3 primary goals using pre- and post-rotation questionnaires (completed by each EBM intern) and by a survey of family practice residents and faculty.
The EBM intern questionnaires consisted of pre- and post-rotation self-assessments of confidence in EBM knowledge and skills (Goal 1). Self-confidence in EBM knowledge and skills was assessed by asking the intern to rate his or her level of comfort from 1 = very uncomfortable to 5 = very comfortable for (1) MEDLINE searching to answer a clinical question; (2) use of Web-based EBM resources to answer a clinical question; and (3) use of EBM principles to critically evaluate articles. At the end of the rotation interns were asked to identify the most useful and least useful aspects of the rotation and what could make the rotation better. In addition, residents completed and returned to the Residency Coordinator a standard evaluation of the rotation.
We evaluated the dissemination and reinforcement of EBM skills and knowledge to residents and faculty (Goal 2) and the application of EBM to patient care (Goal 3) by surveying residents (all of whom had previously completed the EBM rotation as interns) and faculty. These surveys were distributed and collected by a program assistant. Surveys were labeled with a code number for each resident and faculty and results were reported in aggregate to provide anonymity. The resident survey asked "How frequently have you continued to apply EBM concepts and tools from the rotation to answer clinical questions?" and "How frequently have EBM concepts and tools been reinforced via teaching by faculty or senior residents?" For both questions, response options were 1 = never, 2 = seldom (less than once per month on average); 3 = occasionally (1 to 3 times per month on average); 4 = often (1 or 2 times a week on average); and 5 = frequently (3 or more times per week on average). For faculty and residents who had had a clinical question answered by the EBM intern on the inpatient service were asked. "How often did the EBM answer provide useful information?" and "How often did the answer change your management of a patient?" For both questions, response options were 1 = less than 25% of the time; 2 = 25% to 75% of the time; and 3 = more than 75% of the time. Finally, all faculty and residents were asked if they believed that "the presence of the EBM rotation improves the quality of patient care within the family practice residency program."
Comparison of interns' confidence* in EBM skills pre- and post-rotation (n = 10)
Mean Pre-Rotation Score
Mean Post-Rotation Score
Use of PubMed/Medline
Use of other Web-based EBM resources
Use of EBM tools and principles (critical appraisal)
Mean confidence score
Residents' report of how often they applied EBM concepts and tools to patient care, and how often EBM concepts and tools are reinforced by senior residents and faculty (n = 21)
Seldom (< 1/month)
Occasionally (1 to 3 times/month)
Often (1 to 2 times/week)
Frequently (≥3 times/week)
Applied to patient care
Reinforced by senior residents and faculty
Percent of time that the answer to EBM question provided useful information or led to a change in patient management on the Family Practice Inpatient Service
Residents (n = 20*)
Faculty (n = 6**)
25% to 75%
25% to 75%
Provided useful information
Led to a change management of the patient
While the above findings support the acceptance and perceived utility of our EBM program, they do not provide a formal measure of its effectiveness. Changes in resident knowledge, skills and confident were measured over a short period of time. The frequency of reinforcement of EBM and the impact of EBM on clinical care was by resident and faculty report and may have been biased. No attempt was made to observe changes in physician behaviors or patient outcomes. There were no measures of EBM use prior to the introduction of the EBM rotation and no comparison group was available. It is therefore not possible to objectively determine to what degree current levels of awareness and utilization of EBM are the result of the rotation. It is also not possible to separate out the effects of the 2-week EBM rotation from the adjunct changes of establishing a medical information website or promoting the use of PDAs, except to the extent that the questions asked specifically about the 2-week EBM rotation.
Our block EBM rotation differs substantially from the approaches reported in previous studies [9, 11, 16] in that it utilizes a concentrated, individual experience that combines formal learning (via tutorials and a web-based course) with immediate application of EBM to answer an important clinical questions for individual patients – the target goal for EBM training. An advantage of our approach is the ability to tailor the curriculum to the background and needs of each intern and to provide interns with dedicated time during which they can rapidly acquire EBM knowledge and skills and apply them to "real time" clinical questions under the supervision of a faculty member and a librarian information specialist. We believe that our curriculum could be implemented by any faculty member with a working knowledge of EBM. The website tutorial  provides the core didactic portion for the rotation. Moreover, directing the rotation naturally increases the experience and expertise of the faculty member involved.
Interns reported greater confidence in their search skills after the two week rotation. Two randomized controlled trials and a controlled before-after study have demonstrated benefits of training in electronic search skills [17–19]. Interns also reported significantly greater confidence in their ability to apply EBM knowledge and principals in critical appraisal. This is consistent with previous studies that have found training improves critical appraisal skills for residents and practicing physicians .
One study that evaluated the impact of a 1-month pilot program to use EBM methods on an inpatient service, reported results roughly similar to ours . Our curriculum also includes the archiving of EBM answers in a standardized format for future reference by residents and faculty. While our study did not include an evaluation of the usefulness of the archived EBM answers generated by the rotation, archiving EBM answers for web-based access has been shown to provide a useful resource for resident physicians in an internal medicine residency program .
Previous studies have found only small increases in residents' knowledge and skills from journal clubs alone, leading to the suggestion that journal clubs should be used as a component of EBM training rather then being 'stand alone' activities [6–8]. While we did not attempt to evaluate the effects of the journal club per se, including it as a component of our EBM curriculum is concordant with this suggestion.
Other investigators have reported educational interventions aimed at improving faculty knowledge and skills in medical informatics and EBM. One study reported increases in faculty self-rating of EBM skills following an intervention consisting of 2 half-day workshops and substantial amount of individual mentoring . We found that faculty reported our EBM rotation has increased their use of EBM in their clinical practice, as well as their teaching of EBM; this was echoed by residents. While it is difficult to compare the two studies, our findings suggest that integrating EBM into the residency via resident training may improve faculty application of EBM to clinical care and in their teaching, and may be a cost-effective way to reinforce faculty EBM skills.
Our EBM curriculum, based on an individual 2-week EBM rotation in the first year, appears to be successful in increasing resident's EBM skills and confidence. In addition, resident and faculty both perceive EBM as being incorporated and reinforced beyond the rotation, and that the presence of EBM in the residency improves the quality of patient care. We hope that our experience provides a useful model for teaching and integrating EBM into a busy, resource-limited, family practice residency.
The authors would like to acknowledge Juliana Fung for her assistance in the preparation of this manuscript.
- Sackett DL, Straus SE, Richardson WS, Rosenberg W, Haynes RB: Evidence-Based Medicine: How to Practice and Teach EBM. 2000, New York: Churchill Livingston, secondGoogle Scholar
- Schwartz K, Northrup J, Israel N, Crowell K, Lauder N, Neale AV: Use of on-line evidence-based resources at the point of care. Fam Med. 2003, 35: 251-6.Google Scholar
- Ramos K, Linscheid R, Schafer S: Real-time information-seeking behavior of residency physicians. Fam Med. 2003, 35: 257-60.Google Scholar
- Smith CA, Ganschow PS, Reilly BM, Evans AT, McNutt RA, Osei A, Saquib M, Surabhi S, Yadav S: Teaching residents evidence-based medicine skills: a controlled trial of effectiveness and assessment of durability. J Gen Intern Med. 2000, 15: 710-5. 10.1046/j.1525-1497.2000.91026.x.View ArticleGoogle Scholar
- Shaughnessy AF, Slawson DC, Bennett JH: Becoming an information master: a guidebook to the medical information jungle. J Fam Pract. 1994, 39: 489-99.Google Scholar
- Green ML: Evidence-based medicine training in graduate medical education: past, present and future. J Eval Clin Pract. 2000, 6: 121-38. 10.1046/j.1365-2753.2000.00239.x.View ArticleGoogle Scholar
- Green ML: Evidence-based medicine training in internal medicine residency programs. J Gen Intern Med. 2000, 15: 129-33. 10.1046/j.1525-1497.2000.03119.x.View ArticleGoogle Scholar
- Norman GR, Shannon SI: Effectiveness of instruction in critical appraisal (evidence-based medicine) skills: a critical appraisal. C M A J. 1998, 158: 177-81.Google Scholar
- Ross R, Verdieck A: Introducing an evidence-based curriculum into a family practice residency – is it effective?. Acad Med. 2003, 78: 412-7.View ArticleGoogle Scholar
- Cabell CH, Schardt C, Sanders L, Corey R, Keitz SA: Resident utilization of information technology: a randomized trial of clinical question formation. J Gen Intern Med. 2001, 11: 838-44. 10.1046/j.1525-1497.2001.10239.x.View ArticleGoogle Scholar
- Green ML, Ellis PJ: Impact of an evidence-based medicine curriculum based on adult learning theory. J Gen Intern Med. 1997, 12: 742-50. 10.1046/j.1525-1497.1997.07159.x.View ArticleGoogle Scholar
- Friedland DJ, Go AS, Davoren JB, Shlipak MG, Bent SW, Subak LL, Mendelson T: Evidence-Based Medicine: A Framework for Clinical Practice. Stamford Connecticut: Appleton & Lange. 1998Google Scholar
- McKibbon A: PDQ: Evidence-based Principles and Practice. 1999, B.C. Decker, IncGoogle Scholar
- UCSF/CHN Family Practice Residency Program Medical Information Resources. [http://sfghdean.ucsf.edu/barnett/fcm/]
- Michigan State University: An Introduction to Information Mastery. [http://www.poems.msu.edu/InfoMastery/]
- Ramos KD, Schafer S, Tracz S: Validation of the Fresno test of competence in evidence based medicine. Br Med J. 2003, 326: 319-21. 10.1136/bmj.326.7384.319.View ArticleGoogle Scholar
- Rosenberg WMC, Deeks J, Lusher A, Snowball R, Dooley G, Sackett D: Searching skills and evidence retrieval. J R Coll Physicians Lond. 1998, 32: 557-63.Google Scholar
- Cheng GY: Educational workshop improved information-seeking skills, knowledge, attitudes and the search outcome of hospital clinicians a randomized controlled trial. Health Info Lib J. 2003, 20 (Suppl 1): 22-33. 10.1046/j.1365-2532.20.s1.5.x.View ArticleGoogle Scholar
- Ghali WA, Saitz R, Eskew AH, Gupta M, Quan H, Hershman WY: Successful teaching in evidence-based medicine. Med Ed. 2000, 34: 18-22. 10.1046/j.1365-2923.2000.00402.x.View ArticleGoogle Scholar
- Taylor R, Reeves B, Ewings P, Binns S, Keast J, Mears R: A systematic review of the effectiveness of critical appraisal skills training for clinicians. Med Ed. 2000, 34: 120-5. 10.1046/j.1365-2923.2000.00574.x.View ArticleGoogle Scholar
- McGinn T, Selz M, Korenstein D: A method for real-time, evidence-based general medical attending rounds. Acad Med. 2002, 77: 1150-2.View ArticleGoogle Scholar
- Crowley SD, Owens TA, Schardt CM, Wardell SI, Peterson J, Garrison S, Keitz SA: A web-based compendium of clinical questions and medical evidence to educate internal medicine residents. Acad Med. 2003, 78: 270-4.View ArticleGoogle Scholar
- Cartwright CA, Korsen N, Urbach LE: Teaching the teachers: helping faculty in a family practice residency improve their informatics skills. Acad Med. 2002, 77: 385-91.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/4/19/prepub
This article is published under license to BioMed Central Ltd. This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.