- Research article
- Open Access
Characteristics of evidence-based medicine training in Royal College of Physicians and Surgeons of Canada emergency medicine residencies - a national survey of program directors
BMC Medical Education volume 14, Article number: 57 (2014)
Recent surveys suggest few emergency medicine (EM) training programs have formal evidence-based medicine (EBM) or journal club curricula. Our primary objective was to describe the methods of EBM training in Royal College of Physicians and Surgeons of Canada (RCPSC) EM residencies. Secondary objectives were to explore attitudes regarding current educational practices including e-learning, investigate barriers to journal club and EBM education, and assess the desire for national collaboration.
A 16-question survey containing binary, open-ended, and 5-pt Likert scale questions was distributed to the 14 RCPSC-EM program directors. Proportions of respondents (%), median, and IQR are reported.
The response rate was 93% (13/14). Most programs (85%) had established EBM curricula. Curricula content was delivered most frequently via journal club, with 62% of programs having 10 or more sessions annually. Less than half of journal clubs (46%) were led consistently by EBM experts. Four programs did not use a critical appraisal tool in their sessions (31%). Additional teaching formats included didactic and small group sessions, self-directed e-learning, EBM workshops, and library tutorials. 54% of programs operated educational websites with EBM resources. Program directors attributed highest importance to two core goals in EBM training curricula: critical appraisal of medical literature, and application of literature to patient care (85% rating 5 - “most importance”, respectively). Podcasts, blogs, and online journal clubs were valued for EBM teaching roles including creating exposure to literature (4, IQR 1.5) and linking literature to clinical practice experience (4, IQR 1.5) (1-no merit, 5-strong merit). Five of thirteen respondents rated lack of expert leadership and trained faculty educators as potential limitations to EBM education. The majority of respondents supported the creation of a national unified EBM educational resource (4, IQR 1) (1-no support, 5- strongly support).
RCPSC-EM programs have established EBM teaching curricula and deliver content most frequently via journal club. A lack of EBM expert educators may limit content delivery at certain sites. Program directors supported the nationalization of EBM educational resources. A growing usage of electronic resources may represent an avenue to link national EBM educational expertise, facilitating future collaborative educational efforts.
The Royal College of Physicians and Surgeons of Canada (RCPSC) provides a five-year training program in Emergency Medicine (EM) with special emphasis on developing academic emergency physicians . The goal of training for fellows of the RCPSC is to prepare learners for roles in university centers. Academic emergency positions require expertise in medical education, research, and administration . Graduates are expected to apply evidence-based medicine (EBM) in scholarly and managerial roles  (Table 1). Practically, the structured approach of EBM has particular importance in the emergency department, with high levels of diagnostic uncertainty, decision density, and time pressure [4, 5].
EBM education in EM remains in the early stages of development. A recent survey of the Council of Emergency Medicine Residency Directors in the United States (US) indicates a minority of programs had predefined journal club or EBM curricula (25%) or used structured critical appraisal instruments in their journal clubs (29%) . Kuhn and colleagues reported only 22% of EM program directors in the US conducted more than five didactic EBM sessions per year . In both studies, program directors expressed a desire for a defined curriculum [6, 7]. Such work has established a baseline in teaching methods and outlined a needs assessment for training programs in the US, however this first step has not been performed in Canada.
In EM residencies, EBM content delivery has been described and formats include traditional journal club, structured “evidence-based” journal club, didactic formats, and web based EM blogs [8–11]. Specific concerns have been raised regarding the narrow focus placed on critical appraisal during journal club sessions, however the optimal educational method has not been defined [12, 13]. Adding complexity, the rapid expansion of web based educational technology has changed the way learners access, analyze, and catalogue new evidence .
Before evaluating EBM training in EM, current practices must be defined. Our primary objective was to describe the methods of EBM training in RCPSC-EM residencies. Secondary objectives were to (i) explore attitudes regarding current educational practices, including e-learning, (ii) explore potential barriers to journal club and EBM education, and (iii) assess the desire for national collaboration. By defining educational methods across institutions and identifying areas for growth, we hope to create a platform for local and national quality improvement initiatives relevant to EBM education in EM.
Study design and setting
A paper-based survey of all RCPSC-EM program directors in Canada was performed. Ethics approval was obtained from the University of Manitoba Health Research Ethics Board. Contact information for programs was obtained from the Canadian Resident Matching Service (CaRMS) website . A cover letter stated survey completion was adequate evidence of informed consent. Respondents received a fifty-dollar honoraria directed to their resident education fund.
Survey content and administration
An anonymous 16-question survey containing binary, open ended, and 5 point likert-scale questions was developed by the authors. Survey content was refined by consensus discussion during local focus group meetings and informal collaboration with national EBM experts in EM. We explored the following domains: (i) EBM curricula format and core values, (ii) journal club characteristics and limitations, (iii) use and perceived merit of electronic/web based EBM resources, and (iv) basic demographics. All likert scales provided opportunity for neutral responses. Several questions offered “other” selections and free text to elicit alternate responses.
Program directors were mailed introductions to the research initiative beginning in July 2012. Surveys were mailed one week later with a cover letter indicating estimated completion time, voluntariness, honoraria value, and confidentiality via aggregate data reporting. Potential respondents were provided postage paid return envelopes addressed to an administrative assistant who removed identifying packaging before returning data to the authors. Non-respondents received two bimonthly email requests to complete the survey. Responses were collected until August 2012.
Data were stored using Numbers ’09 (Apple Inc. Version 2.1 (c) 2008-2011). Descriptive statistical analysis was performed. Because of the obligatory small sample size and descriptive nature of the study objective, formal sample size calculations and statistical significance were not performed. To preserve the integrity of data trends in likert scale questions, results are represented in both median and percentage where applicable.
Fourteen RCPSC-EM programs were identified, of which 13 programs (93%) responded. Seven (54%) respondent programs belonged to EM programs with departmental status. Additional demographics of respondents are represented in (Table 2). Eleven respondent programs (85%) reported having formal EBM teaching curricula. Ten programs (77%) had a lead EBM educational coordinator. Curricula content was delivered primarily via journal club, with 62% of programs having 10 or more sessions annually. Teaching formats utilized less frequently included didactic and small group sessions, self-directed e-learning, EBM workshops, and library tutorials (Figure 1).
Article selection for journal club occurred most frequently via resident or attending choice (85% and 54% respectively). Four programs (31%) had predefined journal club curricula, and only one program reported journal club article selection from clinical questions generated on shift. Six programs (46%) indicated journal club sessions were consistently led by an EBM expert, defined as an individual with formal training in EBM, extensive background in clinical research, or a longstanding academic interest in biostatistics and research methodology. Nine programs (69%) used a critical appraisal tool in their journal club sessions. Respondents who specified their critical appraisal tool used the JAMA User’s Guide (5/9) or Oxford critical appraisal worksheets (1/9) [16, 17]. Sessions were held at staff residences or university sites in most cases (38% and 48% respectively), with only 2 programs (15%) holding journal club at public venues.
Noise, alcohol, and distractions in public venues were thought to compromise the educational value of journal club, with several programs reporting some (6 (46%)) or significant (2 (15%)) limitation. Five respondents perceived lack of expert leadership and trained faculty educators as posing some (3,23%) or significant (2,15%) limitation. “Narrow focus on critical appraisal” was not frequently rated as a significant limitation to journal club (Table 3).
Program directors attributed highest importance to two core goals in EBM training curricula: critical appraisal of medical literature, and application of literature to patient care (85% rating 5 -“most importance”, respectively). Effective search strategy was also highly valued, with 69% of respondents rating “most importance” to this objective. Conversely, understanding biostatistics and developing a foundation for producing original research were less frequently attributed “most importance” in EBM curricula (15 and 23% respectively) (Figure 2).
Over half of programs operated educational websites with links to external EBM resources and documented journal club summaries electronically (7 (54%) respectively). E-resources including podcasts, blogs, and online journal clubs were valued for certain roles as components of EBM education (Figure 3). Most respondents supported the role of e-resources in creating exposure to emerging literature and linking EBM to clinical practice experience ((5, 38%) rating “4”, (5 (38%) rating “5”, 1 -no merit, 5 - strong merit). However, program directors were more likely to report a neutral response relating to the merit of e-resources in teaching specific aspects of EBM theory (6 (46%) rating “3”-neutral). Respondents indicated referring residents to e-resources that are not formally peer-reviewed may be hazardous (median response “4”, IQR 1; 1-insignificant danger, 5-significant danger).
Only four programs (31%) had evaluated the effectiveness of their EBM teaching methods or journal club in a quality improvement initiative in the last three years. Despite this, several respondents indicated limited EBM trained faculty educators may pose a barrier to effective EBM teaching (Table 4). 46% of respondents strongly supported the creation of a national unified EBM educational resource for the purposes of making EBM expert educators available, linking journal clubs, and creating a sustainable critically appraised topic (C.A.T) database (median 4, IQR1, 1-no support, 5 – strongly support).
To our knowledge, this is the first effort to describe the landscape of EBM training in Canadian EM residencies. Compared with previous reports of EM training programs in the US, EBM teaching curricula were prominent in nearly all survey respondents. While the aim of this study was not to thoroughly review individual curricula, several observations may serve as a basis for self-evaluation and quality improvement initiatives.
The EBM skill set enables an EM trainee to make evidenced-based patient care decisions, but begins with the ability to seek out and appraise resources based on a resident’s clinical questions [12, 18]. A successful training program teaches each step of the EBM process, producing physicians who incorporate EBM as a behavior in their daily practice . Although respondents attributed some importance to each component of EBM, critical appraisal was highly valued. This was also evidenced by the predominance of journal club - a critical appraisal exercise - as the most frequently reported educational activity among EBM curricula.
Much research has focused on journal club as an EBM teaching tool in EM [8–10, 19]. The efficacy of this traditional format to teach the breadth of EBM skills or change behavior has been questioned [10, 11, 20]. In a systematic review of journal club in post-graduate medical education, none of the 101 articles retrieved demonstrated translation of evidence from journal club into the clinical practice of learners . Because of a narrow focus on critical appraisal, journal club may be an unsatisfactory method of teaching the full scope of evidence based decision making . Furthermore, as Green argues: “critical appraisal may be the most expendable skill for the ‘front line’ physician practicing evidence-based medicine in the ‘real world’” . Despite these growing concerns, few respondents in our survey identified the narrow focus on critical appraisal as a significant limitation to their journal clubs or EBM curricula.
Recommendations from an extensive review on improving journal club effectiveness included regular meetings, mandatory attendance, and use of a critical appraisal tool . These respective criteria were reported by 92, 100, and 69% of respondents in our survey. Yet perhaps the most consistent suggestion in EBM education literature is to engage learners in the breadth of the EBM process with applied techniques, beginning and ending at the patient’s bedside [11, 23, 24]. Curricula incorporating experiential learning may best meet the needs of adult learners [22, 23, 25]. As demonstrated in an observational cohort, Friedman et al. described a knowledge translation intervention consisting of a 1 hour tutorial and real time EBM exercise in the emergency department . This encouraging work showed a change in patient management in 16.3% of encounters and the potential to impact future practice in resident learners. Of concern, only one of our respondent programs indicated residents generated journal club topics from clinical questions arising on shift. This may represent an important area for growth in Canadian EM curricula.
Additional teaching formats were used with lesser frequency among respondents - including didactic sessions and library tutorials. Such formats are supported by the literature, as Grant demonstrated a didactic intervention was effective in teaching EBM theory to EM residents . Of much interest, however, is the use and support of e- resources by EM programs in resident education, with over half of programs operating EBM websites and cataloging critically appraised topics. Web 2.0, a term encompassing the collaborative use of information technology (blogs, social media, podcasts) has increasing relevance to EM education . With over twelve thousand MEDLINE articles added weekly, EBM curricula must also educate residents to identify sources of pre-appraised evidence [27, 28]. Peer reviewed sources such as Academic Emergency Medicine, Annals of Emergency Medicine, and Emergency Medicine Journal are among the contributors to over 200 EM blogs (catalogued at http://lifeinthefastlane.com) which may help learners filter this vast amount of information [29–32]. Given the inherent variability in quality of such resources, however, PD’s may wish to develop and incorporate methods of critical appraisal of web-based secondary information sources into resident education.
Our survey demonstrated less than half of programs had consistent leadership by EBM experts at journal club sessions. In addition to being accessible, current, and asynchronous, web-based EBM resources also create a non-hierarchal environment where junior learners can interact with experts in the field, and participate in critical appraisal on a broader scale [14, 33]. Linking respective journal clubs or online EBM resources between institutions may create a bridge for smaller centers with fewer EBM educators to larger institutions with surplus leadership and research experience. Based on the demographic data in this survey, respondents varied widely in program maturity and size. The authors speculate that the five respondents that cited lack of EBM expert leadership and limited faculty educators as limitations to EBM training may belong to younger, smaller programs. Indeed, this further supports the case for linking our nations EBM resources, a proposition met with great support by survey respondents. We feel this may represent an exciting opportunity to transform and strengthen EBM training in Canadian EM, by uniting institutions on a web-based EBM educational platform.
Our survey instrument was not prospectively tested for validity. Respondents represented the lead education coordinators for RCPSC-EM residents, but their opinions may differ from those of respective institutional EBM leaders. Furthermore, because of the relative paucity of specialty EM training programs in Canada, our sample size was small. This may have contributed to type I or II error. Future study defining the values and needs of residents in EBM education may be valuable. Ongoing efforts to identify educational interventions that change behavior in learners have great importance for EBM curricula.
The majority of RCPSC-EM programs have prioritized EBM education by developing dedicated curricula. EBM training is diverse, however journal club and critical appraisal appear to be emphasized. This may not optimize the accomplishment of all core EBM training goals valued by respondents. If continued at the reported frequency, journal club sessions designed for resident education should incorporate applied, patient centered methods based on principles of adult learning theory, increase the use of critical appraisal tools, and assure EBM experts are involved in sessions. Lack of EBM leadership at some centers may hinder optimal resident education. A growing usage and support of web-based resources may represent an avenue to link EBM educators across Canada, and respond to the desire expressed by program directors to create a national unified EBM educational resource.
Royal College of Physicians and Surgeons of Canada
Canadian resident matching service
Learning and teaching formats involving electronic media.
Rutledge T: Emergency medicine training in Canada: learning from the past to prepare for the future. Can J Emerg Med. 2008, 10: 108-110.
Ip S: Emergency medicine practice and training in Canada. Can Med Assoc J. 2003, 168: 1549-1550.
Objectives of Training in Emergency Medicine. http://www.royalcollege.ca/.
Croskerry P, Sinclair D: Emergency medicine: a practice prone to error?. Can J Emerg Med. 2001, 3: 271-276.
Newman DH, Wyer P, Kaji A: Evidence-based medicine. A primer for the emergency medicine resident. Ann Emerg Med. 2002, 38: 77-80.
Carpenter C, Kane B, Carter M, Lucas R, Wilbur L, Graffeo C: Incorporating evidence-based medicine into resident education: a CORD survey of faculty and resident expectations. Acad Emerg Med. 2010, 17 (S2): S54-S61.
Kuhn G, Wyer P, Cordell W, Rowe B: A survey to determine the prevalence and characteristics of training in evidence-based medicine in emergency medicine residency programs. J Emerg Med. 2005, 28: 353-359. 10.1016/j.jemermed.2004.09.015.
Burstein J, Hollander J, Barlas D: Enhancing the value of journal club: use of a structured review instrument. Am J Emerg Med. 1995, 14: 561-563.
Carley SD, Mackway-Jones K, Jones A, Morton RJ, Dollery W, Maurice S, Niklaus L, Donnan S: Moving towards evidence based emergency medicine: use of a structured critical appraisal journal club. J Accid Emerg Med. 1998, 15: 220-226. 10.1136/emj.15.4.220.
Carpenter CR, Katz E, Char D: Re: journal club and teaching evidence-based medicine. J Emerg Med. 2006, 31: 307-308. 10.1016/j.jemermed.2006.06.006.
Genes N, Parekh S: Bringing journal club to the bedside in the form of a critical appraisal blog. J Emerg Med. 2010, 39: 504-505. 10.1016/j.jemermed.2008.10.011.
Hatala R, Guyatt G: Evaluating the teaching of evidence-based medicine. J Am Med Assoc. 2002, 288: 1110-1112. 10.1001/jama.288.9.1110.
Hatala R: Is Evidence-based Medicine a Teachable Skill?. Annals of Emergency Medicine. 1999, 34: 226-228. 10.1016/S0196-0644(99)70236-2.
Berger E: Web 2.0 in emergency medicine. Ann Emerg Med. 2012, 59: 21A-23A. 15
Canadian Resident Matching Service. http://www.carms.ca.
Oxford Critical Appraisal Sheets. http://www.cebm.net/index.aspx?o=1157.
JAMA User’s Guide Critical Appraisal Worksheets. http://jamaevidence.com/resource/520.
Guyatt G, Rennie D, Meade M, Cook D: User’s Guides to the Medical Literature, A Manual for Evidence Based Clinical Practice: McGraw Hill. JAMA Evidence. 2008, 5-6. 6
Bazarian J, Davis C, Spillane L, Blumstein H, Schneider S: Teaching emergency medicine residents evidence-based critical appraisal skills: a controlled trial. Ann Emerg Med. 1998, 34: 148-154.
Ahmadi N, McKenzie M, MacLean A, Brown CJ, Mastracci T, McLeod RS: Teaching evidence based medicine to surgery residents - is journal club the best format? A systematic review of the literature. J Surg Educ. 2012, 69: 91-100. 10.1016/j.jsurg.2011.07.004.
Deenadayalan Y, Grimmer-Somers K, Prior M, Kumar S: How to run an effective journal club: a systematic review. J Eval Clin Pract. 2008, 14: 898-911. 10.1111/j.1365-2753.2008.01050.x.
Green M: Evidence-based medicine training in graduate medical education: past, present and future. J Eval Clin Pract. 2000, 6: 121-138. 10.1046/j.1365-2753.2000.00239.x.
Friedman S, Sayers B, Lazio M, Friedman S, Gisondi MA: Curriculum design of a case- based knowledge translation shift for emergency medicine residents. Acad Emerg Med. 2010, 17: S42-S48.
Hardem RD: Teaching and learning evidence based medicine skills in accident and emergency medicine. J Accid Emerg Med. 1998, 16: 126-129.
Green M, Ellis P: Impact of an evidence-based medicine curriculum based on adult learning theory. J Gen Intern Med. 1997, 12 (12): 742-750. 10.1046/j.1525-1497.1997.07159.x.
Grant DC, Keim S, Telfer J: Teaching Bayesian analysis to emergency medicine residents. J Emerg Med. 2006, 31: 437-440. 10.1016/j.jemermed.2006.04.015.
Glasziou P: Information overload: what’s behind it, what’s beyond it?. Med J Aust. 2008, 189 (2): 84-85.
Rowe BH, Diner B, Camargo C, Worster A, Colacone A, Wyer P: Effective Synthesized/Preappraised evidence formats in emergency medicine and the use of supplemental knowledge translation techniques. Acad Emerg Med. 2007, 14: 1023-1028. 10.1111/j.1553-2712.2007.tb02383.x.
Academic Emergency Medicine. http://vimeo.com/aem.
Annals of Emergency Medicine. http://www.annemergmed.com/content/podcast.
EMJ Podcasts. http://podcasts.bmj.com/emj/.
Life in the Fast Lane EMCC Blog Index. http://lifeinthefastlane.com/resources/emergency-medicine-blogs.
Nickson C: Web 2.0 For Emergency Physicians. http://lifeinthefastlane.com/2009/07/web-20-for-emergency-physicians.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/14/57/prepub
The authors wish to extend their gratitude to Darlene Scibak for contributing significant administrative support to this project. The authors also wish to acknowledge the contributions of Brenden Dufault as a consultant biostatistician. Funding was provided by the University of Manitoba Department of Emergency Medicine Resident Education Fund.
The authors have no financial or other conflicts of interest to disclose.
JB contributed to project conception and design, data acquisition, data analysis and interpretation, and manuscript composition. MP contributed to project conception and design, revision of survey instrument and manuscript revision/critical appraisal. JF contributed to project conception and design, design of survey instrument, data acquisition and manuscript revision. EW contributed to overall inception and project supervision, project conception and design, data analysis and interpretation, and senior manuscript revisions. All authors give final approval for publication and are accountable for all aspects of the work’s accuracy and integrity.
About this article
Cite this article
Bednarczyk, J., Pauls, M., Fridfinnson, J. et al. Characteristics of evidence-based medicine training in Royal College of Physicians and Surgeons of Canada emergency medicine residencies - a national survey of program directors. BMC Med Educ 14, 57 (2014). https://doi.org/10.1186/1472-6920-14-57
- Evidence-based medicine
- Medical education
- Emergency medicine
- Journal club