How are "teaching the teachers" courses in evidence based medicine evaluated? A systematic review
BMC Medical Education volume 10, Article number: 64 (2010)
Teaching of evidence-based medicine (EBM) has become widespread in medical education. Teaching the teachers (TTT) courses address the increased teaching demand and the need to improve effectiveness of EBM teaching. We conducted a systematic review of assessment tools for EBM TTT courses. To summarise and appraise existing assessment methods for teaching the teachers courses in EBM by a systematic review.
We searched PubMed, BioMed, EmBase, Cochrane and Eric databases without language restrictions and included articles that assessed its participants. Study selection and data extraction were conducted independently by two reviewers.
Of 1230 potentially relevant studies, five papers met the selection criteria. There were no specific assessment tools for evaluating effectiveness of EBM TTT courses. Some of the material available might be useful in initiating the development of such an assessment tool.
There is a need for the development of educationally sound assessment tools for teaching the teachers courses in EBM, without which it would be impossible to ascertain if such courses have the desired effect.
Evidence-based medicine (EBM), is defined as the integration of the best research evidence with patient's values and clinical circumstances for clinical decision making . However, for EBM to become a reality for practice, studies have shown that it needs to be taught by competent clinical trainers, who help trainees to learn efficiently by exploiting on-the-job learning opportunities . Trainers who have received formal EBM training are more likely to teach application of EBM in a clinical setting . Teaching EBM should not only equip practitioners with knowledge and skills but also foster their attitudes and encourage the practice of EBM. Critical appraisal and EBM teaching that is integrated into clinical practice appears to be effective in improving substantial outcomes, including behavioural changes . One of the objectives of TTT courses should be to bring about a positive change in the teacher's knowledge, skills and attitude towards EBM teaching in a clinical setting. For professional development in this field, many trainers attend teaching the teachers (TTT) courses for EBM, which are widespread. The Leonardo da Vinci programme, part of the European Commission's Lifelong Learning Programme, has funded a EU project involving collaboration of 10 partners within Europe to design an e-learning curriculum for continuing professional development (CPD) on how to teach EBM in a clinical setting http://www.ebm-unity.org[5, 6]. Through this project, we have developed a European qualification in Teaching Evidence-Based Medicine which aims to improve transparency across the European healthcare sector . Often teachers assess their own learning through their students' feedback. However, this may be naive as it is well known that there is no direct relationship between feedback and learning achievement [8, 9]. The use of assessment tools if any to evaluate such courses is unknown. There are currently many courses that train the trainer in teaching EBM skills. A review of assessment tools used for such courses can help us to gauge how TTT courses are evaluated. We aimed to systematically review existing assessment tools used to evaluate TTT courses in EBM.
We searched the following databases from database inception to November 2008: PubMed, EmBase, BioMed, Cochrane Library and Eric (Education Resources Information Centre, a collection of bibliographic records of education literature). Search strategies for electronic databases were developed based on the review of known relevant literature, and combined MeSH and free text terms, as shown in Additional File 1. We handsearched relevant papers identified through the reference list of retrieved articles, examined our Consortium's personal files, explored for related articles in Pubmed and searched citations in Google Scholar. No language restrictions were applied. Target audiences of the course (teachers) were defined as any health care professional with teaching responsibilities. The definition of a TTT course was based on three characteristics: the aim of the course, the type of participants, and the presence of an assessment strategy. We selected the courses that were related to learning teaching skills in EBM. We included papers that reported development, validation or implementation of any assessment tool or methods to evaluate an EBM TTT course. Due to the specificity of the subject matter we did not use restrictive inclusion criteria related to study quality. All titles and abstracts were screened by two reviewers for studies that met the inclusion criteria. Any disagreement between reviewers was resolved by consensus. We extracted data on the course objectives, duration, target audience, teaching methods, assessment strategy and key outcomes.
Our search identified 1230 primary articles, of which five were included in the review (Figure 1). The summary of all TTT courses, which have been included in this systematic review is shown in Additional File 2, providing details of course objectives, target audience, teaching methods, duration of course, assessment methods and key outcomes.
The target audience of the EBM TTT courses identified in the review were faculty members in family practice residences who help residents develop skills in the use of informatics and evidence based medicine , podiatric medical educators , clinical teachers, and librarians [14–16]. Diverse teaching methods such as instructional workshops, facilitated discussions, hands-on exercises and small group sessions were used in the courses. We did not identify a clinically integrated course teaching application of EBM in routine clinical practice. The total duration of these courses varied from 1 day to 24 months.
The courses used various assessment tools independent of the teaching method to evaluate the effectiveness of the course. The most popular method of course assessment was pre-and post-course appraisal as observed in three of the five chosen articles [12, 13, 15]. One course surveyed the participants only once, after the completetion of the course. The most detailed assessment method to study the effectiveness of the course was by Scherrer et al . It was a collaboration between librarians and medical faculty to train librarians and health professionals to teach principles of EBM. EBM principles and skills were presented and appropriate teaching methods and techniques were considered. As a method of evaluation, a questionnaire with twenty questions was created and designed to be completed by a twenty-minute telephone interview. After a pilot project, the final survey contained twenty-three questions that took about fifteen minutes to complete. Respondents were asked to state if the course resulted in increased teaching of EBM concepts by the participants and methods used when they returned to their institutions. It explained the influence of the course on the participants in teaching EBM principles, their understanding of EBM principles and confidence in teaching EBM concepts. One of the questions aimed to identify the barriers faced by the participants in teaching EBM. The components of this questionnaire may constitute a basis for the development of an assessment tool to evaluate teaching the teachers courses for EBM .
This review has revealed variations in the content of EBM courses, their duration, methods of teaching and course assessment. The reviewers have identified a substantial gap in assessment methods for the evaluation of EBM TTT courses. Very few studies in this field had any detail of course assessments. Existing assessment methods are very few and of poor quality. Moreover the identified assessment tools were intended for teacher' courses focused on courses conducted traditionally (workshops, one-to-one consultation, lectures) rather than clinically integrated courses.
Our review has been the first one to systematically search for assessment tools for teaching the teachers courses in EBM. It was systematically conducted with rigorous methodology and has comprehensively searched, identified and described existing tools for assessing EBM TTT courses. The findings of the review are limited by the small number of published reports on the assessment methods used in this area. EBM can only improve care if it is integrated in clinical practice . This can be achieved by effective application of EBM skills in the clinical environment.To evaluate attainment of this objective teaching the teachers courses need to employ appropriate assessment tools. The TTT EBM curricula is widely based on the model developed in McMaster University, Sackett's books on how to teach EBM and User's guide to medical literature . But little seems to be reported or known about EBM teaching evaluation tools, with much of this knowledge observational .
Until now several tools for assessment are known to be used in different kinds of courses: Research Self-Efficacy Scale (RSES), the stages of change (SOC) model and Kirkpatrick's model [9, 20, 21]. RSES scale has good internal consistency but it has been designed to assess research self-confidence. Thus it lacks face validity for teaching the teachers courses which focus on helping with EBM teaching not on conducting or teaching how to research. The SOC model suffers the same deficiency as it examines learners' attitudes, intentions and actions towards research, not EBM teaching. Moreover, self assessments of this type suffer in the area of criterion validity as there is a loose link between self-perception and objective assessment . The most commonly known tool is the Kirkpatrick's model, which is used frequently as a tool for assessing individual trainee performance but also for measuring the effectiveness of teaching the teacher courses [20, 21]. Evaluation of educational interventions must concern at least four dimensions embedded in Kirkpatrick's hierarchy, such as participant's satisfaction, learning (both knowledge and skills), behavioral change (understood as a transfer of knowledge and skills to workplace), and outcomes which are mainly considered as impact on patients . However none of included articles described and made use of this method. Furthermore, change in knowledge may not equate to change in behaviour.
Successful teaching depends not only on good teaching, but also on willingness and effort by the learner, a supportive social environment with teaching and learning opportunities [8, 23]. For evaluation of a teacher's performance, can we rely on assessments which utilise trainee's evaluations? Two reasons students' feedback is inappropriate as a measure of teacher quality are that they may not measure all that the teachers are trying to achieve and often do not also provide useful information for teachers about what they need to know to teach more effectively [8, 24]. Moreover, data from a randomised controlled trial suggests that trainee feedback can be manipulated by introducing "seduction" in teaching. For these reasons we should evaluate trainers' courses with specifically developed assessment tools. To maintain relevance of the course and to achieve its intended objectives, assessment tools need to be designed as a part of an ongoing evaluation cycle .
Effective teacher education is essential to improve EBM teaching quality  and constitutes the main reason for development of appropriate assessment tools.
Due of the lack of well developed assessment tools for courses for EBM teachers, the effect of teaching skills of tutors is currently not measurable. There is a need to develop educationally sound assessment tools to encourage EBM teachers to provide clinically integrated EBM teaching.
The systematic review was prepared within the study which was funded by the European Union Leonardo da Vinci project grant (grant number: UK/07/LLP-LdV/TOI-062).
Shaneyfelt T, Baum KD, Bell D: Instruments for evaluating education in evidence-based practice: a systematic review. JAMA. 2006, 296 (9): 1116-1127. 10.1001/jama.296.9.1116.
Coppus SF, Emparanza JI, Hadley J, Kulier R, Weinbrenner S, Arvanitis TN, Burls A, Cabello JB, Decsi T, Horvath AR, Kaczor M, Zanrei G, Pierer K, Stawiarz K, Kunz R, Mol BW, Khan KS: A clinically integrated curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU EBM project. BMC Medical Education. 2007, 7: 46-10.1186/1472-6920-7-46.
Kljakovic M, Love T, Gilbert A: Attitudes of teachers to evidence based medicine. Australian Family Physician. 2004, 33 (5).
Coomarasamy A, Khan KS: What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ. 2004, 329: 1017-10.1136/bmj.329.7473.1017.
Kulier R, Hadley J, Weinbrenner S, Meyerrose B, Decsi T, Horvath AR, Nagy E, Emparanza JI, Coppus SF, Arvanitis TN, Burls A, Cabello JB, Kaczor M, Zanrei G, Pierer K, Stawiarz K, Kunz R, Mol BW, Khan KS: Harmonising evidence-based medicine teaching: a study of the outcomes of e-learning in five European countries. BMC Med Edu. 2008, 8: 27-10.1186/1472-6920-8-27.
European Union Evidence Based Medicine Unity. --- Either ISSN or Journal title must be supplied.. [http://www.ebm-unity.org]
Thangaratinam S, Barnfield G, Weinbrenner S, Meyerrose B, Arvanitis TN, Horvath AR, Zanrei G, Kunz R, Suter K, Walczak J, Kaleta A, Oude Rengerink K, Gee H, Mol BW, Khan KS: Teaching trainers to incorporate evidence-based medicine (EBM) teaching in clinical practice: the EU-EBM project. BMC Med Edu. 2009, 9: 59-10.1186/1472-6920-9-59.
Ingvarson L, Rowe K: Conceptualising and evaluating teacher quality: substantive and methodological issues. The Economics of Teacher Quality conference, Australian National University. 2007.
Bates I, Ansong D, Bedu-Addo G, Agbenyega T, Akoto AY, Nsiah-Asare A, Karikari P: Evaluation of a learner-designed course for teaching health research skills in Ghana. BMC Medical Education. 2007, 7: 18-10.1186/1472-6920-7-18.
Khan KS, ter Riet G, Popay J, Glanville J, Sowden AJ, Kleijnen J: (2001b) Study quality assessment: conducting the review. Undertaking Systematic Reviews of Research on Effectiveness CRD Report No. 4, Phase 5, Stage II. 1-20.
Deeks JJ: Systematic reviews in health care: Systematic reviews of evaluations of diagnostic and screening tests. Br Med J. 2001, 323: 157-162. 10.1136/bmj.323.7305.157.
Cartwright CA, Korsen N, Urbach LE: Teaching the teachers: helping faculty in a family practice residency improve their informatics skills. Acad Med. 2002, 77: 385-91. 10.1097/00001888-200205000-00006.
Green ML: A train-the-trainer model for integrating evidence-based medicine training into podiatric medical education. Journal of the American Podiatric Medical Association. 2005, 95: 497-504.
Dorsch JL, Jacobson S, Scherrer CS: Teaching EBM teachers: a team approach. Medical Reference Services Quarterly. 2003, 22: 10.1300/J115v22n02_10.
Kouhpayehzadeh J, Baradaran H, Soltani Arabshahi K, Knill-Jones R: Clinical teachers' attitudes toward the efficacy of evidence-based medicine workshop and self-reported ability in evidence-based practice in Iran. The Journal of Continuing Education in the Health Professions. 2006, 26: 210-214. 10.1002/chp.72.
Scherrer CS: An evaluation of a collaborative model for preparing evidence-based medicine teachers. J Med Libr Assoc l. 2006, 94 (2).
Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer H-H, Kunz R: Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002, 325: 1338-1341. 10.1136/bmj.325.7376.1338.
Yousefi-Nooraie R, Rashhidian A, Keating JL, Schonstein E: Teaching evidence-based practice: the teachers consider the content. J Eval Clin Pract. 2007, 13: 569-75. 10.1111/j.1365-2753.2007.00885.x.
Aiyer MK, Dorsch : The transformation of an EBM curriculum: a 10-year experience. Medical Teacher. 2008, 30: 377-83. 10.1080/01421590701881632.
Ribeiro O: Developing criteria for assessing the impact of a trainer training program in a health care setting: from theory to practice. American Society for Training & Development. 2006
Hutchinson L: Evaluating and researching the effectiveness of educational interventions. BMJ. 1999, 318: 1267-1269.
Khan KS, Awonuga AO, Dwarakanath LS, Taylor R: Assessments in evidence-based medicine workshops: Loose connection between perception of knowledge and its objective assessment. Medical Teacher. 2001, 23: 92-94. 10.1080/01421590150214654.
Fenstermacher GD, Richardson V: On making determinations of quality in teaching. Teachers College Record. 2005, 107: 186-213. 10.1111/j.1467-9620.2005.00462.x.
Darling-Hammond L, Youngs P: Defining "highly qualified teachers": What does "scientifically-based research" actually tell us?. Educational Researcher. 2002, 9: 13-25. 10.3102/0013189X031009013.
Naftulin DH, Ware JE, Donnelly FA: The doctor Fox lecture: a paradigm of educational seduction. Journal of Medicine Education. 1973, 48: 630-635.
Wilkes M, Bligh J: Evaluating educational interventions. BMJ. 1999, 318.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/10/64/prepub
The authors declare that they have no competing interests.
KSK, ST, BWM and JW conceived the project. AK, EG and KK performed literature search, study selection and data extraction. All authors participated in the development of the manuscript and provided input into the final version.
Electronic supplementary material
Additional file 1: Search strategy for identification of articles for the review of assessment methods in EBM TTT courses. (DOCX 13 KB)
Additional file 2: Details of primary articles included in the review of assessment methods of EBM teaching the teachers courses(DOCX 13 KB)
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Walczak, J., Kaleta, A., Gabryś, E. et al. How are "teaching the teachers" courses in evidence based medicine evaluated? A systematic review. BMC Med Educ 10, 64 (2010). https://doi.org/10.1186/1472-6920-10-64
- Assessment Tool
- Continue Professional Development
- Develop Assessment Tool
- Lifelong Learn Programme
- Effective Teacher Education