A clinically integrated curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU-EBM project
BMC Medical Education volume 7, Article number: 46 (2007)
Over the last years key stake holders in the healthcare sector have increasingly recognised evidence based medicine (EBM) as a means to improving the quality of healthcare. However, there is considerable uncertainty about the best way to disseminate basic knowledge of EBM. As a result, huge variation in EBM educational provision, setting, duration, intensity, content, and teaching methodology exists across Europe and worldwide. Most courses for health care professionals are delivered outside the work context ('stand alone') and lack adaptation to the specific needs for EBM at the learners' workplace. Courses with modern 'adaptive' EBM teaching that employ principles of effective continuing education might fill that gap. We aimed to develop a course for post-graduate education which is clinically integrated and allows maximum flexibility for teachers and learners.
A group of experienced EBM teachers, clinical epidemiologists, clinicians and educationalists from institutions from eight European countries participated. We used an established methodology of curriculum development to design a clinically integrated EBM course with substantial components of e-learning. An independent European steering committee provided input into the process.
We defined explicit learning objectives about knowledge, skills, attitudes and behaviour for the five steps of EBM. A handbook guides facilitator and learner through five modules with clinical and e-learning components. Focussed activities and targeted assignments round off the learning process, after which each module is formally assessed.
The course is learner-centred, problem-based, integrated with activities in the workplace and flexible. When successfully implemented, the course is designed to provide just-in-time learning through on-the-job-training, with the potential for teaching and learning to directly impact on practice.
There is an increasing imbalance between the exponential growth of medical knowledge and the opportunities available to average clinicians to read, assimilate and apply this information to improve health care. Estimates suggest that 27 kg of guidelines, 3,000 new papers, 1,000 new indexed Medline articles, and 50 new randomised clinical trials (RCTs) are published each day. However, clinicians only manage to spend one hour a week to digest this huge amount of information . Skills to manage knowledge translation efficiently and to distinguish useful evidence from less useful information have become of paramount importance . Teaching programs in evidence based medicine (EBM) face the challenge to impact on knowledge management and transfer skills among health care workers in order to employ the best available evidence in clinical decisions.
EBM is the integration of up-to-date patient-oriented valid research into clinical decision making by doctors and patients. "Good doctors use individual clinical expertise, the best available external evidence as well as patient preferences, and neither alone is enough. Without current best evidence, practice risks becoming rapidly out of date, to the detriment of patients." For EBM to be practiced conscientiously, professional qualifications system within health care need to integrate the concepts of problem based learning  and lifelong learning . At present, this objective is not yet well achieved in many countries.
Currently, EBM is mainly taught through courses, conferences, workshops, journal clubs or educational meetings, so called "stand alone" courses that are insufficiently integrated into daily clinical practice and clinical postgraduate training . Thereby, physicians go without important opportunities to incorporate the day-to-day problems of real patients in their learning of evidence-based practice. A recent systematic review showed that "stand alone" education improved basic knowledge of EBM, but evidence for improving practice was lacking . For successful implementation in practice, EBM knowledge would need to result in skills, attitudes and appropriate changes in behaviour [8–11]. An improvement on each of these four domains can best be achieved, when courses are integrated into daily clinical routine practice leading to just-in-time learning through on-the-job training [7, 12].
Increasingly, it is required that health care decisions are based on sound evidence . Experts agree that EBM should be a mandatory skill for postgraduate training and continuing medical education . However, EBM remains a difficult-to-teach subject and a joint agreed educational approach within many European countries and certainly across Europe is lacking. In 2005 the Leonardo da Vinci Community Vocational Training Action Programme for life long learning, funded by the European Union, provided a grant for a pilot project to fill this gap in medical professional development.
We report on this initiative named the European Union (EU)-EBM project that aspires to develop a clinically integrated EBM course. In a pilot project, we designed and developed the first teaching unit, with emphasis to its promotion and piloting across the European health care sector and beyond. This paper describes the process of curriculum development along with its current results.
The goal of the curriculum is to develop a postgraduate course integrated into daily clinical practice allowing a maximum of flexibility to learners.
A curriculum committee consisting of experts in the field of EBM, clinical epidemiologists, clinicians and educationalists from the participating countries commenced the work. We based the design and development of the EBM curriculum on an educationally sound methodology  including the following domains:
identification of EBM needs in each partner country
formulation of the aims, objectives and learning outcomes of the curriculum
development and organisation of the content of the curriculum
development of the teaching methods
definition of the educational strategy and educational environment
definition of the assessment strategy
communication of the curriculum to learners
the overall management of the process.
In addition, an independent European Steering Committee provided input into the content, educational approach, applicability and sustainability, giving advice at critical stages in the curriculum development and thereby assured external validation and triangulation of the syllabus prepared.
Identification of EBM needs
First we identified specific needs of learners, which could direct the curriculum development. Definitions needed to take into account differences in levels of EBM education and the variety of teaching and learning opportunities across European health care systems. This was mainly done by an in-depth discussion among the participating experts at the beginning of the project, who had a good overview of the ongoing activities in their countries, with input from the external steering committee. This process helped to identify definable educational goals for the documented learning needs. During meetings, consensus was achieved in a Delphic fashion  on the need for an initial course dealing with systematic reviews of the effectiveness of interventions, and with teaching tuned to the basics of EBM to make a standardised introduction to practicable EBM.
The aims, objectives and learning outcomes
The aim was to familiarise course participants with EBM basics to help them incorporate evidence from systematic reviews on therapeutic interventions into daily clinical practice. It was stipulated that the course should be clinically integrated with a large component of e-learning. To meet these objectives we aimed at short individual sessions to allow learning at the worksite during a short break, with the e-learning modules structured in a way they can be easily interrupted and taken up again. The following learning objectives and outcomes were identified: "After completion of the course, participants should first be able to generate structured questions arising from problems in their own clinical practice in PICO (population, intervention, comparator and outcome) format . Although other formats exists [17–19] we felt that this was the one most well-known and most easy to understand design for a course tuned to basic EBM teaching. Secondly, they should be able to search for relevant literature, aiming for and identifying systematic reviews wherever possible. Third, they should be able to assess the quality (validity) of systematic reviews and the primary research included within them. Fourth, they should be able to assess the applicability and generalisability of the research findings to their own clinical practice. Finally, they should demonstrate their understanding of strategies to effectively implement new research findings into clinical practice, including knowledge of barriers that often block such implementation in clinical practice [1, 20]." These key issues guided the subsequent steps in curricular development.
The curricular content
The curriculum is subdivided into five modules each of which addresses core competencies in evidence-based practice.
These modules are:
Asking and framing clinical questions.
Searching for the evidence
3a. Critical appraisal of primary randomised controlled trials
3b. Measures of effectiveness
3c. Critical appraisal of systematic reviews of intervention studies
Applicability of the evidence to the patient
Implementation of the evidence into practice
Organisation of the content and teaching methods
At the beginning of the course, learners and facilitators receive a handbook with the overall aims and objectives of the curriculum and modules, relevant additional methodological and clinical papers, an outline of the teaching, learning and assessment strategy, and its timetable. The 5 modules of the course are delivered sequentially over a period of 5 weeks. Order and sequence of the teaching sessions ensure that the prerequisites and the basic content appear first while more advanced content appears later. We refer to Table 1 for a comprehensive overview of the curriculum.
Each of the sessions integrates a combination of teaching methods: (Figure 1):
A facilitator (e.g. the supervising consultant) plays a central role in the clinically integrated curriculum by acting as a moderator in the learning process of the learner, by identifying learning opportunities encountered by the learners in daily care for patients and by directing appropriate use of learning resources in a clinical setting. This could be done during ward-rounds, daily meetings when shifts are transferred, in fact the facilitator can point out learning opportunities in every patient-doctor encounter.
Self-directed, independent e-learning via the Internet or CD-ROM. These sessions (one or more per module) are structured such that a learner can work them through in 10 to 20 minutes. This allows maximum flexibility to the learner and easy access from multiple sites, both inside and outside the hospital, which addresses the needs of a busy clinician. The e-learning materials consist of slides and written scripts; a talking head which covers the content of the scripts and guides the lecture; play, pause and skip options; and hyperlinks to main sections within the sessions. An example of one of the e-learning modules is shown in Figure 2. Sessions can be accessed multiple times if necessary. These e-learning sessions aim to provide basic understanding of the key concepts covered in each module.
A variety of minor activities where learners perform practical tasks and thereby transfer their knowledge into skills. The learner is expected to frame five clinical questions according to the PICO format from problems encountered in daily practice, to develop two structured search strategies in Pubmed, to appraise a randomised clinical trial on the topic on several validity items, to identify the (dis)similarity of their clinical patient with the patients included in the systematic review, and to read an essay on barriers that exist when implementing EBM in daily clinical practice.
Each module ends with an assignment that learners hand in to the facilitator, such as the search used to identify a systematic review for one of PICO questions formulated in the first module, or a paper on the learner's deliberations on how to implement the found evidence in their own clinical setting. By providing feedback on learning exercises/assignments, these assignments can act as a starting point for a one-to-one interaction between learner and facilitator or a small group discussion in the resident's regular teaching session. Before moving to the next module, learners will finish each module with a multiple choice questionnaire that covers the content of that specific module. All completed assignments contribute to the learner's documented portfolio.
Educational strategy and educational environment
Many educational models have been developed on which teaching strategies can be based . We chose to incorporate our EBM curriculum within the SPICES framework, in which six educational strategies are represented as a gradual continuum, contrasting the more innovative SPICES model with more traditional teaching strategies . As the educational environment influences participants' motivation, learning and thinking, the curriculum aims to shift from a teacher-centred orientation to a participant-centred approach [14, 22]. We aim for an ethos in which facilitators and participants regard each other as professional colleagues for whom teaching and learning works as a two-ways process. A carefully designed learner's handbook guides participants through the curriculum systematically and the e-learning material allowing learners to advance at their own time, speed and needs and to organise the learning easily around their clinical responsibilities. Appraising the applicability of the evidence in their own setting to their own patients supports the clinical focus of learning in a problem-based fashion. Although our curriculum contains fixed learning objectives that have been precisely matched against the learning materials, for most modules elective course material is provided. Whether or not learners will access additional materials will be largely driven by students' intrinsic motivation and specific needs. The full educational strategy is summarised in Figure 3. As part of the pilot project, the impact of our curriculum and its implementation in the educational environment is studied by using an adapted version of the DREEM questionnaire .
The course will be piloted formally on ten doctors per participating country (in total over 70 doctors). The curriculum assessment concerns evaluation of participants' knowledge, skills and attitudes through formative and summative assessments. For this, a multiple choice questionnaire (MCQ) was modelled according to the validated Berlin Questionnaire  that evaluates the learner's knowledge and skills by following closely the learning objectives of each module. In addition to this, as part of the pilot phase, an existing validated tool for attitudes towards EBM  -slightly adapted for our purpose- will assess learners' attitudes towards the clinically integrated e-learning course and EBM in general. Pre-course attitudes will be compared with the learners' attitudes after completion of the course. In addition, during the pilot phase, we will collect participants' and facilitators' feedback on the course by using semi-structured qualitative interviews, the results of which are used to identify lack of clarity or weaknesses in course material. Information of the level of difficulty, the time and preparation spend by both learners and facilitator is collected, and the level of comfort of facilitators with facilitating the course is explored. All these results will help us to refine the curriculum and to match learner's needs even more precisely.
Management of the process
To assure sustainability, an independent steering committee of clinical epidemiologists, EBM experts and educational experts from countries both participating and non-participating in the partnership overlooked the development of the curriculum, content of the teaching materials, its assessment and the evaluation of the pilot project. The committee gave advice to strategic questions addressing implementation and dissemination across Europe and provided external feedback to the group.
On national levels the project aims to obtain certification by continuous medical education agencies, in line with the Copenhagen Declaration (2002) of the European Union , which is currently preparing a European Credit (Transfer) System for Vocational Education and Training (ECVET). ECVET aims to facilitate mobility in vocational education and training, allowing young people to transfer learning results from one country to another in concordance with national regulations .
The EU-EBM curriculum presented in this paper targets postgraduate training and continuing education, using a clinically integrated e-learning methodology. To our knowledge, clinically integrated teaching of EBM is currently not provided in Europe. Furthermore, existing courses do not have a comparable format with continuous repetitive learning over a longer period, which includes small activities and individualised assignments to acquire skills and to deepen knowledge, standardised assessments at the end of each module and special emphasis on implementation of evidence in practice. Furthermore, one of the main innovations here is to get European countries to work together in order to harmonize an integrated teaching of EBM, which has not been subject of any harmonization effort before.
The main purpose of the curriculum is to provide doctors with the basic skills needed to practice and implement evidence-based therapy. For this, the course covers the 5 key steps of evidence based practice, namely framing the clinical question, searching for the evidence, appraisal and interpretation of the evidence, applying the evidence to the patient and implementing the evidence into practice, using problems encountered in daily patient care. An e-learning platform  or access through CD-Rom ensures timely and immediate access to the necessary methodological knowledge and background materials. The development, piloting and certification of a basic clinically integrated course in EBM offers the learner the opportunity for a standardised acquisition of EBM skills, promotes life long learning, and increases the mobility of doctors within a unified region such as the European Union.
A distinct feature of the EU-EBM curriculum is the low threshold approach to facilitate integrated EBM teaching and thereby addressing specific problems of the target group. Teaching basic concepts in short learning units and allowing maximum flexibility in the location of the learning takes into account the timeframe and working patterns of busy clinicians. The course separates the practice of EBM in the day-to-day patient encounter (framing questions from patients' problems, identifying the literature and applying the evidence to the individual patient) from the acquisition of methodological knowledge (critical appraisal of clinical research) which is mainly taught on the e-learning platform. An EBM enthusiastic senior clinician can act as a moderator in this learning process, but is not expected to teach the methodology of EBM. This division of responsibilities fosters the spread of EBM-teaching and learning to clinical settings with strong interest for EBM but with a lack of specialists teaching the methodology of critical appraisal. Inspired learners will receive additional reading material and guidance on how to advance their skills. Furthermore, groups of residents could pass the course simultaneously as part of their in-house training. We envision that this joint learning experience might stimulate constructive critical reflection on current clinical practice beyond the specific course content, initiating a new culture of discussion and EBM-practice within a department or unit. Last but not least, the translation of the curriculum in French, German, Spanish, Hungarian and Polish should help non-native English speakers to learn new concepts in their own language and ensure greater accessibility of the learning materials.
The key aims of this project will need to be met by raising awareness of the importance of training in EBM throughout Europe as a means of enabling health care professionals to provide better care to patients  and ultimately encouraging the inclusion of the qualification developed into mainstream education. The dissemination systems in place such as the website , presentations on national conferences and workshops, word of mouth and publications in national language will allow access, discussion and dissemination of the results. The project partners started already to share the knowledge and experience from this project with their national networks of institutions that might benefit from the process. Many organisations and institutions look for "easy to use" learning programs in EBM that they can integrate -with limited resources- in their local working environment. We further encourage the incorporation of this course in existing educational lessons offered by other institutions, which will then hopefully form the established network for an exchange of training practices and experiences. Dissemination to good educational practice when successful can help to create a level playing field for mobility of doctors and other clinical professions.
So far, in this pilot project, we only covered one unit of EBM, namely systematic review of therapeutic effectiveness, due to time and financial constraints. Furthermore, the current program is designed for medical doctors. However, numerous organisations of allied health professionals already expressed interest in this course as did consumer organisations and patient self-help groups. We need testing of the existing course in those groups and modify it according to their learning needs. Furthermore, the course is so far only available in six different languages. Additional translation in other European and non-European languages might be warranted. The curriculum assessment currently contains evaluation of participants' knowledge, skills, and attitudes. Although we acknowledge that behavioural changes are essential to fully implement life long learning, measurement of such changes require longer follow-up than is possible within the current duration of this pilot-project.
Once fully implemented, the ultimate outcome of this pilot project will be a European qualification in EBM, which will be used by doctors, hospitals, professional bodies responsible for postgraduate qualifications and continuous medical education. In the long term this project has the potential to benefit the general public as EBM will contribute to a more transparent healthcare system, better-informed patients and a generally better informed society, which is what EBM is all about.
The EU-EBM project has shown that it is possible to harmonize EBM teaching across Europe, with the development of a clinically integrated course based on e-learning technologies. The course is designed to provide just-in-time learning through on-the-job-training, with the potential for teaching and learning to directly impact on practice.
Sackett DL, Richardson WS, Rosenberg W, Haynes RB: Evidence-Based Medicine. 1997, London, England: Churchill Livingstone Inc
Dawes M, Summerskill W, Glaziou P, Cartabellotta A, Martin J, Hopayian K, Porzsolt F, Burls A, Osborne J: Sicily statement on evidence-based practice. BMC Med Educ. 2005, 5 (1): 1-10.1186/1472-6920-5-1.
Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS: Evidence Based Medicine: what it is and what it isn't. BMJ. 1996, 312: 71-72.
Smits PBA, Verbeek JHAM, de Buisonje CD: Problem-based learning in continuing medical education: a review of controlled evaluation studies. BMJ. 2002, 324: 153-156. 10.1136/bmj.324.7330.153.
Harden RM, Laidlaw JM: Effective continuing education: the CRISIS criteria. Med Educ. 1992, 26: 408-422.
Green ML: Evidence-based medicine training in graduate medical education: past, present and future. J Eval Clin Pract. 2000, 6: 121-38. 10.1046/j.1365-2753.2000.00239.x.
Coomarasamy A, Khan KS: What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ. 2004, 329: 1017-10.1136/bmj.329.7473.1017.
Shaneyfelt T, Baum KD, Bell D, Feldstein D, Houston TK, Kaatz S, Whelan C, Green M: Instruments for evaluating education in evidence-based practice: a systematic review. JAMA. 2006, 296: 1116-1127. 10.1001/jama.296.9.1116.
Bloom BS: Taxonomy of Eucational Objectives: Handbook I, Cognitive Domain. 1956, New York, NY: D Mackay, 7-8.
Hutchinson L: Evaluating and researching the effectiveness of educational interventions. BMJ. 1999, 318: 1267-1269.
Khan KS, Coomarasamy A: A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ. 2006, 6: 59-10.1186/1472-6920-6-59.
Slawson DC, Shaughnessy AF: Teaching evidence-based medicine: should we be teaching information management instead?. Acad Med. 2005, 80: 685-689. 10.1097/00001888-200507000-00014.
Dubois C, Nolte E, McKee M: Human Resources for Health in Europe (European Observatory on Health Systems and Policies). 2005, London: Open University Press, 1
Harden RM: Ten questions to ask when planning a course or curriculum. Med Educ. 1986, 20: 356-365.
Kerr M: The Delphi Process. The Delphi Process 2002 City: Remote and Rural Areas Research Initiative. 2001, NHS in Scotland
Oxman AD, Sackett DL, Guyatt GH: Users' guides to the medical literature. I. How to get started. The Evidence-Based Medicine Working Group. JAMA. 1993, 270: 2093-2095. 10.1001/jama.270.17.2093.
The evidence-based medicine working group: User's guides to the medical literature: a manual for evidence-based clinical practice. 2002, American Medical Association
Richardson WS, Wilson MC, Nishikawa J, Hayward RS: The well-built clinical question: a key to evidence-based decisions. ACP Journal Club. 1995, 123: A12-A13.
Haynes RB, Sackett DL, Guyatt G, et al: Clinical Epidemiology: how to do clinical practice research. 2006, Philadelphia, PA: Lippincott Williams & Wilkins, 3
Rosenberg W, Donald A: Evidence based medicine: an approach to clinical problem solving. BMJ. 1995, 310: 1122-1126.
Kaufman DM: ABC of learning and teaching in medicine: Applying educational theory in practice. BMJ. 2003, 326: 213-216. 10.1136/bmj.326.7382.213.
Harden R, Sowden S, Dunn WR: Some educational strategies in curriculum development: the SPICES model. Medical Education. 1984, 18: 284-297.
Till H: Identifying the perceived weaknesses of a new curriculum by means of the Dundee Ready Education Environment Measure (DREEM) Inventory. Med Teach. 2004, 26: 39-45. 10.1080/01421590310001642948.
Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R: Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002, 325: 1338-1341. 10.1136/bmj.325.7376.1338.
Taylor R, Reeves B, Mears R, Keast J, Binns S, Ewings P, Khan K: Development and validation of a questionnaire to evaluate the effectiveness of evidence-based practice teaching. Medical Education. 2001, 35: 544-547. 10.1046/j.1365-2923.2001.00916.x.
Copenhagen declaration. [http://ec.europa.eu/education/copenhagen/copenahagen_declaration_en.pdf]
European Union Evidence Based Medicine Unity. [http://www.ebm-unity.org]
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/7/46/prepub
The EU-EBM partnership consists of the following institutions from eight European countries:
• the University of Birmingham/Birmingham Women's Hospital (UK),
• Agency for Quality in Medicine (Germany),
• Critical Appraisal Skills Programme (CASP International) (UK),
• Critical Appraisal Skills Programme, Polska (CASPolska) (Poland),
• Critical Appraisal Skills Programme of Spain (CASPe) (Spain),
• Academic Medical Center, University of Amsterdam (Netherlands),
• TUDOR (Tudományos Orvoslás) Alapítvány (Hungary),
• Universitá Cattolica del Sacro Cuore (Italy),
• Basel institute of Clinical Epidemiology (Switzerland).
The steering committee members are:
• Paul Glasziou (Chair) – Centre for Evidence Based Medicine, University of Oxford (UK),
• Ahmet Metin Gulmezoglu – Department of Reproductive Health and Research, World Health Organisation (Switzerland),
• Anne Brice – University of Oxford (UK),
• Arri Coomarasamy – Guy's Hospital, London (UK),
• Francisco Pozo Rodriguez – Clinical Epidemiology Unit, University of Madrid (Spain),
• Marjukka Makela – Finnish Office for Health Care Technology Assessment (Finland),
• Nino Cartabellotta – GIMBE, Gruppo Italiano per la Medicina Basata sulle Evidenze, Bologna (Italy),
• Norbert Donner-Banzhoff – Universität Marburg (Germany),
• Peter Matzen – Hvidovre Hospital (Denmark),
• Patrick Bossuyt – Academic Medical Center, University of Amsterdam (Netherlands)
Bernard Burnand is gratefully acknowledged for translating the course materials in French.
The author(s) declare that they have no competing interests.
SC, JE, JH, BWM, RMK, RAK and KSK drafted the first version of the manuscript. All authors revised the manuscript and approved final submission for publication. All authors are member of the EU-EBM and were involved in the design, development and implementation of the course described.
About this article
Cite this article
Coppus, S.F., Emparanza, J.I., Hadley, J. et al. A clinically integrated curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU-EBM project. BMC Med Educ 7, 46 (2007). https://doi.org/10.1186/1472-6920-7-46