This article has Open Peer Review reports available.
The effectiveness of a clinically integrated e-learning course in evidence-based medicine: A cluster randomised controlled trial
- Regina Kulier1Email author,
- Sjors FPJ Coppus2, 3,
- Javier Zamora4, 14,
- Julie Hadley5,
- Sadia Malick5,
- Kausik Das6,
- Susanne Weinbrenner7,
- Berrit Meyerrose7,
- Tamas Decsi8,
- Andrea R Horvath9,
- Eva Nagy9,
- Jose I Emparanza10,
- Theodoros N Arvanitis1,
- Amanda Burls1,
- Juan B Cabello10,
- Marcin Kaczor11,
- Gianni Zanrei12,
- Karen Pierer13,
- Katarzyna Stawiarz11,
- Regina Kunz13,
- Ben WJ Mol2, 3 and
- Khalid S Khan1, 5
© Kulier et al; licensee BioMed Central Ltd. 2009
Received: 10 December 2008
Accepted: 12 May 2009
Published: 12 May 2009
To evaluate the educational effects of a clinically integrated e-learning course for teaching basic evidence-based medicine (EBM) among postgraduates compared to a traditional lecture-based course of equivalent content.
We conducted a cluster randomised controlled trial in the Netherlands and the UK involving postgraduate trainees in six obstetrics and gynaecology departments. Outcomes (knowledge gain and change in attitude towards EBM) were compared between the clinically integrated e-learning course (intervention) and the traditional lecture based course (control). We measured change from pre- to post-intervention scores using a validated questionnaire assessing knowledge (primary outcome) and attitudes (secondary outcome).
There were six clusters involving teaching of 61 postgraduate trainees (28 in the intervention and 33 in the control group). The intervention group achieved slightly higher scores for knowledge gain compared to the control, but these results were not statistically significant (difference in knowledge gain: 3.5 points, 95% CI -2.7 to 9.8, p = 0.27). The attitudinal changes were similar for both groups.
A clinically integrated e-learning course was at least as effective as a traditional lecture based course and was well accepted. Being less costly than traditional teaching and allowing for more independent learning through materials that can be easily updated, there is a place for incorporating e-learning into postgraduate EBM curricula that offer on-the-job training for just-in-time learning.
Trial registration number: ACTRN12609000022268.
Evidence-based medicine (EBM) requires healthcare professionals to engage with contemporaneous research evidence in clinical decision making. To achieve this, EBM curricula need to inculcate amongst learners the skills to acquire, assess and apply new knowledge in the clinical setting. However, training programs to improve evidence-based decision making have generally not been robustly evaluated. There has been much debate about the various EBM teaching and learning methods, but now there is consensus that best educational practice should be clinically integrated. It should result not only in improvements in knowledge and appraisal skills, but also in attitudes and behaviour, which in the end leads to improved practice.
Various models of teaching and learning strategies have been described, taking into account the different needs of adult learners, such as self-directed and problem-centred learning. 
Overview of the clinically integrated e-learning course compared to lecture based course in evidence-based medicine (EBM)
Clinically integrated e-learning course and Lecture based course
Aim: To familiarise course participants with evidence based medicine (EBM) basics
Target participants: Health professionals in a clinical setting.
Upon the completion of the course, participants should be competently able to:
generate structured questions arising from clinical problems in practice
search relevant literature, identifying systematic reviews wherever possible
assess the quality (validity) of systematic reviews and primary research included within them
assess the applicability of research findings in clinical practice
effectively implement the output from above activities into clinical practice
Five models provide learning materials at http://www.ebm-unity.org
Module 1: Asking clinical questions
Module 2: Searching the evidence
Module 3: Critical appraisal of systematic reviews (and their constituent studies)
Module 4: Applicability of the evidence to the patient
Module 5: Implementation of evidence into practice
Multiple choice questions to test knowledge and questionnaire to test attitudes
Clinically integrated e-learning course
Lecture based course
Knowledge needs identification in the clinical setting
Traditional lecture-based course using the power point slides from the e-learning modules
Participants to pursue independent study by using the e-learning modules
Lectures presented during 2–4 sessions
Interaction with facilitator throughout the course
Interaction with the tutor only during lecture
We conducted a cluster randomised controlled trial  to compare the clinically integrated e-learning EBM course to a lecture-based course for its effect on knowledge and attitudes after obtaining approval from the relevant authorities. We chose this approach to avoid the risk of contamination, i.e. the inadvertent delivery of intervention to members of the control group, associated with individual randomisation in educational research. Random allocation sequence was generated by computer to either intervention or control group and was stratified by country. The trial procedures were tested for feasibility in Switzerland in different clinical departments, after which the study was conducted in four centres in the Netherlands and two centres in the United Kingdom between August and December 2007. The participants were obstetrics and gynaecology trainees in clinical teaching hospitals who did not rotate between clusters during the study period. The trainees were all junior medical doctors who had not previously received formal EBM teaching in their postgraduate training. All trainees provided consent for use of their data anonymously at the start of the trial.
The clinically integrated course consisted of five modules, each comprising self-directed e-learning components and clinically related activities, under the guidance of a facilitator (table 1). The curriculum http://www.ebm-unity.org, described in detail elsewhere, was delivered at each site over a 4–6 week time period. In the control group, the material covered in the e-learning module was presented by a tutor during classical lecture-based teaching sessions over the same time period. The tutors presented from the same power point slides used in the intervention group. We assessed gain in knowledge and change in attitude towards EBM by comparing pre and post intervention scores. We adapted previously validated questionnaires for measurements of knowledge and attitudes which is described in detail elsewhere. [4, 9–11] We adapted these questions to match the content and learning objectives of our course. The maximum possible knowledge score was 62 points. Responses to questions about attitude towards EBM were possible on a five point Likert-scale, ranging from 'strongly agree' to 'strongly disagree'.
Data were collected online for the intervention group and on paper for the control group. Before starting the course, trainees in both groups completed all knowledge and attitude questionnaires. Post-course questions were completed immediately after each lecture in the control group. In the intervention group, trainees could complete the questions in their own time after completing the various modules of the course. Additional materials were not allowed to be used during the exams. Attitude questionnaires were re-administered at the end of the course in both groups.
Responses to the knowledge questionnaires were scored and between group comparisons were made taking into account the cluster design of the study. Characteristics of individual doctors in the same cluster, as well as their scores, are likely to be correlated. This correlation should be taken into account in the analyses. Generalized estimating equations (GEE) allow to extend linear models to take into account the correlation between individuals in the same cluster. For a cluster randomized trial, this matrix is assumed to be exchangeable, i.e. all correlations between individuals in the same cluster are the same. The proposed model could also be extended to adjust for individual covariates at baseline. Thus, in our model, the dependent variable was the final score and the intervention and baseline scoring were the independent variables. Attitudinal gain was defined as any change of whatever magnitude towards a more positive attitude towards EBM as measured with a Likert scale. All analyses were performed using Stata (StataCorp. 2007. Stata Statistical Software: Release 10. College Station, TX: StataCorp LP).
The main findings of our study showed that both teaching approaches lead to improvement in basic EBM knowledge. The clinically integrated e-learning course produced slightly better scores compared to the classical teaching approach; however, this difference did not reach statistical significance. Attitudinal gains did not differ between the groups. Although further evidence is needed, one could hypothesize that equivalent performance of the e-learning compared to the lecture based EBM course justifies its implementation from an economic point of view.
While previous research has shown that computer based EBM teaching is equivalent to face-to-face teaching,  there has been a lack of studies evaluating educational provision in the workplace via e-learning fitted around doctors' daily practice. A previous study in complementary and allied health had shown that a tailor-made web-based learning experience in EBM was educationally useful, but this was not a randomised evaluation.  Contamination of members of the control group in an individual randomised trial tends to shift their outcome in the same direction as the outcome in the intervention group. Thus the effectiveness of the intervention, estimated as the ratio or difference between intervention and control groups, tends to be underestimated biasing trial results towards the null.  We used a cluster design that eliminated contamination and employed previously validated and pre-tested tools for outcome measures, thus increasing the validity of our results.
Several issues may raise concern about the interpretation of our findings. Trainees in both groups showed high baseline knowledge for all modules, which left only a small room for improvement in the scores. The pre course scores may have been high due to a pre-existing EBM culture within the participating sites. The modest sample size may have contributed to a limitation in statistical power to detect a small improvement in knowledge between the groups. A small proportion in both groups had shown lower scores post course compared to pre course, but such an observation has also been made in other studies  and might be attributed to cognitive test anxiety.  Post-course knowledge assessments had been conducted immediately after the lectures in the control group. This may have put the intervention group at a disadvantage as delayed and unsupervised knowledge assessment may have influenced the outcome adversely. Despite these concerns, on average, the intervention group had a tendency towards better performance. We are therefore confident that our findings merit consideration.
Previous studies have shown the value of computer based learning. [12, 16–18] Since the costs of delivering e-learning courses after initial investment are small, this may also provide a less costly solution for EBM training than lecture based courses. Moreover, e-learning allows for independent and flexible study, with the possibility of a standardised provision that can be easily updated. The trial's generalisability may be considered limited as participating centres were all obstetrics and gynaecology departments. However, the results of the pilot testing conducted in different medical departments in Switzerland showed that the course was adaptable to different medical specialties. This was further reinforcement of the finding of adaptability across languages and settings in our earlier study. 
We conclude that the clinically integrated e-learning course for basic EBM proved to be at least as effective as a conventional lecture based course and merits consideration for incorporation in on-the-job training for just-in-time learning. Further research should address changes in behaviour after long term follow-up.
- Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn't. BMJ. 1996, 312: 71-72.View ArticleGoogle Scholar
- Khan KS, Coomarasamy A: A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ. 2006, 6: 59-10.1186/1472-6920-6-59.View ArticleGoogle Scholar
- Kaufman DM: Applying educational theory in practice. BMJ. 2003, 326: 213-216. 10.1136/bmj.326.7382.213.View ArticleGoogle Scholar
- Coppus SF, Emparanza JI, Hadley J, Kulier R, Weinbrenner S, Arvanitis TN, et al: A clinically integrated Curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU-EBM project. BMC Med Educ. 2007, 7: 46-10.1186/1472-6920-7-46.View ArticleGoogle Scholar
- Kulier R, Hadley J, Weinbrenner S, Meyerrose B, Decsi T, Horvath AR, et al: Harmonising evidence-based medicine teaching: a study of the outcomes of e-learning in five European countries. BMC Med Educ. 2008, 8: 27-10.1186/1472-6920-8-27.View ArticleGoogle Scholar
- Harden RM: Ten questions to ask when planning a course or curriculum. Med Educ. 1986, 20: 356-365. 10.1111/j.1365-2923.1986.tb01379.x.View ArticleGoogle Scholar
- Donner A, Klar N: Cluster randomization trials in epidemiology: Theory and application. Journal of Statistical Planning and Inference. 2008, 42: 37-56. 10.1016/0378-3758(94)90188-0.View ArticleGoogle Scholar
- Keogh-Brown MR, Bachmann MO, Shepstone L, Hewitt C, Howe A, Ramsay CR, et al: Contamination in trials of educational interventions. Health Technol Assess. 2007, 11.Google Scholar
- Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R: Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002, 325: 1338-1341. 10.1136/bmj.325.7376.1338.View ArticleGoogle Scholar
- Ramos KD, Schafer S, Tracz SM: Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003, 326: 319-321. 10.1136/bmj.326.7384.319.View ArticleGoogle Scholar
- Taylor R, Reeves B, Mears R, Keast J, Binns S, Ewings P, et al: Development and validation of a questionnaire to evaluate the effectiveness of evidence-based practice teaching. Med Educ. 2001, 35: 544-547. 10.1046/j.1365-2923.2001.00916.x.View ArticleGoogle Scholar
- Davis J, Chryssafidou E, Zamora J, Davies D, Khan K, Coomarasamy A: Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: a randomised controlled trial. BMC Med Educ. 2007, 7: 23-10.1186/1472-6920-7-23.View ArticleGoogle Scholar
- Hadley JA, Davis J, Khan KS: Teaching and learning evidence-based medicine in complementary, allied, and alternative health care: an integrated tailor-made course. J Altern Complement Med. 2007, 13: 1151-1155. 10.1089/acm.2007.0512.View ArticleGoogle Scholar
- Patel AA, Patel DD: Education through multimedia among Agricultural Diploma school students: An impact study. International Journal of Education and Development using Information and Communication Technology. 2006, 2: 4-10.Google Scholar
- Cassady JC, Johnson RE: Cognitive Test Anxiety and Academic Performance. Contemporary Educational Psychology. 2002, 27: 270-295. 10.1006/ceps.2001.1094.View ArticleGoogle Scholar
- Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM: Internet-based learning in the health professions: a meta-analysis. JAMA. 2008, 300: 1181-1196. 10.1001/jama.300.10.1181.View ArticleGoogle Scholar
- Coomarasamy A, Khan KS: What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ. 2004, 329: 1017-10.1136/bmj.329.7473.1017.View ArticleGoogle Scholar
- Greenhalgh T: Computer assisted learning in undergraduate medical education. BMJ. 2001, 322: 40-44. 10.1136/bmj.322.7277.40.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/9/21/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.