Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

The effectiveness of a clinically integrated e-learning course in evidence-based medicine: A cluster randomised controlled trial

  • Regina Kulier1Email author,
  • Sjors FPJ Coppus2, 3,
  • Javier Zamora4, 14,
  • Julie Hadley5,
  • Sadia Malick5,
  • Kausik Das6,
  • Susanne Weinbrenner7,
  • Berrit Meyerrose7,
  • Tamas Decsi8,
  • Andrea R Horvath9,
  • Eva Nagy9,
  • Jose I Emparanza10,
  • Theodoros N Arvanitis1,
  • Amanda Burls1,
  • Juan B Cabello10,
  • Marcin Kaczor11,
  • Gianni Zanrei12,
  • Karen Pierer13,
  • Katarzyna Stawiarz11,
  • Regina Kunz13,
  • Ben WJ Mol2, 3 and
  • Khalid S Khan1, 5
BMC Medical Education20099:21

https://doi.org/10.1186/1472-6920-9-21

Received: 10 December 2008

Accepted: 12 May 2009

Published: 12 May 2009

Abstract

Background

To evaluate the educational effects of a clinically integrated e-learning course for teaching basic evidence-based medicine (EBM) among postgraduates compared to a traditional lecture-based course of equivalent content.

Methods

We conducted a cluster randomised controlled trial in the Netherlands and the UK involving postgraduate trainees in six obstetrics and gynaecology departments. Outcomes (knowledge gain and change in attitude towards EBM) were compared between the clinically integrated e-learning course (intervention) and the traditional lecture based course (control). We measured change from pre- to post-intervention scores using a validated questionnaire assessing knowledge (primary outcome) and attitudes (secondary outcome).

Results

There were six clusters involving teaching of 61 postgraduate trainees (28 in the intervention and 33 in the control group). The intervention group achieved slightly higher scores for knowledge gain compared to the control, but these results were not statistically significant (difference in knowledge gain: 3.5 points, 95% CI -2.7 to 9.8, p = 0.27). The attitudinal changes were similar for both groups.

Conclusion

A clinically integrated e-learning course was at least as effective as a traditional lecture based course and was well accepted. Being less costly than traditional teaching and allowing for more independent learning through materials that can be easily updated, there is a place for incorporating e-learning into postgraduate EBM curricula that offer on-the-job training for just-in-time learning.

Trial registration

Trial registration number: ACTRN12609000022268.

Background

Evidence-based medicine (EBM) requires healthcare professionals to engage with contemporaneous research evidence in clinical decision making.[1] To achieve this, EBM curricula need to inculcate amongst learners the skills to acquire, assess and apply new knowledge in the clinical setting. However, training programs to improve evidence-based decision making have generally not been robustly evaluated. There has been much debate about the various EBM teaching and learning methods, but now there is consensus that best educational practice should be clinically integrated.[2] It should result not only in improvements in knowledge and appraisal skills, but also in attitudes and behaviour, which in the end leads to improved practice.[2]

Various models of teaching and learning strategies have been described, taking into account the different needs of adult learners, such as self-directed and problem-centred learning. [3]

However, traditionally, postgraduate education is generally delivered via lecture-based courses and workshops which do not directly exploit learning opportunities in the workplace. The EU-EBM Unity project was developed to address these shortcomings. [4] The project was conducted within the European Union's Leonardo da Vinci vocational training programme involving 11 European partners. [4, 5] We used established methodology to develop a learner-centred, problem-based course that employs e-learning to provide on-the-job training exploiting learning opportunities during the course of clinical activity. [46] In brief, the course consists of five modules, to be followed consecutively and is outlined in table 1. Each module consists of a self-directed e-learning part and practical assignments that have to be completed and discussed with the facilitator before moving on to the next module. Our piloting showed that this course can be adapted to different clinical settings in different languages and it is well accepted by the trainees and tutors. [5]. In this trial, we set out to evaluate the educational effectiveness of our clinically integrated e-learning course for teaching basic EBM to postgraduates compared to a traditional lecture based course of equivalent content in terms of knowledge gain and change in attitude towards EBM.
Table 1

Overview of the clinically integrated e-learning course compared to lecture based course in evidence-based medicine (EBM)

Clinically integrated e-learning course and Lecture based course

Aim: To familiarise course participants with evidence based medicine (EBM) basics

Target participants: Health professionals in a clinical setting.

Learning objectives:

Upon the completion of the course, participants should be competently able to:

   generate structured questions arising from clinical problems in practice

   search relevant literature, identifying systematic reviews wherever possible

   assess the quality (validity) of systematic reviews and primary research included within them

   assess the applicability of research findings in clinical practice

   effectively implement the output from above activities into clinical practice

E-learning modules:

Five models provide learning materials at http://www.ebm-unity.org

   Module 1: Asking clinical questions

   Module 2: Searching the evidence

   Module 3: Critical appraisal of systematic reviews (and their constituent studies)

   Module 4: Applicability of the evidence to the patient

   Module 5: Implementation of evidence into practice

Assessment:

Multiple choice questions to test knowledge and questionnaire to test attitudes

Clinically integrated e-learning course

Lecture based course

Learning/teaching methods

Knowledge needs identification in the clinical setting

Traditional lecture-based course using the power point slides from the e-learning modules

Participants to pursue independent study by using the e-learning modules

Lectures presented during 2–4 sessions

Interaction with facilitator throughout the course

Interaction with the tutor only during lecture

Methods

We conducted a cluster randomised controlled trial [7] to compare the clinically integrated e-learning EBM course to a lecture-based course for its effect on knowledge and attitudes after obtaining approval from the relevant authorities. We chose this approach to avoid the risk of contamination, i.e. the inadvertent delivery of intervention to members of the control group, associated with individual randomisation in educational research.[8] Random allocation sequence was generated by computer to either intervention or control group and was stratified by country. The trial procedures were tested for feasibility in Switzerland in different clinical departments, after which the study was conducted in four centres in the Netherlands and two centres in the United Kingdom between August and December 2007. The participants were obstetrics and gynaecology trainees in clinical teaching hospitals who did not rotate between clusters during the study period. The trainees were all junior medical doctors who had not previously received formal EBM teaching in their postgraduate training. All trainees provided consent for use of their data anonymously at the start of the trial.

The clinically integrated course consisted of five modules, each comprising self-directed e-learning components and clinically related activities, under the guidance of a facilitator (table 1). The curriculum http://www.ebm-unity.org, described in detail elsewhere,[4] was delivered at each site over a 4–6 week time period. In the control group, the material covered in the e-learning module was presented by a tutor during classical lecture-based teaching sessions over the same time period. The tutors presented from the same power point slides used in the intervention group. We assessed gain in knowledge and change in attitude towards EBM by comparing pre and post intervention scores. We adapted previously validated questionnaires for measurements of knowledge and attitudes which is described in detail elsewhere. [4, 911] We adapted these questions to match the content and learning objectives of our course. The maximum possible knowledge score was 62 points. Responses to questions about attitude towards EBM were possible on a five point Likert-scale, ranging from 'strongly agree' to 'strongly disagree'.

Data were collected online for the intervention group and on paper for the control group. Before starting the course, trainees in both groups completed all knowledge and attitude questionnaires. Post-course questions were completed immediately after each lecture in the control group. In the intervention group, trainees could complete the questions in their own time after completing the various modules of the course. Additional materials were not allowed to be used during the exams. Attitude questionnaires were re-administered at the end of the course in both groups.

Responses to the knowledge questionnaires were scored and between group comparisons were made taking into account the cluster design of the study. Characteristics of individual doctors in the same cluster, as well as their scores, are likely to be correlated. This correlation should be taken into account in the analyses. Generalized estimating equations (GEE)[9] allow to extend linear models to take into account the correlation between individuals in the same cluster. For a cluster randomized trial, this matrix is assumed to be exchangeable, i.e. all correlations between individuals in the same cluster are the same. The proposed model could also be extended to adjust for individual covariates at baseline. Thus, in our model, the dependent variable was the final score and the intervention and baseline scoring were the independent variables. Attitudinal gain was defined as any change of whatever magnitude towards a more positive attitude towards EBM as measured with a Likert scale. All analyses were performed using Stata (StataCorp. 2007. Stata Statistical Software: Release 10. College Station, TX: StataCorp LP).

Results

The total number of postgraduate trainees included in the analysis was 61, 28 in the intervention group and 33 in the control group (figure 1) recruited from six clusters (three clusters in each arm). The baseline knowledge scores were the same for the intervention and control groups and were generally high (43.3 points corresponding to 70% of the maximum possible scoring). Post-course scores were improved in both groups compared to baseline (figure 2). After adjusting for baseline knowledge, the intervention group outperformed the control group by 3.5 scoring points (95% CI -2.7 to 9.8) but this difference was not statistically significant (figure 2). In relative terms the difference was 5.6% (95% CI -4.4 to 15.8). About 18% of trainees in the intervention group and 27% in the control group had lower scores post course than pre course. Although a slightly higher proportion of participants in the control group showed an attitudinal gain towards EBM compared to the intervention group, the low number of participants did not allow to show any statistically significant difference (figure 3).
Figure 1

Flow chart of participants in the trial.

Figure 2

Pre-and post-course scores adjusted for baseline knowledge.

Figure 3

Attitudinal gain. (A): Original research is confusing (B) Study design is important in article selection (C) Evidence-based decision making is ' health care by numbers' (D) Contracts for health care professionals should include time taken away from patient care for reading and appraising the literature (E) I am confident that I can assess research evidence (F) Systematic reviews play a key role in informing evidence-based decision making.

Discussion

The main findings of our study showed that both teaching approaches lead to improvement in basic EBM knowledge. The clinically integrated e-learning course produced slightly better scores compared to the classical teaching approach; however, this difference did not reach statistical significance. Attitudinal gains did not differ between the groups. Although further evidence is needed, one could hypothesize that equivalent performance of the e-learning compared to the lecture based EBM course justifies its implementation from an economic point of view.

While previous research has shown that computer based EBM teaching is equivalent to face-to-face teaching, [12] there has been a lack of studies evaluating educational provision in the workplace via e-learning fitted around doctors' daily practice. A previous study in complementary and allied health had shown that a tailor-made web-based learning experience in EBM was educationally useful, but this was not a randomised evaluation. [13] Contamination of members of the control group in an individual randomised trial tends to shift their outcome in the same direction as the outcome in the intervention group. Thus the effectiveness of the intervention, estimated as the ratio or difference between intervention and control groups, tends to be underestimated biasing trial results towards the null. [8] We used a cluster design that eliminated contamination and employed previously validated and pre-tested tools for outcome measures, thus increasing the validity of our results.

Several issues may raise concern about the interpretation of our findings. Trainees in both groups showed high baseline knowledge for all modules, which left only a small room for improvement in the scores. The pre course scores may have been high due to a pre-existing EBM culture within the participating sites. The modest sample size may have contributed to a limitation in statistical power to detect a small improvement in knowledge between the groups. A small proportion in both groups had shown lower scores post course compared to pre course, but such an observation has also been made in other studies [14] and might be attributed to cognitive test anxiety. [15] Post-course knowledge assessments had been conducted immediately after the lectures in the control group. This may have put the intervention group at a disadvantage as delayed and unsupervised knowledge assessment may have influenced the outcome adversely. Despite these concerns, on average, the intervention group had a tendency towards better performance. We are therefore confident that our findings merit consideration.

Previous studies have shown the value of computer based learning. [12, 1618] Since the costs of delivering e-learning courses after initial investment are small, this may also provide a less costly solution for EBM training than lecture based courses. Moreover, e-learning allows for independent and flexible study, with the possibility of a standardised provision that can be easily updated. The trial's generalisability may be considered limited as participating centres were all obstetrics and gynaecology departments. However, the results of the pilot testing conducted in different medical departments in Switzerland showed that the course was adaptable to different medical specialties. This was further reinforcement of the finding of adaptability across languages and settings in our earlier study. [5]

Conclusion

We conclude that the clinically integrated e-learning course for basic EBM proved to be at least as effective as a conventional lecture based course and merits consideration for incorporation in on-the-job training for just-in-time learning. Further research should address changes in behaviour after long term follow-up.

Declarations

Authors’ Affiliations

(1)
The University of Birmingham, Edgbaston
(2)
Academic Medical Center, University of Amsterdam, Department of Obstetrics and Gynaecology
(3)
Academic Medical Center, University of Amsterdam, Department of Clinical Epidemiology and Biostatistics
(4)
Clinical Biostatistics Unit, Hospital Ramon y Cajal, Ctra Colmenar
(5)
Birmingham Women's Hospital, Metchley Park Road
(6)
Heart of England NHS Foundation Trust, Solihull Hospital
(7)
Agency for Quality in Medicine
(8)
Department of Paediatrics, University of Pécs
(9)
TUDOR, University of Szeged, Albert Szent-Gyorgyi Medical and Pharmacological Centre
(10)
CASPe (CASP Espana)
(11)
CASPolska
(12)
Universitá Cattolica del Sacro Cuore
(13)
Basel Institute for Clinical Epidemiology
(14)
CIBER Epidemiologia y Salud Publica (CIBERESP)

References

  1. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS: Evidence based medicine: what it is and what it isn't. BMJ. 1996, 312: 71-72.View ArticleGoogle Scholar
  2. Khan KS, Coomarasamy A: A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ. 2006, 6: 59-10.1186/1472-6920-6-59.View ArticleGoogle Scholar
  3. Kaufman DM: Applying educational theory in practice. BMJ. 2003, 326: 213-216. 10.1136/bmj.326.7382.213.View ArticleGoogle Scholar
  4. Coppus SF, Emparanza JI, Hadley J, Kulier R, Weinbrenner S, Arvanitis TN, et al: A clinically integrated Curriculum in Evidence-based Medicine for just-in-time learning through on-the-job training: The EU-EBM project. BMC Med Educ. 2007, 7: 46-10.1186/1472-6920-7-46.View ArticleGoogle Scholar
  5. Kulier R, Hadley J, Weinbrenner S, Meyerrose B, Decsi T, Horvath AR, et al: Harmonising evidence-based medicine teaching: a study of the outcomes of e-learning in five European countries. BMC Med Educ. 2008, 8: 27-10.1186/1472-6920-8-27.View ArticleGoogle Scholar
  6. Harden RM: Ten questions to ask when planning a course or curriculum. Med Educ. 1986, 20: 356-365. 10.1111/j.1365-2923.1986.tb01379.x.View ArticleGoogle Scholar
  7. Donner A, Klar N: Cluster randomization trials in epidemiology: Theory and application. Journal of Statistical Planning and Inference. 2008, 42: 37-56. 10.1016/0378-3758(94)90188-0.View ArticleGoogle Scholar
  8. Keogh-Brown MR, Bachmann MO, Shepstone L, Hewitt C, Howe A, Ramsay CR, et al: Contamination in trials of educational interventions. Health Technol Assess. 2007, 11.Google Scholar
  9. Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R: Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002, 325: 1338-1341. 10.1136/bmj.325.7376.1338.View ArticleGoogle Scholar
  10. Ramos KD, Schafer S, Tracz SM: Validation of the Fresno test of competence in evidence based medicine. BMJ. 2003, 326: 319-321. 10.1136/bmj.326.7384.319.View ArticleGoogle Scholar
  11. Taylor R, Reeves B, Mears R, Keast J, Binns S, Ewings P, et al: Development and validation of a questionnaire to evaluate the effectiveness of evidence-based practice teaching. Med Educ. 2001, 35: 544-547. 10.1046/j.1365-2923.2001.00916.x.View ArticleGoogle Scholar
  12. Davis J, Chryssafidou E, Zamora J, Davies D, Khan K, Coomarasamy A: Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: a randomised controlled trial. BMC Med Educ. 2007, 7: 23-10.1186/1472-6920-7-23.View ArticleGoogle Scholar
  13. Hadley JA, Davis J, Khan KS: Teaching and learning evidence-based medicine in complementary, allied, and alternative health care: an integrated tailor-made course. J Altern Complement Med. 2007, 13: 1151-1155. 10.1089/acm.2007.0512.View ArticleGoogle Scholar
  14. Patel AA, Patel DD: Education through multimedia among Agricultural Diploma school students: An impact study. International Journal of Education and Development using Information and Communication Technology. 2006, 2: 4-10.Google Scholar
  15. Cassady JC, Johnson RE: Cognitive Test Anxiety and Academic Performance. Contemporary Educational Psychology. 2002, 27: 270-295. 10.1006/ceps.2001.1094.View ArticleGoogle Scholar
  16. Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM: Internet-based learning in the health professions: a meta-analysis. JAMA. 2008, 300: 1181-1196. 10.1001/jama.300.10.1181.View ArticleGoogle Scholar
  17. Coomarasamy A, Khan KS: What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ. 2004, 329: 1017-10.1136/bmj.329.7473.1017.View ArticleGoogle Scholar
  18. Greenhalgh T: Computer assisted learning in undergraduate medical education. BMJ. 2001, 322: 40-44. 10.1136/bmj.322.7277.40.View ArticleGoogle Scholar
  19. Pre-publication history

    1. The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/9/21/prepub

Copyright

© Kulier et al; licensee BioMed Central Ltd. 2009

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Advertisement