Impact on learning of an e-learning module on leukaemia: a randomised controlled trial

e-learning is evolving in parallel with technological advances, and as a result is able to create more authentic learning experiences. e-learning resources have the potential to contribute to the goals of higher education, by supporting autonomous, life-long, student-centred learning. This can be achieved by creating a platform that is malleable for different types of learners and different ability levels while maintaining the same standard of information and accessibility for students and teachers [1].

In recent years, e-learning has become increasingly integrated into mainstream medical education, due to greater levels of acceptance by staff and increased expectations by students. As a consequence, medical students currently have access to an abundance of e-learning resources, and are often paralysed by the overwhelming amount of information available [2]. Furthermore, most such information is either too general or too detailed for the purposes of their studies. In contrast, e-learning that is appropriately structured and focused on topics known to be conceptually difficult [3] may potentially be of significant value to students.

In general, there is a dearth of reliable data regarding the efficacy of e-learning, as well as a lack of funding available to support long-term studies that monitor and evaluate the ongoing impact of e-learning innovations [4, 5]. This is compounded by highly variable methodology between studies, as well as the lack of widely accepted metrics to evaluate e-learning resources. All of this is exacerbated by the ethical difficulties of running randomised controlled trials to establish the efficacy of e-learning. Nevertheless, existing studies of e-learning for medical students in a variety of disciplines, such as those in anatomy [6], paediatrics [7, 8] and pathology [9], indicate that e-learning can have significant benefits.

Importantly, a meta-analysis of studies comparing e-learning with face-to-face teaching revealed that although e-learning is significantly better than no intervention, in most cases it is equivalent to ‘traditional’ teaching methods [10]. The logical extension of this finding is that research should now focus on comparing the efficacy of differing forms of e-learning, to determine the modes and contexts in which e-learning might be most useful [11].

Adequate integration of e-learning technology into the curriculum is an important factor in perceived efficacy for both students and teachers [12, 13]. Integration requires that an e-learning module has explicit learning objectives, which provide a high level of relevance and validity [14]. In addition, Wong and colleagues [15] suggest that e-learning is most engaging for students when it places learning in context, as well as drawing on prior learning.

Based on the above, we hypothesised that effective e-learning in medical education might be best achieved via curriculum-based modules that bring diverse concepts together in an authentic clinical context, while emphasising interactivity and feedback, as well as integrating with, and expanding on, prior learning.

At the University of New South Wales (UNSW), the six-year undergraduate medicine program is structured around the development of core graduate capabilities, which are considered fundamental for successful practice directly following graduation and throughout a career in medicine. The medicine program at UNSW is divided into three phases. Students in Phase 1 (Years 1 and 2) of the program engage in scenario-based learning, predominantly on the university campus, with a component of clinical and communication skills from the outset. Phase 2 (Years 3 and 4) is balanced between clinical attachments and campus-based learning of associated biomedical sciences, while students in Phase 3 (Years 5 and 6) are engaged in hospital- or community-based clinical attachments, integrated with a novel biomedical sciences curriculum supported by e-learning modules.

According to a needs analysis survey, the topics in Pathology that were most conceptually difficult for senior medical students were glomerulonephritis, lymphoma and leukaemia [3]. e-learning modules which bring together concepts in a clinical context are potentially effective ways of overcoming such perceived difficulty. We have previously shown that provision of e-learning modules on glomerulonephritis and lymphoma had a positive impact on learning in randomised controlled trials, compared with traditional teaching methods [3, 9].

However, using PubMed and Google Scholar, we found no published reports that evaluated e-learning modules to assist medical students’ understanding of leukaemia. Therefore, this study aimed to evaluate the learning impact on senior medical students of a targeted e-learning module on leukaemia, compared with existing e-learning resources.

Of the 23 students in the Study group, 21 (91.3 %) completed all components of the trial, i.e., pre-test, post-test and questionnaire, while 21 of the 22 students in the Control group completed all components (95.5 %). There was no significant difference in the mean documented previous academic performance (WAM) for participants in the Study and Control groups, which were essentially identical (Study: 71 ± 1; Control: 71 ± 1) (t (43) = −0.163, P = 0.871).

There was no significant difference in mean percentage scores between groups for the pre-test on leukaemia (Study: 51 ± 3; Control: 54 ± 2) (t (43) = 1.055, P = 0.297). However, the Study group achieved significantly higher mean percentage scores in the post-test on leukaemia (Study: 80 ± 3; Control: 66 ± 3) (t (42) = −3.591, P < 0.001). Importantly, this striking difference was noted even though the Control group’s performance also improved significantly compared with the pre-test (Figure 4).

The Study group reported spending more time on average studying the topic of leukaemia than the Control group during the 2-week trial period, but this difference (equating to three minutes per day) was not statistically significant (Study: 2.3 ± 0.4 hours; Control: 1.6 ± 0.4 hours) (t (39) = −1.347, P = 0.186).

In a stepwise multiple regression analysis, group allocation was the only significant predictor of performance in the post-test (R = 0.516, R ² = 0.266, P = 0.001), i.e. 26.6 % of the variance in post-test scores was accounted for by group membership. In this model, neither known academic ability nor time spent studying was predictive, although the latter parameter in isolation correlated significantly with post-test scores (R = 0.363, P = 0.02). The correlation between WAM and post-test scores was not statistically significant (R = 0.294, P = 0.052).

Via the online evaluation questionnaire, participants in both the Study and Control groups rated the perceived difficulty of the topic of leukaemia before and after the trial period on a 10-point Likert scale (1 = least difficult, 10 = most difficult). Pre-trial, there was no significant difference between groups in ratings of the perceived difficulty of leukaemia (Study median rating: 9; Control median rating: 8). Post-trial, there was a significant decrease in the median perceived difficulty of leukaemia in the Study group (median rating pre-trial 9; post-trial 6, P < 0.001, Dunn’s multiple comparisons test), but not in the Control group (median rating pre-trial 8; post-trial 7) (Figure 5).

The Study group’s online evaluations of the e-learning module were overwhelmingly positive, and their rating of each aspect (including enjoyment, guide to study and overall value for learning) was significantly higher compared with evaluations of the alternative e-learning resources by the Control group (all P < 0.001, Mann-Whitney U tests) (Figure 6). Of particular note, the Study group found the module to be the most useful resource for learning about leukaemia, compared with lectures, tutorials, private study and clinical experience, whereas the median ranking of the e-learning resources available to the Control group was third out of five.

Students in the Study group found the following features of the module most helpful: interactivity; feedback; case studies; multimedia (videos, animations, audio); and histopathological images. In addition, participants in the Study group commented that revision of basic science concepts in the concepts and causes section of the module was extremely useful.

Study group participants also provided feedback to improve the module. Several suggested that a transcript of the audio should be available on each screen. This feature has subsequently been implemented. Participants generally preferred more text on each screen, rather than audio. Some participants, who had limited internet speeds due to their rural location, found that the module took a long time to download.

In contrast, online evaluation data provided by the Control group was less positive. The participants in the Control group indicated that the content of the e-learning resources they accessed was not sufficiently tailored to their context. Participants frequently commented that the information was too theory-based, yet they perceived that the pathophysiology underlying the results of diagnostic procedures was not well explained.

Control group participants suggested that the online learning resources needed to be more interactive, and more specific to their syllabus. Students commented that while there were “a lot of resources out there” many students have issues deciding “what level of knowledge is expected”.

As expected, the mean scores of both groups improved from pre-test to post-test, indicating that the existing online resources provided to the Control group did confer some benefit. However, the learning benefits of the tailored e-learning module utilised by the Study group markedly exceeded those of the resources provided to the Control group. Although the cohort size for this trial was relatively small, the statistical power of the study was sufficient to yield valid results.

Both the pre-test and post-test were reviewed by two senior members of the academic staff in Pathology at UNSW, as well as a senior clinical haematologist. All of them rated both tests as being of equivalent difficulty, and ensured that the material covered by the post-test was addressed by both the e-learning module and the alternative e-learning resources. Furthermore, the post-test was developed after the e-learning module had been completed, thereby avoiding the potential for ‘teaching to the test’. This reduces the likelihood of bias towards the Study group in the design of the post-test.

The Study group’s success might be at least partially accounted for by the targeted nature and overarching conceptual framework provided by our e-learning module. Certainly, the lack of such focus in the existing online resources was remarked upon by participants as a major disadvantage. Furthermore, in accordance with reports of previously successful e-learning interventions [7, 9, 11, 12], the high level of interactivity and dialogicfeedback provided by the e-learning module, particularly compared with the online text book,is likely to have resulted in improved efficacy. In any form of e-learning, those features are important for both engagement and educational impact [11, 15, 20, 21].

Surprisingly, stepwise regression analysis revealed that the variance in post-test scores was significantly influenced only by group allocation. It was anticipated that time spent studying the topic and students’ previous academic performance (WAM) would also have had significant effects. Indeed, of those variables, only time studying leukaemia correlated significantly with post-test scores, whereas the correlation between post-test scores and WAM was not statistically significant. From these data, we infer that the e-learning module provided significant learning gains, without requiring a significant extra investment of students’ time. It seems reasonable to conclude that the design of the module provided students with a time-efficient learning experience. The weak relationship between WAM and post-test scores might be accounted for by the benefits of e-learning for weaker performing students, more so than higher performers. This phenomenon is consistent with previous studies of e-learning interventions [22, 23].

The questionnaire data revealed that the e-learning module significantly reduced the perceived difficulty of leukaemia for participants in the Study group at the end of the trial period, compared with the Control group. This is an important finding, which suggests that exposure to the integrated conceptual framework of the e-learning module had an impact not only on learning, but also on confidence and attitudes. This is reinforced by the significantly increased interest in leukaemia espoused by participants in the Study group compared with those in the Control group.

The majority of participants in the Study group rated the e-learning module as being the most helpful mode for their study of leukaemia (compared with lectures, tutorials, texts and individual study), while students in the Control group rated the existing e-learning resources as being significantly less helpful. Those in the Study group remarked that interactivity and feedback within the module added to their enjoyment of the topic. Participants also commented that their ability to understand and interpret the case studies was important to their success in the post-test. These data indicate that the e-learning module had both qualitative and quantitative beneficial effects on learning. These observations are consistent with previous studies of e-learning interventions [9, 13, 18].

The current study expands on earlier reports of studies regarding e-learning modules on lymphoma [9] and glomerulonephritis [2]. The validity of this trial was also bolstered by the inclusion of a pre-test, which established that the mean baseline knowledge of leukaemia was equivalent for both the Control and Study groups.

Many studies have compared e-learning to face to face teaching in a wide variety of contexts. However, there have been few randomised controlled trials that have compared the learning impact of different forms of e-learning [10] and this is the first such report regarding the topic of leukaemia. Furthermore, we believe that this study has important implications for the design of future e-learning modules. In particular, the findings suggest that targeted e-learning modules which embed case-based examples within an overarching conceptual framework are much more effective than generic online cases and online texts. Indeed, our results highlight the notion that in the sphere of e-learning, not all formats are equal in value [24].

The results of this randomised, controlled trial indicate that a purpose-built e-learning module on leukaemia significantly improved students’ understanding of the topic, and measurably decreased students’ perceptions of its difficulty. The e-learning module on leukaemia had a significant impact on learning in this cohort, compared with existing online resources. We believe that the curriculum-based conceptual framework embedded in the structure of the module, as well as interactivity and dialogic feedback, contributed to its impact. The results of this study have implications for the design of e-learning in medicine and other disciplines.