A Comparison of Online versus On-site Training in Health Research Methodology: A Randomized Study

There is great need and demand for building health research capacity globally, to ensure that opportunities and competence to undertake scientifically and ethically sound research exist in all regions [1, 2]. Current strategies to train investigators in clinical research skills rely on traditional class-room teaching that is expensive to provide and difficult to scale up in resource-limited settings.

Distance learning, including on-line teaching, may provide an attractive, cost-effective, scalable and efficient alternative to on-site classroom training. On-line education formats use diverse media such as text, images, audio, video and interactive formats [3]. On-line learning has been widely utilized for education in the US and other developed country settings [4]. In these settings, much research has been undertaken to assess the effectiveness of online training in diverse fields. These studies have shown that well-designed online medical education courses result in knowledge gains similar to, and at times superior to, traditional classroom teaching [5, 6]. In addition, several courses have also shown evidence of significant self-reported practice change [7]. Further, more recently, various online training formats have been compared to identify those that offer the best opportunities for imparting e-learning in terms of participant satisfaction and gains in knowledge [8]. In addition, several guidelines and reviews on the subject of e-learning and design of content for such training have been published [9–11].

Distance education has also supported training of students in resource-limited settings. India has the largest distance learning university in the world, offering nearly 350 courses with approximately 2.8 million students currently enrolled [12, 13]. However, evaluations of the impact and acceptance of on-line learning platforms in such settings are limited. In addition, there is limited information on the ability of distance learning to improve knowledge and skills relevant for health research.

To better understand the potential value of on-line learning platforms to expand health research capacity, we undertook a randomized study comparing on-line with on-site (i.e. face-to-face) delivery of information in two distinct domains relevant for international health research: Biostatistics and Research Ethics. Further, to assess the feasibility and potential of utilizing on-line platforms for expanding health research capacity in resource-limited settings, we conducted this study in India. Our hypothesis was that both on-site and on-line course formats would lead to similar gains in knowledge for students, for both content domains. We further hypothesized that trainees would report a similar level of satisfaction for on-line and on-site course platforms.

Our study comparing on-line to on-site teaching formats in a resource-limited setting demonstrated that both formats significantly increased knowledge from baseline and this increase in knowledge was observed for both content areas (Biostatistics and Research Ethics). In addition, the increases in knowledge were sustained for 3 months after completion of the courses. There was a high level of student satisfaction for all courses, although the on-site format was associated with a somewhat higher level of satisfaction related to instructor accessibility and quality of faculty feedback.

A recent meta-analysis of 126 on-line learning interventions demonstrated that all but two of these were associated with a gain in knowledge [15]. Most of these 126 studies were from developed countries and three evaluated the impact of courses related to building health research capacity (including one each on research methodology, statistics and institutional review board policies). Our study from India demonstrated significant improvements in knowledge after on-line courses in a resource-poor setting for two quite disparate research training domains, a finding consistent with those in resource-rich setting.

In another meta-analysis covering the impact of on-line training in diverse fields from school level to professional level [4], on-line training was associated with a higher gain in knowledge. However, the difference was very limited when analysis was restricted to studies that compared face-to-face and pure on-line training formats. In comparison, a much larger gain in knowledge was seen when blended (a mixture of on-line and face-to-face) instruction programs and face-to face programs were compared [4], though even these effects were considered to be related to differences in time spent in learning, curriculums and pedagogy than to differences in training formats. Thus, our findings at 3-months after on-site training and on-line training that included interactive sessions in a developing country setting mirror the findings from resource-rich settings. In this context, it is important to note we used scheduled group e-learning sessions, in which the participants could log-in and discuss the course material with other participants and facilitators; this format has previously been shown to perform better than an on-demand format where such discussion sessions were not scheduled [8].

Although results were similar when assessed 3 months after the courses, for both knowledge domains, we found higher knowledge scores, particularly for Biostatistics, immediately after the on-site courses, than after the on-line course. Several factors may account for this observation. First, our study participants may not have had internet access of reasonable quality. Although we verified each participant's access to sufficient broadband internet connectivity at the beginning of the study, intermittent outages may have occurred. Second, participants in a resource-limited setting may not be as familiar with on-line courses as in US universities where 20-25% of all students take at least one on-line course [3]. Another explanation may be related to the work pattern of physicians in developing countries. Since such physicians, who constituted a majority of our study subjects, typically spend a larger share of their time on clinical duties than those in industrialized countries, they may find it difficult to spend time reading and reviewing on-line course material during a routine day. In contrast, during an on-site course, being away from their institutions and clinical work may have permitted the participants to focus better on the training activity. In the previously cited meta-analysis [4], the better outcome of on-line training was considered to have resulted from the on-line trainees spending more time on training activities than that spent during face-to-face courses, rather than to on-line delivery format per se. Alternatively, the difference may be related to better learning during on-site training, possibly due to the influence of factors such as face-to-face rather than screen-to-screen interaction with the instructor.

The difference in immediate knowledge gain between the on-line and on-site formats was more marked for Biostatistics, than for Research Ethics. A possible explanation of this difference is that the quantitative skills required for Biostatistics may be poor in medical professionals. Greater faculty-trainee interaction during on-site training may thus be helpful when training medical professionals in this domain. The results of the participant satisfaction survey, in which the on-site course scored higher in questions related to faculty-trainee interaction, would support this explanation. The greater satisfaction with on-site courses may also mean that further efforts are needed to enhance faculty-trainee interaction during on-line training.

Our study has several potential limitations. First, participants in our study may not be adequately representative of the population of prospective biomedical researchers that require training in research methodology. Our study participants may have been highly motivated, and therefore likely to have greater gains in knowledge scores with any type of training. However, due to the randomized design, this effect would have been similar for on-line and on-site courses, and would not affect our conclusions of comparative performance of the two training formats. Another potential limitation is that our study participants were from India and our results may not be generalizable to other populations with less access to and familiarity with the internet. Finally, although both the on-line and on-site Research Ethics courses used the same faculty, this was not possible for the two Biostatistics courses. However, in a previous meta-analysis, it was shown that a change in faculty member did not influence effectiveness of an on-line course, provided the course content and method of delivery were unchanged [4].

Another limitation of our study, designed as a comparative efficacy trial of two formats of training, was the inability to accurately assess the costs per person for each training format. Attendance at a course entails several different types of costs, including course fees, costs of travel and accommodation, and costs due to lost wages or work, etc; further, the course fees include costs of faculty time, course material, and facilities used. Ours being a research study, travel and hotel accommodation for all participants were arranged on a uniform scale, and these had no relationship with the costs that the participants would have incurred if they had arranged and paid for these. Because we used some of the pre-existing course materials and online facilities, costs of development of new courses for either on-site or online training courses could not be assessed accurately. Also, for the on-site ethics course, faculty members travelled from USA to India; this expenditure is unlikely to occur in real-life. Further, the 'cross-over' study design limited the number of participants in the online courses, precluding true assessment of per capita costs. Thus, in view of the unusual settings, we could not compare comparison of costs incurred per capita for the two training formats.

A training program can be evaluated at various levels. A popular approach to evaluation of training, the Kirkpatrick evaluation model, delineates four levels of learning outcomes [16, 17]. These include: (i) Reaction (assessment of participants' reaction to and satisfaction with the training program); (ii) Learning (degree of increase in intended knowledge, skills, attitudes and confidence); (iii) Behavior (application of learnt knowledge and skills once trainees are back on the job); and, (iv) Results (degree of targeted outcomes in terms of effect on business, efficiency, monetary terms, etc). Our study assessed only the first two domains. The remaining higher-level domains, though more important, cannot be easily assessed in an efficacy study of the kind we undertook. Assessment of those domains is possible only with long-term training programs, and further studies on the role of online training in attaining improvement in these domains are warranted.

On-line teaching provides several advantages over more traditional on-site course formats, particularly for building research capacity in resource-limited settings. On-line courses can be more flexible, convenient and accessible [9, 18], particularly for busy clinicians in communities where there is a shortage of health care providers and allow interactivity and adaptability to individual learner styles [5, 9, 11]. Also, such courses can be accessed by a much larger number of persons, from diverse geographic locations. This scalability is likely to offset the time and cost investment required for the on-line course development and for the interactive components. Although preparation of on-line training material may require a high initial financial investment, the recurring costs of such training are generally lower, because faculty and student travel costs are eliminated, and faculty time required for delivering didactic course material, though not for interactive sessions, is reduced in the long run.

In our randomized controlled study, On-line and on-site training formats led to marked and similar improvements of knowledge in Biostatistics and Research Ethics. Since on-line training has several logistical and cost advantages over on-site training, distance learning using on-line tools may be a particularly useful, efficient, cost-effective and scalable strategy for expanding health research capacity in resource-limited settings.