The purpose of this pilot study was to give first impressions of the effect of a blended learning concept in orthopedics and traumatology called NESTOR both on students’ satisfaction and on its contribution to acquisition of knowledge and clinical skills in a problem-based learning curriculum, which already provides an intensely practice-oriented teaching environment. To the best of the author’s knowledge, this is the first study examining the influence of blended learning not only on satisfaction, but also on knowledge and practical clinical skills of students in traumatology and orthopedics, two highly practically oriented medical subjects.
Evaluations of students’ opinion and acceptance can be seen as first step when establishing a new e-learning program [4, 14, 21]. Consistent with the literature, this study revealed a high approval of the participating users for the additionally offered e-learning contents. While a broad acceptance is crucial for successful e-learning implementation [1], it is also important to evaluate its influence on students’ gain of knowledge and skills [9].
Thus, as second step, not only the impact on users’ satisfaction, but also on their knowledge should be demonstrated [6, 21, 22]. For this pilot study we have chosen newly developed written tests to evaluate improvement in theoretical knowledge, which seem to be valid as indicated by the measured Cronbach’s Alpha values. While some studies showed benefits of e-learning on improvement of students’ knowledge [6, 23] others did not – despite of positive evaluation [2, 8, 22]. In this pilot study, we found a significant improvement from pre- to post-tests for both groups. NESTOR users scored better in the written post-tests than non-users and showed further improvement between post-test 1 and 2. A possible interpretation for this success in the group of NESTOR users may be students’ very positive attitude towards e-learning and a high satisfaction with structure and contents. However, these results should be interpreted carefully, especially when referring this effect exclusively to the use of NESTOR. In a recent review, Rowe et al. [12] showed that the existing data to evaluate an improvement of clinical competencies by blended learning can still be regarded as rudimentary. It seemed to be a problem of the study design in a clinical environment to determine the effect of blended learning exclusively. Rowe et al. identified 71 studies dealing with the role of blended learning in the clinical education of healthcare students, but only 7 articles were enrolled for the review due to methodological flaws of the remaining 64.
As potential third step it might be anticipated that even practical clinical skills may be improved through blended learning in this context [9], which has been shown in some studies [9, 16, 24]. The results of the OSCE revealed no differences between users and non-users in this pilot study with high scores in both groups. These findings are consistent with other studies which failed to detect significant benefits on examination performance or other practical clinical skills with e-learning implementation [21, 25, 26]. A possible explanation could be that the pre-existing, highly clinically oriented curriculum made it difficult for any additional e-learning exposure to further improve skills.
In this context, the question may arise what e-learning potentially can achieve [1]. It can be argued that clinical examination skills will always be preferentially based on personal experiences and training rather than on the use of e-learning – unlike acquiring skills in other areas such as radiological diagnosis [16]. In the presented pilot study, e-learning enhanced competencies for gaining theoretical medical knowledge. Further research will be necessary to determine, if it is possible to adjust the components of a blended learning approach in this context to achieve also an improvement of practical skills compared to mere face-to-face teaching. However, as knowledge about diseases is an important basis for developing treatment and examination skills, this and the overall high approval provide good arguments for the continued use of NESTOR in the preparation for the tested subjects. Following suggestions of the non-users, acceptance of the program might be further increased by improving announcements about it. Additionally, it could be made even more appealing with links to e-learning programs of other subjects (e.g. anatomy) of the faculty.
Concerning the willingness to use e-learning offerings voluntarily, an additional inference can be taken from this study en passent: As the use of NESTOR was significantly linked to the use of LMS Blackboard, the likelihood of voluntarily using an e-learning offering may be directly connected to the acceptance and use of the hosting LMS. This would require the need for the entire faculty to join in a combined effort to establish e-learning offerings broadly to increase students’ familiarity with such resources. Thereby, voluntary and perhaps even mandatory use of e-learning components could be increased.
There are some limitations to be noted in this study. A selection bias cannot be excluded due to the voluntary nature of participation and use of NESTOR, also with respect to the significant correlation between this aspect and a pre-existing use of the LMS Blackboard. This can be seen as main shortcoming, which was tolerated because data evaluation was incorporated into an ongoing mandatory course to establish this new blended learning concept. This pilot study design guaranteed a high practical orientation and a sufficient number of participants. However, for final conclusions on the chosen blended learning technique a randomized controlled trial should rather be performed in the future. Furthermore, preexisting experiences in students’ physical examination skills will have to be evaluated in the final implementation study by a pre-test. The possibility that the exposure to or interaction with any additional resources or experiences led to an increase in knowledge (i.e. due to a possibly more in-depth coverage of the topic), cannot be excluded completely with the study design. In a future study, there should also be a tracking function to detect the individual accessed e-learning tools or the time students spend with them. Such a study design would also avoid the present division of users and non-users merely according to their own declaration, as this step does not guarantee a correct assignment of data to the test and control group, respectively.
Finally, no direct correlation between test results and questionnaires was possible due to guaranteed complete anonymity. This could have potentially given more information about individual user’s attitudes towards the program and their concomitant test results.
