The development of RLOs on meta-analysis was in response to lecturers on healthcare courses recognising from assignments and assessments that students did not fully understand the statistical technique of meta-analysis or the various important concepts that underpin meta-analysis. Students reported they felt their understanding had been improved by the RLOs (Figure 4) and there were certain features of the RLOs described in Tables 1 and 2 that appeared to contribute to this enhancement by adding value to lecture delivery. These findings are supported by other studies describing the educational benefits of RLOs in blended learning settings in healthcare curricula [7, 8] and improved learning performance in clinical laboratory sciences [13]. Generally, RLOs have been identified as having an important role to play in new medical curricula abroad [14, 15].
Visual learning approaches have been shown to enhance learning by providing multiple representations of a topic and by supporting learner preferences [16, 17]. The visual and interactive elements along with the use of analogies to describe the meta-analysis process seemed to contribute to the students' perceived improvement in understanding as illustrated by the quotes (T2) in Table 2. A recent report also alludes to the importance of appropriate visual representation describing an approach using simplified graphics of Forest plots to increase nurses' awareness of effect measures for different levels of data (adjusted means for continuous data and odds ratios for discrete data). These authors however provide only anecdotal evidence of the usefulness of these simplified plots [18].
A criticism directed at RLOs is their instructivist approach [19]. We would argue that pedagogical strategies to encourage active learning can be built into the RLO design and how the RLOs are integrated into a course determines their value in promoting reflection and deeper learning. Both the value of self assessment and reflection on the material were highly scored by the students (Table 1 Educational Value). Further research is needed to test the assertion that the use of RLOs can lead to deeper learning and the context of use of the RLOs will be an important variable to consider.
Boyle and Cook [20] suggest that RLOs should fit a cohesion and decoupling model referring to the idea that they are self contained learning units and the content is aligned with the learning goal and assessment and free from external links. The feedback suggests that some students would have struggled with the content of the RLO using it alone as a replacement for the didactic lecture and they would need the support of a teacher (Table 1 Learning Support; Table 2 T1, T3). Some students however wanted more detail and felt that the RLO should be offered prior to the lecture. Whilst the pedagogical design of the RLOs provides the flexibility for them to be used as stand alone units, most students still prefer RLOs to support blended delivery [7, 8].
The similar positive responses of the PGD and MPH students suggest that the flexibility of the utility of RLOs makes them suitable for multiprofessional learning perhaps providing support for having shared open repositories for healthcare education [15]. The framework for the design process [11] certainly caters for multiprofessional debate amongst healthcare educators around content and associated learning activities [21].
Previous investigations of the effectiveness of e-learning technologies for health professionals identified a number of barriers to its success among them cost, poorly designed packages, lack of skills, need for a component of face-to-face teaching, time intensive nature of e-learning and computer anxiety [6, 22]. This study has shown how many of these factors have been addressed. One of the reasons for making our resources freely and openly accessible were the difficulties students had accessing materials requiring usernames and passwords and remembering to access them. Two students alluded to this in their comments (Table 2 T4).
With the plethora of open educational resources now available to students, there are a new set of issues around 'discovery' and 'filtering' to find the good quality pedagogically defined learning materials.
The value of these meta-analysis RLOs may be partly because they are by their very nature different to traditional e-learning tools. Our data suggest that individual RLOs do not require more than around 15 minutes to complete thus they do not require a time-intensive input making them more flexible for students to use at work or home, attributes previously shown to be valued by students [5, 22]. Ratings of flexibility and control were rated more highly than the value of the media attributes (Table 1 Flexibility and Control vs Media Attributes), this has been reported before in e-learning studies [7, 8, 23]. It is asserted that a sense of ownership and control over learning is important for healthcare students coping with busy curricula and work placements [6, 8]. Learners taking more active control of visual learning approaches construct a deeper understanding of the subject [24].
The visual, audio and interactive nature of these RLOs means that they have an appeal for visual, auditory and kinesthetic learners an important issue bearing in mind data which suggests that learning style is important in web-based e-learning [9, 25].
However, not all students liked the RLO or some of the media used, though these students were in the minority (Table 1).
In terms of the usability and media attributes of the RLOs the majority of students
either agreed or strongly agreed with statements relating to the ease of use of the RLOs and the value of the different components which make up the RLO reinforcing their importance in enhancing learning and understanding (Table 1 Usability and Media Attributes).
The 'not invented here' view has been suggested to be a barrier to the reuse of resources [26] because lecturers want to make them 'their own' and contextualise the resource for their particular students. This does not seem to be a problem for students however since we found no differences in the RLO ratings for Nottingham students compared to non-Nottingham students. A non-Nottingham student valued "The step-by-step description of each part of the Forest plot as well as basic information imparted by each tab given".
Limitations
This was a small scale evaluation study based on a convenience sample that relied on students' self-reporting their levels of understanding. Objective measures of improvement in knowledge of meta-analysis were not collected because students agreed to take part in the study on the basis that their responses were anonymous and would not be followed up in summative assessments.
