Teaching tools in Evidence Based Practice: evaluation of reusable learning objects (RLOs) for learning about Meta-analysis

"All health care professions need to have an understanding of EBP, understand the principles of evidence based practice (EBP), recognise EBP in action, implement evidence-based policies, and have a critical attitude to their own practice and to evidence. Without these skills, professionals and organisations will find it difficult to provide best practice" [1]

Evidence based medicine is described as "the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients" [2]. However research based information is not used in isolation but together with patient preference and an individual's knowledge and expertise. Similarly, evidence based nursing can be defined as the application of valid, relevant, research-based information in nurse decision-making [3]. Standards of conduct, performance and ethics for nurses and midwives state that nurses and midwives 'must deliver care based on the best available evidence or best practice' [4]. To be able to do this a knowledge of evidence based practice (EBP) is required and the skills to perform EBP.

Whilst these definitions recognise the importance of individual clinical expertise and patient choice as determinants of clinical decision making, the randomised trial is regarded as the gold standard for judging whether a treatment is beneficial. The volume of data that needs to be considered by practitioners is constantly expanding and keeping up to date with literature evidence can be a challenge. Therefore, reviews have become essential tools to keep up to date with new evidence since they collate and evaluate primary research on a focused topic. Systematic reviews allow for a more objective appraisal of the evidence compared to traditional literature reviews and a technique called meta-analysis is commonly used in scientific papers and systematic reviews looking at effectiveness of treatments. Meta-analysis is a statistical technique for pooling the results from similar studies in order to increase the statistical power and therefore is an important concept to get to grips with in EBP.

In the School of Nursing, Midwifery & Physiotherapy at the University of Nottingham, EBP is an integral part of the curriculum for pre registration through to post registration and postgraduate education courses. EBP is also a component of the undergraduate Medical courses and the Master in Public Health programme. Lecturers on these courses recognised from assignments and assessments that students did not fully understand the statistical technique of meta-analysis or the various important concepts that underpin meta-analysis.

Whilst lectures result in delivery of information they do not necessarily engender learning and understanding which may be better supported by blended or more applied teaching methodologies. Our previous studies of students in health sciences (particularly in areas of the curriculum that they find difficult) have shown that by supplementing lectures with e-learning resources that are interactive, visual, and small in size and highly aligned with their perceived learning needs [5, 6] improve their understanding and attainment [7, 8].

In view of this, we wanted to enhance students learning of meta-analysis by designing and developing flexible e-learning tools in the form of two reusable learning objects (RLOs) to accompany the usual didactic lectures. An evaluation study with healthcare students would determine the impact of the RLOs on their perceived understanding of meta-analysis concepts and views on usability and design of the e-learning materials.

Numerous definitions for an RLO exist [9, 10] however our definition of an RLO is: 'an interactive, multimedia web-based resource based on a single learning objective which can be used in multiple contexts'. Basically, they are bite sized chunks of e-learning, focusing on a specific topic. They are highly visual with an auditory component and high quality graphics and take the average student about 15 minutes to complete.

We will also report briefly on re-use of the RLOs by other students outside of the study group for whom the RLOs were originally designed for.

The development of RLOs on meta-analysis was in response to lecturers on healthcare courses recognising from assignments and assessments that students did not fully understand the statistical technique of meta-analysis or the various important concepts that underpin meta-analysis. Students reported they felt their understanding had been improved by the RLOs (Figure 4) and there were certain features of the RLOs described in Tables 1 and 2 that appeared to contribute to this enhancement by adding value to lecture delivery. These findings are supported by other studies describing the educational benefits of RLOs in blended learning settings in healthcare curricula [7, 8] and improved learning performance in clinical laboratory sciences [13]. Generally, RLOs have been identified as having an important role to play in new medical curricula abroad [14, 15].

Visual learning approaches have been shown to enhance learning by providing multiple representations of a topic and by supporting learner preferences [16, 17]. The visual and interactive elements along with the use of analogies to describe the meta-analysis process seemed to contribute to the students' perceived improvement in understanding as illustrated by the quotes (T2) in Table 2. A recent report also alludes to the importance of appropriate visual representation describing an approach using simplified graphics of Forest plots to increase nurses' awareness of effect measures for different levels of data (adjusted means for continuous data and odds ratios for discrete data). These authors however provide only anecdotal evidence of the usefulness of these simplified plots [18].

A criticism directed at RLOs is their instructivist approach [19]. We would argue that pedagogical strategies to encourage active learning can be built into the RLO design and how the RLOs are integrated into a course determines their value in promoting reflection and deeper learning. Both the value of self assessment and reflection on the material were highly scored by the students (Table 1 Educational Value). Further research is needed to test the assertion that the use of RLOs can lead to deeper learning and the context of use of the RLOs will be an important variable to consider.

Boyle and Cook [20] suggest that RLOs should fit a cohesion and decoupling model referring to the idea that they are self contained learning units and the content is aligned with the learning goal and assessment and free from external links. The feedback suggests that some students would have struggled with the content of the RLO using it alone as a replacement for the didactic lecture and they would need the support of a teacher (Table 1 Learning Support; Table 2 T1, T3). Some students however wanted more detail and felt that the RLO should be offered prior to the lecture. Whilst the pedagogical design of the RLOs provides the flexibility for them to be used as stand alone units, most students still prefer RLOs to support blended delivery [7, 8].

The similar positive responses of the PGD and MPH students suggest that the flexibility of the utility of RLOs makes them suitable for multiprofessional learning perhaps providing support for having shared open repositories for healthcare education [15]. The framework for the design process [11] certainly caters for multiprofessional debate amongst healthcare educators around content and associated learning activities [21].

Previous investigations of the effectiveness of e-learning technologies for health professionals identified a number of barriers to its success among them cost, poorly designed packages, lack of skills, need for a component of face-to-face teaching, time intensive nature of e-learning and computer anxiety [6, 22]. This study has shown how many of these factors have been addressed. One of the reasons for making our resources freely and openly accessible were the difficulties students had accessing materials requiring usernames and passwords and remembering to access them. Two students alluded to this in their comments (Table 2 T4).

With the plethora of open educational resources now available to students, there are a new set of issues around 'discovery' and 'filtering' to find the good quality pedagogically defined learning materials.

The value of these meta-analysis RLOs may be partly because they are by their very nature different to traditional e-learning tools. Our data suggest that individual RLOs do not require more than around 15 minutes to complete thus they do not require a time-intensive input making them more flexible for students to use at work or home, attributes previously shown to be valued by students [5, 22]. Ratings of flexibility and control were rated more highly than the value of the media attributes (Table 1 Flexibility and Control vs Media Attributes), this has been reported before in e-learning studies [7, 8, 23]. It is asserted that a sense of ownership and control over learning is important for healthcare students coping with busy curricula and work placements [6, 8]. Learners taking more active control of visual learning approaches construct a deeper understanding of the subject [24].

The visual, audio and interactive nature of these RLOs means that they have an appeal for visual, auditory and kinesthetic learners an important issue bearing in mind data which suggests that learning style is important in web-based e-learning [9, 25].

However, not all students liked the RLO or some of the media used, though these students were in the minority (Table 1).

In terms of the usability and media attributes of the RLOs the majority of students

either agreed or strongly agreed with statements relating to the ease of use of the RLOs and the value of the different components which make up the RLO reinforcing their importance in enhancing learning and understanding (Table 1 Usability and Media Attributes).

The 'not invented here' view has been suggested to be a barrier to the reuse of resources [26] because lecturers want to make them 'their own' and contextualise the resource for their particular students. This does not seem to be a problem for students however since we found no differences in the RLO ratings for Nottingham students compared to non-Nottingham students. A non-Nottingham student valued "The step-by-step description of each part of the Forest plot as well as basic information imparted by each tab given".

Understanding meta-analysis will give confidence in reading scientific papers and systematic reviews which in turn will encourage evidence-based practice ultimately leading to improved patient care. The RLOs were successful in supporting the students understanding of meta-analysis, resulting in a perceived increase in understanding of the various components of meta-analysis. The RLOs evaluated extremely positively by the students, and the students reported that they continued to access the RLOs after the evaluation. RLOs are used flexibly; not only to help students with difficult topics but also to illustrate concepts or skills. Students use them to plug gaps in their knowledge and to prepare for lectures/tutorials or as revision aids. The feedback showed that the RLOs supported learning but students still valued the lectures therefore we recommend that RLOs are used as a supplement but not to replace other delivery approaches. This is the first study to report on the evaluation of meta-analysis resources.