Evaluation of a learner-designed course for teaching health research skills in Ghana

Participants

Study participants were all 15 learners who enrolled on the DPDM course in KATH in 2003 (6) and 2004 (9). None of the learners had had previous experience of designing and implementing their own research and all volunteered to take part in the study. Learners were assured of anonymity and that participation would not affect their academic progress. Learners' specialities included paediatrics, adult medicine, ENT surgery, obstetrics and gynaecology, physiotherapy, pharmacy and health management.

Selection of evaluation tools

There were no specific evaluation tools available for evaluating such courses in developing countries so we based our evaluation on a published framework incorporating criteria derived from different perspectives [13]. We included indicators of process (i.e. how the course was delivered), content (i.e. what was delivered) and outcomes (i.e. completed assignments and projects; improved competence and confidence in research skills) [14]. The criteria for selecting the evaluation tools were that they had been published in peer-reviewed journals, were relevant to the course outcomes and met requirements for evaluating innovative educational interventions based on social learning [11, 12], and they could be applied within the available time and resource constraints. Two different methods were used to assess each of the learning outcomes. Student assessments took place in months 3, 8 and 12; the evaluation was carried out at the end of the course in month 12.

Evaluation of competence and confidence

Competence was assessed from learners' performance in two assignments, the research proposal (month 4) and project report (month 12). The assignments were marked independently by two markers from either institution or by three markers if marks were discrepant by ≥10%. Four of the authors (one from Liverpool, three from Kumasi) and three other academics who were not involved in this research (one from Liverpool, two from Kumasi), were involved in marking assignments. As part of the examination process, and to ensure that the research did not bias student marks, all marks were reviewed by two independent examiners who had not been involved in the course, one from Liverpool (internal examiner) and one from an external UK university (external examiner). The course curriculum, assessment and final marks were reviewed and agreed by the examiners to ensure fairness and transparency and that the course met UK academic standards [15].

Corroboration of data obtained from the SOC tool was sought from learners' reflective commentaries concerning their experiences of doing research. The commentaries were analysed by one of the authors (IB) using elements of the grounded theory approach [19]. Codes for analysis of the reflective commentaries in relation to confidence in research skills were derived from data in the first few commentaries. The codes were applied to data that specifically mentioned 'confidence' as well as students' personal, institutional and contextual reasons for lack or acquisition of confidence (e.g. non-clinical status, recognition of application of research to clinical practice). All reflective commentaries were then analysed iteratively until no new themes emerged. Major themes were identified and inferences made about linkages between themes. Reflective commentaries were re-examined for data that may discredit the theories and this information was used to refine and consolidate the theories.

Identification of factors that promoted learning

The nominal group technique [20] was used to identify elements within the course that most effectively promoted learning. This technique enabled individual learners to contribute to the process equitably and the whole group to identify and rank the critical elements. Using 'post its' each learner wrote three elements about the DPDM course that had facilitated their learning. These elements were pooled, grouped into themes and ranked through learners' group discussions. To corroborate data from the nominal group technique, learners' reflective commentaries were analysed as above but with codes that incorporated skills the students identified they had learnt (e.g. budgeting, time-keeping), the process by which they had learnt these skills (e.g. by doing a pilot study, by reviewing someone else's proposal) and how they would use these skills in their professional work (e.g. improve record keeping, communicate better with laboratory staff).

Evaluation of competence and confidence

The course examiners determined that of the 15 participants who completed both the proposal and report course assignments, 15 had passed the project proposal assignment, 14 passed the report assignment (mean mark 61.1 out of 100; SD 8.1) and two gained distinctions (i.e. >70%). The learners' mean (range, SD) score for the 11 statements on the RSES was 8.43 (7.42–9.70, 0.63) out of a possible maximum of 10 (table 1). Overall 82.5% of responses were graded as 8, 9 or 10 (i.e. very able). There were no responses graded <5 (i.e. not able).

Elements of course that promoted learning

Principal findings

Using the model we have developed to evaluate a novel educational initiative in a resource-poor setting, we have demonstrated that a learner-designed course to teach research skills to health professionals in Ghana can be effective and meet international standards for quality education. The course met the needs of individual learners, who considered themselves to have become confident and competent in research, and of the institution, which increased its research capacity. A secondary but important outcome was the enhancement of learners' professional skills and better research awareness and advocacy within KATH. Critical to the success of the course was ownership by the learners, support from peers and learning by doing research and reflecting on experiences.

Strengths and limitations of the study

In developing countries demand for high quality education is increasing. When resources are scarce it is essential to demonstrate that educational interventions are effective. Our study provides a scheme for evaluating an innovative educational course, based on the theory of social learning, which was feasible to implement and interpret in a developing country context. Our study incorporates several features that identify it as research rather than a straightforward evaluation [7, 23]. The study used valid, reproducible and appropriate methods leading to neutral conclusions rather than decisions; it was not influenced by the funder and it advanced knowledge, contributed to theory and explored opportunities for transferability. A major strength of this study was the use of diverse methods to examine process, content and outcomes [14] and two different methods [13] to assess the same outputs.

Because of the nature of social learning our results need to be interpreted cautiously because measured benefits may have been influenced by other educational experiences and variations in the students' work or personal environment [7]. In the absence of subject-relevant benchmark statements, assessment of learners' research competence was judged against UK higher education quality standards which included alignment of learning outcomes with curriculum content and assessment, use of assessments to support learning, and opportunities to reflect on learning [3]. Self-reported ratings tend to overestimate student confidence and competence especially when the evaluation is conducted by the tutors as in this study. As skills are learned, students' self-rated ability more closely reflects actual ability levels [24] This 'research-naivety' among the students in our study may partly explain why they had slightly higher post-course scores in the RSES (7.4–9.7) than students in a previous study (55.3 – 82.4, scored out of 100 rather than out of 10: see table II p194 in reference 17). Despite these limitations, the RSES and SOC have been found to be useful for demonstrating the process of developing research competence and confidence and for ranking improvements in research-related attitudes, intentions and actions in previous studies [16–18]. The evaluation methods we used were simple to adapt and use in the Ghanaian setting but their usefulness needs to be assessed in a variety of different developing country contexts. Such tools can benefit from complementary qualitative data from learners' reflections and the nominal group technique to obtain richer and deeper understanding of how learners acquired research skills.

Transferability of evaluation model to other settings

Each educational setting is unique. Our learners and their learning environment had some characteristics that may not be reproduced elsewhere and which may impact on potential sustainability and transferability of our methods and findings. Our learners were highly motivated to learn research skills in order to pass professional examinations and to obtain an internationally-recognised Diploma qualification. The learners had designed the course themselves and therefore understood what they needed to do to succeed. The experience of future cohorts may be less intense, as they will not be 'pioneers' and this may influence their motivation and commitment to the success of the course. The learners were a unique mix of middle-grade, health professionals who were prepared to share their wealth of pooled expertise to compensate for the lack of local tutors. Learning outcomes can be affected by the social mix, culture and lifestyle of students [25] and our approach may not work so well with more junior or less motivated or cohesive learners. The KATH managers had a clear vision and strategy for developing their institution's research capacity [26] and provided resources rapidly and flexibly to ensure the success of the course. Although by the end of the course only a few students considered themselves to be capable of managing a research project independently, our findings show that through the course the students have developed an appreciation of the process of research and the role that research can play in improving evidence-based clinical care. This will contribute to improving the research culture within the institution [26].

Study outcomes in the context of theories about social learning

Our research highlights how learners set up their own 'community' of research practitioners at KATH and utilised the learning opportunities provided by the social aspects of their work environment [27, 28] to underpin changes at institutional level [5]. Such 'communities of practice' and peer support groups facilitate learning [29], promote reflective interactions [30, 31] and generate 'creative turmoil' and innovation [32], and may explain why the DPDM course played such a key role in strengthening KATH's institutional, as well as individuals', research capacity.

To promote knowledge around coherent topics, educational research should be located within a theoretical framework [33]. The course that we evaluated was developed using theories of social learning and workplace-learning synergies. The course evaluation, which demonstrated that students had achieved the learning outcomes, and our research into effective strategies to promote student learning in a resource-poor environment, demonstrated the importance students placed on learning by sharing their own varied knowledge and experiences. Our research therefore confirmed previous findings that learning outcomes are influenced by the social and professional diversity of students [25] as some learners felt disadvantaged within the cohort because of perceived differences in professional status or lack of previous research experience [34].

The findings from our study support theories from developed countries concerning the process of social learning at work [4, 8, 34]. Key characteristics of work-based learning are that it is managed by the learners, it is team based, innovating and empowering, and it can be enhanced by group activities that promote reflective practice and higher order thinking [35]. Our research has demonstrated that these concepts can be successfully applied to a work-based course for health professionals in Ghana. This study therefore contributes evidence that the process of social learning at work, and theories about the role of social interactions and institutional culture on improving effectiveness of learning which have emanated from developed countries, also apply in developing countries.

Implications for educators and future research

International development policies are urging UK universities to expand access to high quality education particularly in Africa. There is an urgent need for practical tools, such as rigorous assessment processes [36], RSES, SOC models and qualitative analysis of student reflections, to guide delivery and evaluation of accredited courses in resource-poor countries. These tools should be sensitive to the need to understand and build on social and institutional interactions to promote effective learning. The combination of an innovative learner-designed course underpinned by peer-supported learning, originating in a developing country, and an educational quality framework, generated by developed countries, offered unique opportunities for bilateral exchange of best practice between South and North.