Learning needs analysis to guide teaching evidence-based medicine: knowledge and beliefs amongst trainees from various specialities
© Hadley et al; licensee BioMed Central Ltd. 2007
Received: 02 October 2006
Accepted: 10 May 2007
Published: 10 May 2007
We undertook a needs assessment exercise using questionnaire survey of junior doctors' knowledge and beliefs concerning evidence-based medicine (EBM) and critical literature appraisal, as this is a core competence in postgraduate medical education.
We surveyed 317 junior doctors in various specialities in the UK West Midlands Deanery. Using validated questionnaires we compared the needs of different trainee groups. Results overall were internally consistent (Cronbach's alpha 0.929).
Respondents' generally felt that they had poor training in EBM (Mean score 2.2, possible range 1 – 6) and that they needed more education (Mean score 5.3, possible range 1–6). Male trainees felt more confident at evaluating statistical tests than females (p = 0.002). Female trainees considered patient choice above the evidence more often than males (p = 0.038). Trainees from surgical speciality felt more confident at assessing research evidence (p = 0.009) whereas those from medical speciality felt more confident at evaluating statistical tests (p = 0.038) than other specialities. However, non-surgical specialities tended to believe that EBM had little impact on practice (p = 0.029). Respondents who had been qualified for 11 years or over felt overall more confident in their knowledge relating to EBM than those who had been qualified less than 10 years. In particular, they felt more confident at being able to assess study designs (p = < 0.001) and the general worth of research papers (p = < 0.001). Trainees with prior research experience were less likely to find original work confusing (p = 0.003) and felt more confident that they can assess research evidence (p = < 0.001) compared to those without previous research experience. Trainees without previous research experience felt that clinical judgement was more important than evidence (p = < 0.001).
There is a perceived deficit in postgraduate doctors' EBM knowledge and critical appraisal skills. Learning needs vary according to gender, place of basic medical qualification, time since graduation, prior research experience and speciality. EBM training curricular development should take into account the findings of our needs assessment study.
High quality health care implies clinical practice that is consistent with the current best evidence. Evidence-based Medicine (EBM) has thus become an impetus for incorporating critical appraisal of research evidence alongside routine clinical practice. Increasingly, acquisition of knowledge and skills for EBM is becoming a core competence to be acquired by all doctors. The UK Foundation Programme Committee  states that trainee doctors should be able to understand, critically appraise and evaluate the evidence base for medical practice and implement available evidence into clinical care.
To develop relevant teaching and learning opportunities for trainee doctors, it is essential that a needs assessment exercise is undertaken, particularly because EBM is not uniformly taught in undergraduate education. The findings of such an exercise can provide critical evidence for development and tailoring of EBM curricula improving the effectiveness of teaching. We undertook such an exercise using questionnaire survey of postgraduate doctors' knowledge, skills and beliefs in various specialties in the UK West Midlands Deanery. This allowed us the opportunity to compare and contrast different needs of groups according to gender, specialty, place of basic qualification, time since qualification and previous research experience.
During 2004–05 we surveyed all 317 junior doctors who attended one-day voluntary EBM workshops prior to commencement of teaching. Attendance to these workshops was not a mandatory part of the junior doctors' training. The study was planned prospectively using recommended methods for educational needs analyses  and questionnaire surveys. Ethical approval for the study was not required. Participants were made aware of the purpose of the survey, the anonymous nature of the dataset generated and the option to not respond if they so wished. This information served as the basis for an informed consent from each respondent.
We developed a questionnaire to measure clinician's basic knowledge and beliefs concerning the main principles of EBM including questions from previously published and validated questionnaires [3, 6, 7]. This included assessment of literature searching behaviour, self perceived knowledge of critical appraisal skills and beliefs. Closed questions with multiple choice answers were used along with those seeking responses on six-point Likert scales. For example, participants were asked how often they searched for evidence? Participants selected responses from a range of options, which included the statements 'more than once a week', 'every 1–2 weeks', 'every 3–4 weeks'. 'less than once a month' and 'never'. Participants were asked how confident did they think they were at assessing various aspects of a published paper or how confident they felt that they were able to assess adequacy of sample size, ascertain bias and evaluate statistical tests. Answers ranged from '1' not confident at all to '6' very confident. Items on beliefs about EBM included statements such as 'EBM is essential in my practice', 'clinical judgement is more important than EBM' and 'I feel that I need more training in EBM'. Participants scored their answers on a range from '1' indicating that they disagreed strongly with the statement to '6' suggesting that they agreed strongly with the statement.
Questionnaire administration and analysis
The questionnaires were self-administered by the candidates on arrival to the teaching session. All data obtained were entered into a Microsoft Excel spreadsheet and exported for analyses using SPSS versions 12.0.1 and 13.0. We used standard approaches to statistical analysis of questionnaire data including frequencies and descriptive summaries for the categorical data, and means, ranges, standard deviations and 95% confidence intervals for the Likert data .
Our analysis for internal consistency of the Likert scale questionnaire items used Cronbach's alpha, both overall and using the alpha for item deleted function, to look for "rogue" questions – that is questions answered in a quite different and inconsistent way. Cronbach's alpha assesses the internal consistency of the questionnaire results, that is, do the items to be measured look at much the same thing? There does not appear to be a consistent opinion on the value of Cronbach's alpha for scale reliability. An alpha of 0.5 or above is considered by Bowling  as an indication of good internal consistency, whereas an alpha of 0.7 or above is considered satisfactory by Howitt and Cramer . We used a figure of 0.7, and our data for questions on knowledge about EBM had an alpha of 0.929 – a very high figure.
For comparisons of Likert scale data by other variables (such as gender, place of basic qualification, years since obtaining qualification, and speciality) we used non- parametric tests, the Mann Whitney test for two categories and Kruskall Wallis test for three or more categories. These are non-parametric tests, which Jamieson  and Cohen, Manion and Morrison  consider the most appropriate for Likert scale data. These data are ordinal, does have a rank order, but the intervals between values cannot be considered equal. However, it is becoming common practice to use ANOVA for the analysis of such data . In addition, Pell  considers the use of ANOVA acceptable practice. In this study we used both categories of statistical tests on our Likert scale data, and in the text of the results below we only report the p-values for non-parametric tests. In most instances the results were similar.
The respondents' background of exposure to research and EBM showed that the majority of the respondents (246, 77.6%) had not attended a literature appraisal skills workshop. Only 24 (7.6%) of respondents stated that they had received formal training in research methods, six people (1.9%) had had epidemiology training and 24 (7.6%) had attended statistics courses. However, 192 (60.6%) stated that they had actually been personally involved in conducting some research activity.
Questions regarding clinicians' access to medical literature and evidence showed that all the clinicians except for two had access to a medical library and had access to medical literature on the Internet (n = 312). However, only 63 (19.9%) searched for medical literature more than once a week, 78 (24.6%) every 1–2 weeks, the rest searched less frequently or not at all. Furthermore, of all the respondents only 121 (38.2%) reported that they read every week regularly to keep up to date with their professional literature.
Doctors' perceived knowledge and beliefs
Doctors' gender, place of basic medical qualification and speciality
Effect of length of time since qualification
Effect of previous research experience
For a more in-depth breakdown of the results please refer to the accompanying additional file 1.
Our study identified several major issues that require addressing, not only in postgraduate medical education but also in the undergraduate curriculum. Many factors were found to be important determinants of improved EBM knowledge and appropriate attitudes. Learning needs varied according to gender, place of basic medical qualification, time since graduation, prior research experience and speciality.
Many key lessons have emerged for EBM teachers to consider carefully from our study. Many junior doctors support the principles of EBM, but they are undecided regarding whether patient choice and their own clinical judgment are more important and should override research evidence. Interestingly, those who had qualified overseas felt that clinical judgement was more important than the evidence compared to those who had qualified in the UK. Women and trainees without research experience gave greater emphasis to patient choice and clinical judgment, whereas men and those with previous research exposure value EBM and research more. Surgical trainees, those with previous research experience and those clinicians who had been qualified for over 11 years were more confident in assessing research studies. Trainees in medical specialities, those without research experience and those who had been qualified 10 years or less were less confident. We found that gender, speciality, time since qualification and previous research experience were important determinants of knowledge and beliefs. Elicitation of these variations is critical to planning curricula content and emphasis as prior knowledge and beliefs are key factors in determining learning achievement.
The validity and generalisability of our findings depends on the rigour of our questionnaire design, execution and analyses. We selected questions that had previously been validated and missing responses were few. One of the strengths of our study is that we surveyed a large sample of doctors from a variety of specialities and with varying lengths of time since qualification. However, our sampling strategy may be criticised, as it does not employ a random process. There are currently approximately 1000 postgraduate trainees in the West Midlands Deanery. We therefore, surveyed approximately 32% off all eligible trainees. Our respondents may have been more aware and self-motivated than trainees generally. If this is so, our findings provide a more conservative view of the training needs. We may also be criticised in methodological terms as the questionnaire relied on the doctors' self-perceived assessment of their own knowledge and beliefs [3, 15]. Research participants may feel pressurised into completing the questionnaire, or unwilling to divulge knowledge and skill deficiencies . Questionnaire research is never completely objective and as such potential biases can be introduced. We feel that our findings merit consideration by EBM educationalists, particularly those involved in postgraduate education.
Our survey substantiates and extends findings of previous studies by Olatunbosun et al  and Awonuga et al , who also found that clinicians lack methodological competence in critical appraisal skills and EBM. It concurs with previous findings that there is a deficit in postgraduate doctors' knowledge of EBM and critical appraisal skills, which requires addressing through education as the majority of trainees feel unconfident in their ability to assess research studies. Additionally, it provides information to tailor teaching curricula to the needs of specific subgroups of trainees. The overall aims and objectives of existing teaching courses need not change, but teaching methods need to be adapted to student's profiles that suggest a particular learning need. Thus, when planning teaching careful attention should be paid to the factors identified in this study.
We would like to thank Dr Arri Coomarasamy, Miss Denise Hardy and Mrs Zoe Brentall for all of their help and assistance in this project.
- Guyatt GH, Meade MO, Jaeschke RZ, Cook DJ, Haynes RB: Practitioners of evidence based care. BMJ. 2000, 320: 954-955. 10.1136/bmj.320.7240.954.View ArticleGoogle Scholar
- Committee TFP: Curriculum for the foundation years in postgraduate education and training. 2005, Department of HealthGoogle Scholar
- Awonuga AO, Dwarakanath LS, Khan KS, Taylor R: Post-graduate obstetrics & gynaecology trainees' views and understanding of Evidence-based Medicine. Medical Teacher. 2000, 27-29. 10.1080/01421590078779.Google Scholar
- Harden RM: Ten questions to ask when planning a course or curriculum. Medical Education. 1986, 20: 356-365.View ArticleGoogle Scholar
- Morrison J: ABC of learning and teaching in medicine: Evaluation. BMJ. 2003, 326: 385-387. 10.1136/bmj.326.7385.385.View ArticleGoogle Scholar
- Fritsche L, Greenhalgh T, Falck-Ytter Y, Neumayer HH, Kunz R: Do short courses in evidence based medicine improve knowledge and skills? Validation of Berlin questionnaire and before and after study of courses in evidence based medicine. BMJ. 2002, 325: 1338-1341. 10.1136/bmj.325.7376.1338.View ArticleGoogle Scholar
- Taylor R, Reeves B, Mears R, Keast J, Binns S, Ewings P, Khan K: Development and validation of a questionnaire to evaluate the effectiveness of evidence-based practice teaching. Medical Education. 2001, 35: 544-547. 10.1046/j.1365-2923.2001.00916.x.View ArticleGoogle Scholar
- Field A: Discovering statistics using SPSS for Windows. 2000, London, Sage PublicationsGoogle Scholar
- Bowling A: Research Methods in Health. 1997, Buckingham, Open University PressGoogle Scholar
- Howitt D CD: A Guide to Computing Statistics with SPSS 11 for windows. 2003, Harlow, Pearson Education Limited, revisedGoogle Scholar
- Jamieson S: Likert scales: how to (ab)use them. Medical Education. 2004, 38: 1217-1218. 10.1111/j.1365-2929.2004.02012.x.View ArticleGoogle Scholar
- Cohen L, Manion L, Morrison K: Research Methods in Education. 2000, London, Routledge Falmer, 5thView ArticleGoogle Scholar
- Blaikie N: Analysing Qualitative Data. 2003, London, Sage PublicationsGoogle Scholar
- Pell G: Use and misuse of Likert scales. Medical Education,. 2005, 39: 970-10.1111/j.1365-2929.2005.02237.x.View ArticleGoogle Scholar
- Khan KS, Awonuga AO, Dwarakanath LS, Taylor R: Assessments in evidence-based medicine workshops: loose connection between perception of knowledge and its objective assessment. Med Teach. 2001, 23: 92-94. 10.1080/01421590150214654.View ArticleGoogle Scholar
- Boynton P: Administrating, analysing, and reporting your questionnaire. British Medical Journal. 2004, 328: 1372-1375. 10.1136/bmj.328.7452.1372.View ArticleGoogle Scholar
- Olatunbosun OA, Edouard L, Pierson RA: Physicians' attitudes toward evidence based obstetric practice: a questionnaire survey. BMJ. 1998, 316: 365-366.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/7/11/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.