- Research article
- Open Access
Effectiveness of national evidence-based medicine competition in Taiwan
BMC Medical Education volume 13, Article number: 66 (2013)
Competition and education are intimately related and can be combined in many ways. The role of competition in medical education of evidence-based medicine (EBM) has not been investigated. In order to enhance the dissemination and implementation of EBM in Taiwan, EBM competitions have been established among healthcare professionals. This study was to evaluate the impact of competition in EBM learning.
The EBM competition used PICO (patient, intervention, comparison, and outcome) queries to examine participants’ skills in framing an answerable question, literature search, critical appraisal and clinical application among interdisciplinary teams. A structured questionnaire survey was conducted to investigate EBM among participants in the years of 2009 and 2011. Participants completed a baseline questionnaire survey at three months prior to the competition and finished the same questionnaire right after the competition.
Valid questionnaires were collected from 358 participants, included 162 physicians, 71 nurses, 101 pharmacists, and 24 other allied healthcare professionals. There were significant increases in participants’ knowledge of and skills in EBM (p < 0.001). Their barriers to literature searching and forming answerable questions significantly decreased (p < 0.01). Furthermore, there were significant increases in their access to the evidence-based retrieval databases, including the Cochrane Library (p < 0.001), MD Consult (p < 0.001), ProQuest (p < 0.001), UpToDate (p = 0.001), CINAHL (p = 0.001), and MicroMedex (p = 0.024).
The current study demonstrates a method that successfully enhanced the knowledge of, skills in, and behavior of EBM. The data suggest competition using PICO queries may serve as an effective way to facilitate the learning of EBM.
Evidence-based medicine (EBM), a clinical practice consistent with the current best evidence, has been proposed as a core competence to help health professionals improve care quality [1–3]. It integrates clinical epidemiology, biostatistics, research methods, and informatics into health care. The process mainly involves four sequential steps: formulating a well-focused question based on a clinical problem, accessing and verifying relevant evidence from the literature, critical appraising the validity of the contemporary research, and applying the findings to clinical decision-making . A clear answerable question is composed of four components – problem, intervention, comparison, and outcome – referred to by the acronym PICO [5, 6]. In order to match the relevant scientific literatures and improve the retrieval of evidence, it is important to determine what is the problem of patient (P) first, then to what intervention (I) has been done, followed by comparing (C) the effect of different interventions, and finally measuring the outcome (O). PICO has been a major component of teaching evidence-based searching [7, 8].
There are increasing examples illustrating that health professionals hold positive attitude toward EBM [9–12]. Nevertheless, their knowledge and skills pertaining to the implementation of EBM are relatively insufficient [12–15]. Many campaigns have been used to disseminate EBM into health professionals, mostly are organized lectured models, such as education, training programs, and workshops [16–21]. Yet, they often lack interdisciplinary interaction and relevance in their clinical settings. It is imperative for continuing education to maintain the active cognitive learning process in this respect.
Competition may offer an opportunity to attract all disciplines of healthcare professions by introducing the fun and excitement of learning. It has been used as an active tool for acquiring cognitive, effective psychomotor skills and knowledge. Education and competition are intimately related and can be combined in many ways. The role of competition in education has been investigated [22, 23]. However, there is no general agreement as to what constitutes the best way of putting competitions to good use in educating EBM. Thus, the purpose of current study is to introduce a useful model of EBM competition and investigate its impact on the belief, knowledge, skill, barrier, and behavior of the healthcare professionals.
The National EBM competitions in Taiwan
Since 2006 Taiwan Joint Commission of Hospital Accreditation (TJCHA) introduced EBM competitions with collaboration of National Health Research Institutes (NHRI), and later joined with Taiwan Evidence-based Medicine Association (TEBMA) in an effort to transfer EBM into clinical practice. The competition was developed using information infrastructure and game-specific elements. There were two main phases for the contest: preparation and execution.
The initial phase involved preparing the entire framework: competition rules, competition tasks including scenario setting, and judging procedures. The rules are well formulated and transparent to avoid misleading participants (Table 1). The competition teams were classified into two levels – basic and advanced – by the ability of EBM implementation. Overall, the teams composed of non-experienced participants were categorized into basic groups. In contrast, the teams composed of repeat participants were categorized into advanced groups. In addition, the teams from awarded hospitals were requested to participate in the advanced groups.
Five months prior to the EBM competition, NHRI and TJCHA invited all hospitals and medical centers to organizing their healthcare professionals for participation by an official letter stating the details of the competition (Figure 1). Participants were organized as interdisciplinary teams and requested to register at three months or earlier prior to the EBM competition.
The second phase is the execution of actual competition. Participants were requested to determine what the problem, intervention, alternative and outcome are according to the preset scenario. They are required to verify relevant evidences from the literatures, critically appraise the validity of the evidences, and apply the findings to clinical decision-making. Results of the team search and discussion were presented to judges by oral slide presentation in a well-formulated fashion relevant to the scenario. The judgment of the competition was according to an evaluation form including the quality and quantity of PICO, search skills, critical appraisal skills, application and evaluation (Additional file 1) by a team of judges. The winners and their affiliated hospitals were cited and awarded by TJCHA.
A structured questionnaire was developed by the Division of Preventive Medicine and Health Services Research, Institute of Population Health Sciences, NHRI, Taiwan. The targets in this study were healthcare professionals participating in the EBM competitions in the years of 2009 and 2011. The questionnaires were distributed to all participants twice, first at 3 months prior to the competition and second right after the competition (Figure 1).
Questions in this survey were designed by modifying previous validated questionnaires [11, 24], included items for measuring the beliefs in, knowledge of, skills in, barriers toward EBM implementation (Additional file 1). These questions were rated by Likert’s 5-point scale (strongly agree, agree, neutral, disagree, and strongly disagree). In addition, the questionnaire explored the utilization of 7 commonly-used online databases, including the Cochrane Library, MD Consult, MEDLINE/PubMed, ProQuest, UpToDate, Micromedex, and CINAHL (the Cumulative Index to Nursing & Allied Health Literature). Utilization was defined as access in recent three months by Likert’s 5-point scale (more than 12 times/month, 9–12 times/month, 5–8 times/month, 1–4 times/month, and never). The background characteristics – including gender, age, faculty position, director position, working experience, academic degree, and motivation of participation – were further examined.
Content validity was examined by 10 experts with more than 15 years of clinical experience each. The experts were asked to rate the relevance of each question item based on their expertise and to provide editorial recommendations for item improvement or elimination. Items with strong relevance were included in the final questionnaire. The internal consistency of all indexes was estimated by using Cronbach’s coefficient alpha. In this survey, the content validity index of 0.98 and Cronbach’s coefficient alpha of 0.91 indicated sufficient validity and reliability of parameters in the questionnaire.
The Ethical Review Board of the National Health Research Institutes approved the study protocol. The questionnaire was accompanied by an introductory letter stating the purpose of this study and promising confidentiality. Return of the completed questionnaire was considered as consent of participating in the study.
The statistical analyses were conducted using a commercially available program (SPSS 12.0 for Windows, SPSS Inc., Illinois, USA). Categorical variables were analyzed using the chi-square test or Fisher’s exact test. Pair-sample t test was used to compare the values of means between the pretest and posttest questionnaires. Effect sizes were expressed by Cohen’s d. Significance was defined as p < 0.05.
The questionnaires were mailed to a total of 459 participants. Among them, 358 participants completed both pretest and posttest questionnaires, including 162 physicians, 71 nurses, 101 pharmacists, and 24 other allied healthcare professionals. The return rate of valid questionnaires was 78.0%. Participants came from 24 regional hospitals and 15 medical centers in Taiwan. Their background information is summarized in Table 2. Nurses were older and had longer working period than the other health professionals. In addition, nurses were more likely to have a director position than the others. Furthermore, pharmacists tended to have a faculty position than the other health professionals. In all of the four groups, more than 90% of health professionals rated assignment as the motive to participate in the EBM competition.
Impact of EBM competition on the belief, knowledge and skill of participants
After the competition, participants were more likely to believe that EBM is helpful in the decision-making of clinical practice than before (Cohen’s d = 0.156) (Table 3). Furthermore, their reported knowledge of applying EBM principles significantly increased (Cohen’s d = 0.489). Their reported understanding of EBM terminology – including relative risk (RR) (Cohen’s d = 0.235), odds ratio (OR) (Cohen’s d = 0.312), type I error (α error) (Cohen’s d = 0.361), type II error (β error) (Cohen’s d = 0.363), systematic review (Cohen’s d = 0.131), meta-analysis (Cohen’s d = 0.179), and number needed to treat (NNT) (Cohen’s d = 0.337) – also significantly increased. In addition, their reported skills in literature searching (Cohen’s d = 0.257), critical appraisal (Cohen’s d = 0.448) and clinical application (Cohen’s d = 0.496) significantly increased after the competition.
Barriers to the implementation of EBM
Perceived barriers to the implementation of EBM are summarized in Table 4. Difficulty in critical appraisal was the most common barrier. There were significant decreases in a couple of personal barriers after the competition: literature searching (Cohen’s d = 0.160) and forming answerable questions (Cohen’s d = 0.148). In contrast, no significant change was noted in the organizational barriers, including insufficient designated personnel, lack of support from authorities and colleagues.
Behavior of access to the online databases
The impact of competition on participants’ searching behavior is demonstrated in Table 5. The most commonly used database was MEDLINE/PubMed, followed by UpToDate, MicroMedex, Cochrane Library, MD Consult, ProQuest, and CINAHL. During the competition period, the access rate of the following 6 online databases significantly increased – including UpToDate (Cohen’s d = 0.166), MicroMedex (Cohen’s d = 0.095), Cochrane Library (Cohen’s d = 0.442), MD Consult (Cohen’s d = 0.197), ProQuest (Cohen’s d = 0.225), and CINAHL (Cohen’s d = 0.206).
In this study, we introduced the competition as a means to disseminate the implementation of EBM. Although numerous initiatives have emerged to promote EBM [4, 20, 26–28], literature pertaining to examining the effectiveness of competition program is lacking. Our study is the first survey to evaluate the impact of competitions on the beliefs in, knowledge of, skills in, barriers to, and behaviors of EBM. The data demonstrate that the competition may serve as an active and useful measure to bridge the gap of evidence to practice and foster the implementation of EBM.
Educational intervention is the most common way to disseminate EBM. It can improve the knowledge of and skills in EBM. However, the improvement of behavior toward EBM is limited [4, 20, 29, 30]. Therefore, additional methods are required in attempt to improve this step. In our study, we demonstrate that EBM competitions not only enhance the knowledge and skills but also increase the behavior of access to the online databases. These results suggest that an incorporation of competition into education could facilitate EBM learning.
In our study, the abilities to formulate a clear question, retrieve relevant literature, critical appraise study findings have improved significantly. Furthermore, the knowledge and skill of EBM implementation have increased. The competition program demonstrated benefit in helping health professionals overcoming some of the personal barriers to EBM. However, difficulty in critical appraisal although improved, but not statistically significant. It probably would take more time of practice to improve the skill of critical appraisal.
Among all the online databases that the participants have accessed, the reported usage of Cochrane Library increased the most. The Cochrane Library has been regarded as the most important resource of EBM . In addition, the Cochrane Library used well-established PICO queries as a methodological standard to conduct systematic reviews . Therefore it’s not surprising that our participants tended to retrieve the systematic reviews of Cochrane Library while preparing the EBM competition. In contrast, the usage of MEDLINE did not significantly increase in our study. It’s probably because MEDLINE does not sort search results based on PICO queries.
Our data provide several valuable evidences in the strategy for accelerating the dissemination of EBM implementation. First, our competition encourages team-based learning. There are a variety of attributes and dimensions that can be used to formulate a competition. Studies have supported the fact that cooperative task can achieve better learning than individual teaching [33–35]. There are an increasing number of publications showing that interactive team learning can stimulate an energetic discussion and help to consolidate learning in EBM [36, 37]. Nevertheless, cooperative learning is still underutilized in postgraduate medical education. Our results support the finding that establishing multidisciplinary collaborative teamwork for competition can enhance EBM learning. Second, our findings suggest that PICO is a useful tool as a framework for the conduction of competition. The formation of a focused clinical question containing well-articulated PICO elements is one of the methods that have been suggested to search high-quality evidence efficiently . Thus, mastery of skills in PICO will help health professionals in decision makings based on evidence [5, 38]. Third, our competitions used validated grading system to score the results. This system offers easy direction for participants on how to prepare the skills that are required for the competition. Fourth, our competition has a preparation period of 3 months. This allows participants to learn and better understand EBM. In the meantime, it improves participants’ behaviors of EBM implementation.
This study has two strengths. First, our survey is based on the participants coming from nationwide hospitals. Second, our survey respondents in pretest and posttest were completely the same individuals. It enhanced the reliability of the questionnaire survey. With respect to limitations, this study did not include control group. Furthermore, our study is a self-report survey, not an audit of actual practice. Therefore we cannot be sure that these self-reported changes were fully translated into improved clinical care. In addition, our survey investigated the short-term effects of EBM competition. Further studies are needed to determine its long-term impact after the competition terminated.
Our study has introduced a specially constructed game-based learning module for postgraduate health professionals. The EBM competition in our study is bound by rules and organized by specialists. The data indicate that applying competitions as an active learning process of EBM can help motivating the cognitive function in postgraduate education of healthcare professionals. Our data showed an improvement in relation to the implementation of EBM. The results obtained in this study suggest our nationwide structured competition model designed to inspire the learning of EBM may speed up EBM dissemination among healthcare professionals.
Thomas A, Saroyan A, Dauphinee WD: Evidence-based practice: a review of theoretical assumptions and effectiveness of teaching and assessment interventions in health professions. Adv Health Sci Educ Theory Pract. 2011, 16 (2): 253-276. 10.1007/s10459-010-9251-6.
Riba AL: Evidence-based performance and quality improvement in the acute cardiac care setting. Crit Care Clin. 2008, 24 (1): 201-229. 10.1016/j.ccc.2007.09.002.
McCoy JM, Byers JF: Using evidence-based performance improvement in the community hospital setting. J Healthc Qual. 2006, 28 (6): 13-17. 10.1111/j.1945-1474.2006.tb00639.x.
Straus SE, Ball C, Balcombe N, Sheldon J, McAlister FA: Teaching evidence-based medicine skills can change practice in a community hospital. J Gen Intern Med. 2005, 20 (4): 340-343. 10.1111/j.1525-1497.2005.04045.x.
Schardt C, Adams MB, Owens T, Keitz S, Fontelo P: Utilization of the PICO framework to improve searching PubMed for clinical questions. BMC Med Inform Decis Mak. 2007, 7: 16-10.1186/1472-6947-7-16.
Huang X, Lin J, Demner-Fushman D: Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annu Symp Proc. 2006, 359-363.
Balakas K, Sparks L: Teaching research and evidence-based practice using a service-learning approach. J Nurs Educ. 2010, 49 (12): 691-695. 10.3928/01484834-20100831-07.
Lai NM: Dissecting students’ bedside clinical questions using the’PICO’ framework. Med Educ. 2009, 43 (5): 479-480. 10.1111/j.1365-2923.2009.03320.x.
Young JM, Ward JE: Evidence-based medicine in general practice: beliefs and barriers among Australian GPs. J Eval Clin Pract. 2001, 7 (2): 201-210. 10.1046/j.1365-2753.2001.00294.x.
McColl A, Smith H, White P, Field J: General practitioner’s perceptions of the route to evidence based medicine: a questionnaire survey. BMJ. 1998, 316 (7128): 361-365. 10.1136/bmj.316.7128.361.
Jette DU, Bacon K, Batty C, Carlson M, Ferland A, Hemingway RD, Hill JC, Ogilvie L, Volk D: Evidence-based practice: beliefs, attitudes, knowledge, and behaviors of physical therapists. Phys Ther. 2003, 83 (9): 786-805.
Chiu YW, Weng YH, Lo HL, Hsu CC, Shih YH, Kuo KN: Comparison of evidence-based practice between physicians and nurses: a national survey of regional hospitals in Taiwan. J Contin Educ Health Prof. 2010, 30 (2): 132-138. 10.1002/chp.20070.
Sherriff KL, Wallis M, Chaboyer W: Nurses’ attitudes to and perceptions of knowledge and skills regarding evidence-based practice. Int J Nurs Pract. 2007, 13 (6): 363-369. 10.1111/j.1440-172X.2007.00651.x.
McAlister FA, Graham I, Karr GW, Laupacis A: Evidence-based medicine and the practicing clinician. J Gen Intern Med. 1999, 14 (4): 236-242. 10.1046/j.1525-1497.1999.00323.x.
Hadley JA, Wall D, Khan KS: Learning needs analysis to guide teaching evidence-based medicine: knowledge and beliefs amongst trainees from various specialities. BMC Med Educ. 2007, 7: 11-10.1186/1472-6920-7-11.
Weberschock TB, Ginn TC, Reinhold J, Strametz R, Krug D, Bergold M, Schulze J: Change in knowledge and skills of Year 3 undergraduates in evidence-based medicine seminars. Med Educ. 2005, 39 (7): 665-671. 10.1111/j.1365-2929.2005.02191.x.
Taylor RS, Reeves BC, Ewings PE, Taylor RJ: Critical appraisal skills training for health care professionals: a randomized controlled trial [ISRCTN46272378]. BMC Med Educ. 2004, 4: 30-10.1186/1472-6920-4-30.
Shuval K, Shachak A, Linn S, Brezis M, Feder-Bubis P, Reis S: The impact of an evidence-based medicine educational intervention on primary care physicians: a qualitative study. J Gen Intern Med. 2007, 22 (3): 327-331. 10.1007/s11606-006-0055-6.
Ghali WA, Saitz R, Eskew AH, Gupta M, Quan H, Hershman WY: Successful teaching in evidence-based medicine. Med Educ. 2000, 34 (1): 18-22. 10.1046/j.1365-2923.2000.00402.x.
Forsetlund L, Bradley P, Forsen L, Nordheim L, Jamtvedt G, Bjorndal A: Randomised controlled trial of a theoretically grounded tailored intervention to diffuse evidence-based public health practice [ISRCTN23257060]. BMC Med Educ. 2003, 3: 2-10.1186/1472-6920-3-2.
Bradley P, Oterholt C, Herrin J, Nordheim L, Bjorndal A: Comparison of directed and self-directed learning in evidence-based medicine: a randomised controlled trial. Med Educ. 2005, 39 (10): 1027-1035. 10.1111/j.1365-2929.2005.02268.x.
Morritz T, Seehafer RW, Maatz-Majestic E: A student competition to develop an innovative alcohol education strategy. J Am Coll Health. 1993, 41 (6): 283-286. 10.1080/07448481.1993.9936346.
Htwe TT, Sabaridah I, Rajyaguru KM, Mazidah AM: Pathology crossword competition: an active and easy way of learning pathology in undergraduate medical education. Singapore Med J. 2012, 53 (2): 121-123.
Chiu YW, Weng YH, Lo HL, Hsu CC, Shih YH, Kuo KN: Impact of a nationwide outreach program on the diffusion of evidence-based practice in Taiwan. Int J Qual Health Care. 2010, 22 (5): 430-436. 10.1093/intqhc/mzq049.
Cohen J: A power primer. Psychol Bull. 1992, 112 (1): 155-159.
Nicholson LJ, Warde CM, Boker JR: Faculty training in evidence-based medicine: improving evidence acquisition and critical appraisal. J Contin Educ Health Prof. 2007, 27 (1): 28-33. 10.1002/chp.90.
Markey P, Schattner P: Promoting evidence-based medicine in general practice-the impact of academic detailing. Fam Pract. 2001, 18 (4): 364-366. 10.1093/fampra/18.4.364.
Carboneau C: Using diffusion of innovations and academic detailing to spread evidence-based practices. J Healthc Qual. 2005, 27 (2): 48-52. 10.1111/j.1945-1474.2005.tb01117.x.
Shuval K, Berkovits E, Netzer D, Hekselman I, Linn S, Brezis M, Reis S: Evaluating the impact of an evidence-based medicine educational intervention on primary care doctors’ attitudes, knowledge and clinical behaviour: a controlled trial and before and after study. J Eval Clin Pract. 2007, 13 (4): 581-598. 10.1111/j.1365-2753.2007.00859.x.
McCluskey A, Lovarini M: Providing education on evidence-based practice improved knowledge but did not change behaviour: a before and after study. BMC Med Educ. 2005, 5: 40-10.1186/1472-6920-5-40.
Weng YH, Hsu CC, Shih YH, Lo HL, Chiu YW, Kuo KN: Dissemination of systematic reviews in a hospital setting: a comparative survey for spreading use of the Cochrane Library. Postgrad Med J. 2012, 88 (1043): 511-514. 10.1136/postgradmedj-2011-130652.
Hoogendam A, de Vries Robbe PF, Overbeke AJ: Comparing patient characteristics, type of intervention, control, and outcome (PICO) queries with unguided searching: a randomized controlled crossover trial. J Med Libr Assoc. 2012, 100 (2): 121-126. 10.3163/1536-5050.100.2.010.
Tan NC, Kandiah N, Chan YH, Umapathi T, Lee SH, Tan K: A controlled study of team-based learning for undergraduate clinical neurology education. BMC Med Educ. 2011, 11: 91-10.1186/1472-6920-11-91.
Sisk RJ: Team-based learning: systematic research review. J Nurs Educ. 2011, 50 (12): 665-669. 10.3928/01484834-20111017-01.
Kanthan R, Mills S: Cooperative learning in the first year of undergraduate medical education. World J Surg Oncol. 2007, 5: 136-10.1186/1477-7819-5-136.
Tai BC, Koh WP: Does team learning motivate students’ engagement in an evidence-based medicine course?. Ann Acad Med Singapore. 2008, 37 (12): 1019-1023.
Hunt DP, Haidet P, Coverdale JH, Richards B: The effect of using team learning in an evidence-based medicine course for medical students. Teach Learn Med. 2003, 15 (2): 131-139. 10.1207/S15328015TLM1502_11.
Villanueva EV, Burrows EA, Fennessy PA, Rajendran M, Anderson JN: Improving question formulation for use in evidence appraisal in a tertiary care setting: a randomised controlled trial [ISRCTN66375463]. BMC Med Inform Decis Mak. 2001, 1: 4-10.1186/1472-6947-1-4.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/13/66/prepub
The authors are grateful to the participants who gave their time in this study. This work was supported by research grants from the National Health Research Institutes, Taiwan [grant number: PH-099-SP02, PH-098-SP03, and 96AI-HDSP05-018]. We thank Ya-Hui Shih and Chih-Wen Hsiao for their assistance in the collection and data input of questionnaires.
The authors declare that they have no competing interests.
All authors contributed substantially to the intellectual conception of this manuscript. YHW wrote the initial draft of the paper. YWC served as the principal investigator in this work and was responsible for the study design and the drafting of the manuscript. KNK, CYY, HHL, CC, HLL, WCL contributed to the acquisition, analysis and interpretation of the data. All authors contributed to the critical revision of this draft and approved the final manuscript for publication.
Electronic supplementary material
Authors’ original submitted files for images
Below are the links to the authors’ original submitted files for images.
About this article
Cite this article
Weng, YH., Kuo, K.N., Yang, CY. et al. Effectiveness of national evidence-based medicine competition in Taiwan. BMC Med Educ 13, 66 (2013). https://doi.org/10.1186/1472-6920-13-66
- Healthcare Professional
- Cochrane Library
- Online Database
- Answerable Question
- National Health Research Institute