The design, fate and impact of a hospital-wide training program in evidence-based medicine for physicians – an observational study

Abstract

Background

Many doctors fail to practice Evidence-Based Medicine (EBM) effectively, in part due to insufficient training. We report on the design, fate and impact of a short learner-centered EBM train-the-trainer program aimed at all 2400 doctors at the Karolinska University Hospital in Sweden on the heels of a tumultuous merger, focusing particularly on whether it affected the doctors’ knowledge, attitudes and skills regarding EBM.

Methods

We used a validated EBM instrument in a before-and-after design to assess the impact of the training. Changes in responses were analyzed at the individual level using the Wilcoxon matched pairs test. We also reviewed documentation from the program – including the modular EBM training schedule and the template for participants’ Critically Appraised Topic reports – to describe the training’s content, design, conduct, and fate.

Results

The training, designed to be delivered in modules of 45 min totaling 1.5 days, failed to reach most doctors at the hospital, due to cost cutting pressures and competing demands. Among study participants (n = 174), many reported suboptimal EBM knowledge and skills before the training. Respondents’ strategies for solving clinical problems changed after the training: the proportion of respondents reporting to use (or intend to use) secondary sources “Often/very often” changed from 5 % before the training to 76 % after the training; in parallel, reliance on textbooks and on colleagues fell (48 to 23 % and 79 to 65 %, respectively). Participants’ confidence in assessing scientific articles increased and their attitudes toward EBM became more positive. The proportion of correct answers in the EBM knowledge test increased from 52 to 71 %. All these changes were statistically significant at p < 0.05.

Conclusions

Many study participants, despite working at a university hospital, lacked basic EBM knowledge and skills and used the scientific literature suboptimally. The kind of short learner-centered EBM training evaluated here brought significant improvements among the minority of hospital doctors who were able to participate and, if applied widely, could contribute to better, safer and more cost-effective care.

Background

Clinicians face the daunting task of addressing patient needs by drawing on the ever-increasing amount of knowledge yielded by basic science and clinical research. The development of evidence-based medicine (EBM) can be seen as an effort to enhance the application of scientific knowledge to achieve better, safer and more cost-effective care [1]. EBM proponents argue that “Doctors owe it to themselves and their patients to make sure that they keep up with what’s new and important.” [2] Several studies have found, however, that there is a considerable gap between the best evidence and the care actually delivered [3–5].

A number of barriers to EBM practice have been identified, including time constraints, limited access to electronic information resources, poor information-searching skills, lack of motivation and an inhospitable institutional culture [6, 7].

Although EBM has been widely accepted as a cornerstone of good healthcare [8–10], many clinicians lack adequate familiarity with its principles and practices [11–13]. While EBM training has been integrated in medical school curricula, many currently practicing physicians graduated before this development. Furthermore, a systematic review of the relationship between clinical experience and the quality of health care suggests that “physicians who have been in practice for more years and older physicians possess less factual knowledge, are less likely to adhere to appropriate standards of care, and may also have poorer patient outcomes [14].”

In theory, training in EBM should help [15]. Numerous examples of EBM training interventions have been described [16–19] but evidence on the effectiveness of different EBM training designs has been slow to accumulate [20, 21]. The heterogeneity of training programs and the methodological limitations of many evaluative studies have made it difficult to conclude what works best [22–25].

While we already know that EBM is best learned when the training is learner-centered, patient-related, participant-activating, and problem-based [26, 27], it is less clear how best to organize effective EBM training throughout an organization, to reach busy clinicians [28]. We report here on the design, fate and impact on participants’ EBM knowledge, attitudes and skills of an EBM train-the-trainer intervention at a large university hospital aimed at reaching all of its physicians. The training rested on the assumption that many doctors had limited EBM knowledge and skills, with suboptimal ways of drawing on the scientific literature, and that a short EBM training program could influence this situation in a desirable direction.

Methods

Study setting

The EBM training was designed to reach all physicians at the Karolinska University Hospital, a public tertiary care academic medical center with two main campuses in Stockholm, Sweden, with approximately 15,000 employees, including 2400 salaried physicians.

The EBM training

On the heels of a tumultuous hospital merger [29], the authors received funding to develop and offer training to all doctors in how to apply EBM in practice. In order to be able to reach all of the hospital’s 2400 doctors, we designed a two-phase “cascade” training program. Phase 1 was a train-the-trainer intervention, whereby some 100 doctors from all departments underwent training in one of six course iterations between October 2005 and March 2006 to become EBM teachers. In Phase 2, they requested materials for training their departmental colleagues.

In Phase 1, trainers attended 2 days of interactive training on the EBM process in a computer lab, where they could perform searches as the course progressed, followed by self-study over 2–3 weeks or more to apply EBM to a clinical question from their own practice and develop a Critically Appraised Topic (CAT) report [30] – a structured way to summarize evidence on a focused clinical question – modeled on BestBets (http://www.bestbets.org/) (see template in Additional file 1). The training concluded with a half day follow-up session for participants to report on their CATs and receive feedback from peers and faculty. The course covered the definition of EBM – integrating the best available scientific evidence with clinical experience and the patient’s perspective – its underlying principles and the EBM process: formulating a good question (PICO-format [31]; for Patients-Intervention-Control-Outcomes); searching for and critically appraising relevant literature; and applying findings in the clinical situation. The training included brief teaching sessions, demonstrations of relevant search engines and databases with support from hospital librarians, individual practice with faculty support, and self-directed study. The training used the Swedish edition of a course book on EBM [32], and a workbook designed by one of the authors (JN) specifically for this initiative in the hospital’s context.

In Phase 2, the 100 newly trained teachers were invited to request sets of the same course materials – including the course book, the workbook and the electronic template for writing a CAT – free of charge for their departmental colleagues. They also received presentation slides to explain training concepts, such as the 4-step EBM process outlined above, a heuristic for how to identify the best available evidence (see Fig. 1), critical appraisal skills, and how to communicate regarding evidence and care decisions with patients. The trainers also received a template modular curriculum (Additional file 2) which could be broken down into 45 min sessions, whereby the course content could be covered in a total of 1.5 days (12 h) at a pace tailored to each local context. The curriculum covered the same content as the trainers’ own training, including the CAT assignment, to be completed at participants’ leisure, before the concluding sessions.

Fig. 1

A heuristic for identifying the best available evidence. This is the order of resources recommended in the training for identifying the best available evidence [67–76]

Trainers were advised to solicit assistance from colleagues in their own department with special interest and skill in various aspects of the EBM process, e.g., statistics or epidemiological study design. Several trainers also enlisted hospital medical librarians to assist with training in information retrieval.

After completing their own training, trainers received a checklist for how to prepare and run the training themselves. It instructed them to contact their Department Chief to agree on when and how to teach the department’s physicians, to request course materials, to alert their colleagues about the upcoming training and ask them each to identify and bring a clinical case from their practice for use in the training, to prepare for and deliver the training, to hand out course certificates to all participants who completed a CAT, and to collect and return all questionnaires to this study’s authors.

While the training was awarded funding by the hospital management, and was meant to reach all of the hospital’s physicians, scheduling the training in the departments was in effect voluntary for department chiefs. There was no follow-up by hospital management on the dissemination of the EBM training.

Study design and questionnaire instrument

For this observational study of the EBM training, we found the “Critical appraisal skills programme [CASP] workshop evaluation questionnaire” developed by Taylor et al. [33], geared to practicing clinicians, to be most appropriate for our purposes. This questionnaire captures demographic information (age, sex, academic training, and professional rank); current approaches to keeping up to date and to solving clinical problems; reading habits; attitudes to EBM; and, finally, knowledge of EBM assessed through 18 true-false statements. It is “a valid tool for measuring the impact of [EBM] training on participants’ knowledge, and attitudes toward [EBM].” ([33] p. 546) We translated the questionnaire into Swedish and adapted it to the local setting.

Using a before-and-after study design, we asked participants to fill in the questionnaire at the beginning of the first training session and then at the end of the concluding session, at least 2–3 weeks, and sometimes several months, later, and to use the same identifying information (Swedish social security number or cellular phone number) both times. The intention was for participants to have completed their CAT assignment at that time. The trainers received printed questionnaires to give to their colleagues, and were asked to use the same before-and-after design as they had experienced themselves.

We linked responses in pre- and post-training questionnaires at the individual level. In addition to descriptive statistics, we used Wilcoxon matched pairs test at the individual level regarding i) reading patterns (for keeping up-to-date and for solving clinical problems) and ii) responses before and after the training with respect to confidence in assessing different aspects of a published paper, approaches to solving clinical problems, attitudes to EBM, and demonstrated EBM knowledge.

For this study, we also reviewed our documentation from the planning and conduct of the training program at the hospital. This included course materials, records of course material requests, project planning documents, course evaluations and correspondence with participants and hospital managers. We drew on these materials, and our roles as faculty in the training, to describe the training program and its fate, as recommended by evaluation researchers [34].

Research ethics

This study received ethical clearance from the Regional Board for Vetting of Research Ethics in Stockholm (#2006/33-31). Participants received written and oral information about the study and the voluntary nature of their participation. They were informed that their responses would be treated anonymously once the pre-post version linking was complete and that no information that could be linked to particular individuals would be reported. They handed in each completed questionnaire in a sealed envelope, knowing that they thereby explicitly expressed their informed consent to participate in the study.

Results

Reach of the training and respondent demographics

Phase 1, training-the-trainers, reached 104 doctors. The experience of these initial trainers varied; a few managed to train all their departmental colleagues, but most only reached some of their colleagues, and often only for part of the training. In several departments no training was scheduled at all, reportedly due to cost cutting pressures and staffing shortages. Ultimately, between February of 2006 and June of 2007, 60 of the 104 trained trainers requested materials for 1472 colleagues (range: 10–70 sets of materials per department), or 66 % of hospital physicians, across 42 (81 %) of the 52 departments invited to the initiative. We lack data on how many physicians actually received the materials or the training. We received 462 pre-training questionnaires (104 (100 %) from Phase 1 trainers and 358 (24 %, n = 1472) from their Phase 2 participants, which combined equals 19 % of the hospital’s physician population). Post-training we received 258 questionnaires (88 (85 %) and 170 (18 %), respectively, equal to 11 % of all physicians). Due to missing ID information, we were only able to connect pre- and post-training questionnaires for 174 individuals (63 (61 %) and 111 (7.5 %), equal to 7.2 % of all physicians). We analyzed questionnaire responses in this dataset, starting with respondent demographics (Table 1).

Table 1 Respondent demographics

Information seeking behavior

Keeping up to date and solving clinical problems

When asked before the training “What types of resources do you use to keep up to date?”, the most common responses were consulting a colleague (79 % of respondents reported relying on colleagues “often/very often”), Internet resources, e.g., PubMed and Google (56 % of respondents reported using internet resources “often/very often”), and textbooks (48 % of respondents reported using textbooks “often/very often”). Less common responses were journal articles, e.g., original research reports and review articles (44 % of respondents reported using these “often/very often”), electronic resources/databases available from the hospital library (20 % of respondents reported using these “often/very often”), and secondary journals and clinical practice guidelines, e.g., the Cochrane Library, Clinical Evidence, and UpToDate (5 % of respondents reported using these “often/very often”).

Asked about their weekly reading patterns, respondents reported reading fewer articles (p < 0.0001), and spending less time (p < 0.0001), to solve clinical problems than to keep up to date (Fig. 2).

Fig. 2

Reading to keep up to date and to solve clinical problems. The number of articles respondents reported that they read (above), and the time they spent reading articles (below), on average per week to keep up to date (dotted blue) and to solve a clinical problem (solid red)

Changes in reported information seeking strategies and confidence in assessment skills

The respondents’ intended use of resources to solve clinical problems changed after the training (Table 2), as did their confidence in their assessment skills (Table 3).

Table 2 Resources to solve a health care problem

Table 3 Confidence in assessing studies

Impact on attitudes to EBM

Asked about their views regarding critical appraisal and the use of evidence, participants agreed particularly with three statements – even more so following the training: “by practicing EBM we increase the chances for effective care” (90 % responding “agree/agree strongly” before the training; 97 % after), “systematic reviews play a key role in informing evidence-based decision-making (from 84 to 94 %) and “study design is important in article selection” (from 80 to 92 %). Conversely, a greater proportion of respondents disagreed with the following statements after the training: “EBM has become a buzz word which does not add much to patient care” (70 % responding “disagree/disagree strongly” before the training; 91 % after it) and “evidence-based decision-making is cookbook medicine” (from 44 to 72 %). All of these changes were statistically significant (p = 0.024 or less).

Changes in EBM Knowledge

The proportion of correct answers to the 18 true/false statements about the application of critical appraisal skills increased following the training, as the proportion of “Don’t know/blank” answers declined (Table 4; p < 0.0001).

Table 4 Test of critical appraisal skills

Trainers and their participants improved equally much after the training, although trainers started at a higher proportion of correct answers before the training, (trainers went from an average of 10.1 correct answers before to 13.7 correct answers after the training while their participants went from 9.0 to 12.3 correct answers on average, both at p < 0.0001).

Clinical application and participant experience

CAT reports addressed contemporary clinical problems where participants sought to compare current practice with the best available evidence. They included, for example, assessment of whether to operate or practice watchful waiting for patients with peptic ulcer perforation; whether anticoagulation treatment affects the pregnancy outcome among women with multiple spontaneous abortions of unknown cause; and whether local injection with a corticosteroid would prevent recurrence of keloid formation on the ear. Participants indicated that they would change their practice in light of their CAT findings and, in some cases, that their CAT had prompted updating of departmental clinical guidelines.

Participants’ course evaluations were predominantly favorable regarding both the content and educational design. Free-text comments (translated from Swedish) included:

“I’ve gained a structured approach and thereby a lower threshold for applying EBM.”

“[The training introduced] important search engines to get answers to clinical questions.”

“EBM isn’t so scary and difficult – you have defused it very well.”

“I ought to have had this type of training at the beginning of my career.”

Discussion

This study supports the initial assumptions that participating doctors had limited EBM knowledge and skills, with suboptimal ways of drawing on the scientific literature, and that a short EBM training program could influence this situation in a desirable direction. Before the training, participants reported that they most commonly relied on colleagues to solve clinical problems, mirroring studies elsewhere [35, 36]. This is problematic [37] since colleagues’ opinions may not reflect the best available evidence. Respondents’ second most common pre-training resource was Internet search engines (PubMed, Google) while only a few used readily available secondary sources of evidence, such as the Cochrane Library, generally recognized as providing the best evidence in a usable and understandable format [38]. The training instead promoted another priority for information seeking (see Fig. 1). This priority is reflected in post-training responses regarding which sources participants anticipated using to solve clinical problems after the training, e.g., the propensity to use secondary sources of evidence, which rose from 5 to 76 % – a large and welcome shift [28].

Participants reported reading more articles and spending more time keeping up to date than solving clinical problems. This can be characterized as learning “just-in-case” to prepare for potential future clinical problems – rather than drawing on the literature in a more focused manner guided by actual current patient needs – learning “just-in-time” [27, 39]. Just-in-time learning involves the use of PICO elements, integral to the EBM process [31]. Cheng [40] and Green et al. [41]. have found that the more senior the professional, the more likely she or he is to use just-in-time learning. Most respondents in this study were clearly senior (more than 50 % held MD-PhD degrees) and it is unclear why our results are at odds with previous studies. The training promoted just-in-time learning as a way to manage the “information overload” caused by the ever-increasing number of scientific publications in medicine. No one can keep abreast of all relevant developments concerning their field of practice by reading new publications “just-in-case”; certainly not if reading on average 1–2 articles per week.

Participants’ confidence in appraising scientific articles – particularly study design, bias, generalizability, and the general worth of an article – increased following the training. Furthermore, a greater proportion of respondents agreed that EBM increases the chances for effective care and disagreed with the claim that EBM is “cookbook medicine”. Contrary to earlier research [21], this supports the idea that doctors’ attitudes and skills can be influenced, even by a short training program [23, 42, 43]. We surmise that the more favorable clinicians’ attitudes to EBM are, the greater will be their propensity to actually practice EBM, although this has proven difficult to demonstrate empirically [43, 44]. Prior to the training, the proportion of correct answers to the true/false statements designed to gauge respondents’ critical appraisal skills was similar to what could be achieved through random guesswork. After the course, the proportion of correct answers increased from 52 to 71 % and “don’t know” responses decreased (from 31 to 13 %). The impact was similar among trainers and their phase 2 trainees, suggesting the potential effectiveness of using a train-the-trainer approach [45]. The changes associated with this short training were substantial and encouraging, even if there was room for further improvement, here as in other settings [22, 25, 42, 46].

The curricular design in earlier EBM-studies has often been poorly described [21] which is unfortunate, since it is essential for understanding the mechanisms of training effectiveness [34, 47]. The design reported here, while involving classroom sessions, exhibits most features of the highest level in a research-based hierarchy of effective EBM teaching and learning methods [27]. We hypothesize that the learner-centered, patient-related, participant-activating, problem-based design – with modules that could be delivered in 45 min installments – and the concluding CAT assignment, contributed to the impact of the training. An international survey of EBM teachers identified language as a barrier to learning in non-Anglophone countries [48]. In our study setting, although proficiency in English is widespread, providing instruction and teaching materials in Swedish might have promoted the training’s impact.

While we learned that participants’ CAT reports directly led to changes in local practice and to departmental guidelines in a few cases – indicating that the training was clinically relevant – further research is needed to unpack the effectiveness of this and other curricular designs, including e-learning formats [23, 24, 49–51]. While many e-learning designs lack room for interactivity and higher-level thinking [52], the present design including the CAT assignment, provided such opportunities. This should be considered in the on-going debate about effective educational designs [24].

The training, though relatively short, did not reach as far as intended, despite the centrality of EBM to the mission of a university hospital. We have no indications that the fate of the training, i.e., to not conduct it at all or only partially in some departments, was due to a lack of interest in EBM as such on the part of clinicians and managers, or that EBM knowledge was already considered sufficient there. It is possible that the Phase 1 training was too limited to give all trainers enough confidence in their ability to replicate it in Phase 2 in their own departments. Since participants gave very favorable feedback on the training, however, we have no reason to believe that the training itself discouraged spread. Instead, competing demands seem to have hampered the full dissemination of the training, possibly related to post-merger cost cutting and re-organization [29], mirroring challenges found in similar situations elsewhere [53, 54]. The hospital leadership, after funding the training, did not ensure that it reached all hospital physicians; some department chiefs reported that while they supported the training in principle, they were unable under the circumstances to make room for it. The limited reach of the EBM training thus illustrates how leadership support is essential for organizational improvement initiatives [55] and how its absence reduces the likelihood of success [56].

Study limitations and methodological considerations

Firstly, we received completed questionnaires only from a minority of potential participants at the hospital. While course materials were distributed for over 1500 doctors, we only received 462 pre-training and 258 post-training questionnaires; the training clearly failed to reach the entire intended audience. We do not fully know why. In phase 2, the training was the responsibility of the trainers and their departmental chiefs and colleagues. The limited reach of the training signals a gap between the intention – as initially proposed to, and endorsed by, hospital leadership – and the ability/propensity/willingness among departmental chiefs and physicians to actually carry out the training, as indicated above.

While the training thus had a limited reach, we believe it is still important to report on the data we were able to collect, both because they indicate important possibilities to enhance EBM practices and because the challenges in reaching all intended participants and in data collection can inform future interventions and studies [34]. The questionnaire responses should be generalized to the entire population of physicians at the study hospital (or beyond) with caution since we cannot determine how representative the respondents are. That said, they do show that it is possible to influence university hospital physicians’ EBM attitudes and knowledge in a desirable way with this type of short training program, and that a train-the-trainer design can extend this influence also to a phase 2. If we can assume that participants were more, rather than less, positive towards EBM than the entire population of physicians, the findings are both disconcerting and encouraging. They are disconcerting because at baseline, EBM knowledge and practice left much to be desired. They are encouraging because it was possible, even with such a short training, to achieve a significant change for the better.

The final dataset included 174 respondents for whom we were able to connect questionnaires before and after the training (a strategy employed also in a cluster randomized controlled trial comparing two EBM training modalities [46]). Rather than performing analyses using all questionnaires at group level (yielding a larger study population), we chose to restrict our analyses to this dataset with before-and-after responses linked for each individual to gain greater confidence when assessing the direction and size of changes. Respondents were asked to identify themselves on both the pre- and post-training questionnaire; meant to be cost-effective, this turned out to be an ineffective design. A similar challenge in capturing respondent identity occurred in a German setting [18]. Better ways to link repeated questionnaires at an individual level, while protecting respondents’ integrity, are clearly needed.

Secondly, the limited scope of the evaluation using the validated questionnaire [33], with its restricted focus on reported reading patterns, critical appraisal, knowledge, skills and attitudes regarding EBM, did not enable us to assess changes in participants’ actual practice patterns or related patient outcomes. Intended future practice patterns following the training may not be borne out as reported. Respondents may have been susceptible to social desirability, thus projecting overly favorable practice changes [57]. The fact that the questionnaire was self-administered and that responses were treated confidentially should limit this tendency [58]. The challenge of linking EBM training to changes in clinical practice and patient outcomes is not unique for this study but remains a worthy goal for methodological development [20, 23, 25, 51, 59–61]. A rare example of such linking showed that an EBM-intervention led to more evidence-based practice [19]. A systematic review of 113 EBM training assessment studies found that only 10 % (n = 11) measured change in participant behavior; only 3 % (n = 4) measured changes in the delivery of care [62]. The authors deemed it “surprising that so few teaching efforts were accompanied by appropriately developed assessments.” Related to concerns about our training’s impact on practice are concerns about its durability. While not possible for us (and rare in reviews of studies [23]), it would have been interesting to follow respondents over time, to assess whether EBM skills and attitudes persisted and developed or dwindled over time.

Thirdly, a before-and-after study design carries inherent potential weaknesses, particularly regarding the potential confounding effect of a secular trend in EBM knowledge. The use of a control group, and random assignment of study participants to either training or control group, while not feasible in our situation, could have strengthened our confidence in attributing the changes in participants’ reading patterns, knowledge and attitudes, to the EBM training and has subsequently been employed elsewhere [62–64]. We have no particular reason, however, to believe that any secular trend or other concurrent initiatives confounded the effects of the training captured by questionnaires before and after the training. Using a similar design to evaluate an e-EBM training program, Kulier et al. noted that “the assessments before the course served as control for each individual” and that “we can be reasonably sure the gain in knowledge was due to” the training [42].

Given that the questionnaires were similar before and after the training, respondents might remember some of the questions (e.g., true/false statements) from before the training and then identify the correct answer before completing the after-training questionnaire. If so, the before-training survey played a kind of diagnostic role, by alerting respondents to areas where they needed to gain more knowledge. Any learning effects then arguably were real, in the sense that participants had to know the subject matter to get the answers right. We did not go over, or hand out, the correct answers to the true/false statements at any point – memorizing questionnaire item responses by heart thus was not possible. Therefore, we suggest that the changes in responses are attributable to learning from the training.

Implications for practice and further research

Our findings suggest that not all clinicians – in this case at an academic medical center – possessed basic EBM knowledge and skills nor used the scientific literature optimally. This can help explain the observed gaps between the best available evidence and clinical practice [3–5]. Given that some physicians have not had any EBM training, and that the short training we designed and report on here had a favorable impact on EBM knowledge, attitudes and skills, we suggest that this kind of training can be beneficial and should be offered widely. It remains an important research task to assess the impact on clinical practice and patient outcomes of such training initiatives, as well as to develop and evaluate educational designs and clinical support systems that enable clinicians to learn about EBM and integrate it into their daily practice within existing resource constraints [65, 66].

Conclusions

Given how central EBM is to the mission of doctors and healthcare organizations, there is room for substantial improvement regarding its application in practice. The short EBM training evaluated in this study promoted knowledge, skills and attitudes conducive to such application among the minority of hospital doctors who were able to participate. If similar training were offered and supported widely, with concurrent evaluative studies, it could help improve clinical practice and patient health while developing the knowledge base for doing so.