Appraising and applying evidence about a diagnostic test during a performance-based assessment

Evidence-based medicine (EBM) has been described as the "conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients" [1]. Thus, to practice EBM clinicians need to critically appraise articles in the medical literature and then apply the evidence to specific patients. In the area of diagnostic testing, EBM requires the use of Bayesian inference so that appraised evidence can be used in the evaluation of a specific patient [2].

Nearly all medical schools now provide their students with instruction on EBM [3]. Students at the University of Iowa Carver College of Medicine receive instruction on EBM during required course work during their two preclinical years. Medical students are introduced to EBM in a series of lectures in the first year, including one on the evaluation of diagnostic tests that introduces students to the concept of test characteristics and probability revision. Other lectures focus on critiquing the medical literature. In later semesters, students are asked to complete evidence-based projects and are given further training on the use of Bayes' Theorem. In addition, our students have a required two-week Laboratory Medicine clerkship taken during the third or fourth year. With the aid of a clinically oriented textbook [4], students on this rotation are expected to master the concepts underlying diagnostic testing. To complete this clerkship, students need to demonstrate comprehension of test performance characteristics and probability revision by passing an exam in which they are asked to calculate the sensitivity, specificity, positive predicative value, and negative predictive value for two different testing procedures.

In undertaking this research, we wanted to assess the critical appraisal and probability revision skills of our medical students shortly before graduation within the domain of diagnostic testing. We specifically wanted to evaluate medical students who had previously demonstrated competency at probability revision in a classroom setting. Because EBM is designed to support the delivery of clinical care, transfer of these skills from the classroom to the exam room seemed an appropriate measure of instructional success. We were also interested in comparing the skills of our students nearing graduation to physicians just entering the Internal Medicine residency at our medical center with the intention of comparing the skills of our students to those of students who had trained at other medical schools.

Thirty-eight medical students on the Psychiatry clerkship completed the critical appraisal exercise. Twenty-eight of these students were 4^th year medical students who had demonstrated mastery of Bayesian probability revision during a preceding Laboratory Medicine clerkship. The data from these students were analyzed for this report. The other 10 students were dropped from the analysis because they were either M3s or M4s who had not yet completed their Laboratory Medicine clerkship.

Twenty-two first year Internal Medicine residents participated in this performance-based assessment and completed the EBM appraisal task and Bayesian inference exercises during their clinical skills assessment. For this report we used the data from the 15 residents who were recent graduates of US medical schools. The data from the 7 non-US graduates were dropped from the analysis.

All 15 residents (100%) correctly identified whether there was an independent, blind comparison of the test with a reference standard. Eight (53%) correctly identified the reference standard. Eleven (73%) correctly assessed whether referral bias was present, and fourteen (93%) were able to comment on the generalizability of the study. Thirteen (87%) of the Internal Medicine residents were able to identify the sensitivity and specificity of the test for congestive heart failure. The internal reliability of the 6-item appraisal work sheet was 0.66 for this group of subjects. On average, a resident, who had just recently graduated from a US medical school, correctly answered 81% of the questions related to critical appraisal but none of the 15 residents (0%) were able to revise the pretest probability given a test result.

Over the past decade, EBM has become a major driving force world wide, impacting medical education, policymaking, and research. The teaching of evidence-based medicine has been increasingly integrated into curricula at all levels of medical education as advocated by the Medical School Objectives Program developed by the Association of American Medical Colleges (AAMC) [8]. Like most medical schools, the Carver College of Medicine at the University of Iowa has integrated EBM into its curriculum and our data indicates that students who are soon to graduate can demonstrate proficiency at critically appraising an article about diagnostic testing. But in a simulated clinical encounter, few students are able take the last step of using EBM in diagnosing medical illness. We found almost uniform failure of our students to correctly revise a pretest probability of disease given a test result despite their earlier demonstration of competency with Bayes' Theorem on their Laboratory Medicine clerkship. This finding raises questions about whether our students can fully utilize their EBM training in the clinical setting.

Our finding that incoming residents demonstrate similar levels of skill at critically appraisal and also are unable to revise a pretest probability implies that our medical students' skill deficit is not solely due to a local curricular problem. Because the residents had only just graduated from 12 different US medical schools, this finding suggests that many medical school graduates are able to critically appraise articles on diagnostic testing but few are able to take the next step- that of revising the probability of disease given a test result.

There are few other assessments of the EBM skills of graduating medical students using simulated clinical encounters. In an earlier study, 3^rd year medical students demonstrated good performance at critically appraisal [9], a finding which is similar to ours. These results generally replicated classroom-based studies on the success of critical appraisal instruction [10]. To our knowledge, only one other study has investigated whether students are able to integrate critically appraised information about a diagnostic test into clinical decision making. The results of this earlier study also raised concerns of students' abilities to transfer their EBM skills to simulated clinical encounters [11].

Our study has a number of limitations. The first is the very small sample size. However, it is unlikely that a larger sample size would change our conclusion that by the end of medical school students have largely mastered critical appraisal of an article on diagnostic testing but are unable to use this information to revise a patient's probability of disease. A second limitation is that these data were collected at only one medical school. However, as we find the same pattern of competencies in the recently graduated students who are entering our Internal Medicine program it is likely that our findings apply to many other medical schools. A third limitation is that we had our two groups of subjects critically appraise two different articles although they were very similar from this perspective. A final limitation is that it is possible that our subjects would have demonstrated competency in probability revision if we had provided them with a Bayes nomogram or computer spreadsheet. But we wanted to assess whether our students could apply EBM skills to a clinical encounter without any other external supports except for a simple calculator. In the same way, we do not allow our students to take handbooks or other work aids into their 15 minute OSCE encounters with standardize patients.

The poor performance of our students and residents at probability revision is worrisome although previous studies have shown that many clinicians do not master Bayesian inference. Nearly 25 years ago, Casscells documented that few students or faculty at Harvard Medical School were able to correctly complete a probability revision problem [12]. Eddy duplicated this finding in a second group of physicians [13]. However, some cognitive psychologists suggest that humans have most likely always used Bayesian inference in order to survive in our uncertain world. They argue that it is the probability format of the numbers and not the inference task that makes most people fail at the task [14]. A promising line of research suggests that learners show sustained mastery of Bayesian inference using probabilities if they are taught how to first translate probabilities into natural frequencies [15]. Whether this will prove to be the solution deserves study.

Currently, most of our medical students are able to critically appraise research articles about diagnostic testing but few are able to apply this information at the patient level using Bayesian inference. Because we are able to document the same pattern of skills in entering Internal Medicine residents, we expect this lack of competence with probability revision to be wide spread amongst medical learners. This raises concerns about the clinical utility of the EBM training many students are currently receiving within the domain of diagnostic testing.