The setting
The data were collected during a performance-based assessment utilizing standardized patients (SPs). During this assessment, subjects had a series of 15 minute encounters with SPs followed by a 10 minute post-encounter activity, thus making each station 25 minutes in length. A 25 minute non-SP based station was integrated into this assessment during which subjects were asked to read and appraise an article about a diagnostic test and apply the information to a preceding SP encounter.
The subjects
Two different groups of subjects participated in this research. The first group was composed of medical students who were assigned to the Psychiatry clerkship in the late winter and spring of 2003. The second group was composed of all incoming first-year Internal Medicine residents who were in the process of orienting to the residency in June 2003 in preparation for their clinical duties.
The task
Subjects were asked to critically appraise a research study about a diagnostic test using a worksheet derived from the article about diagnostic tests published in Users' Guides to EBM series [5]. They were asked to assess the validity of the study by identifying the reference standard, whether there was independent and blinded comparison with the reference standard, whether the results of the test being evaluated influenced the decision to perform the reference standard, and whether all subjects underwent the reference standard as well as the test being evaluated. Subjects were also asked to evaluate whether the setting of the study was similar to a community setting in which they would anticipate using the new diagnostic test. Lastly, the subjects were asked to identify the results of the study in terms of the sensitivity and specificity of the test. Thus, in critically appraising the study, subjects were asked to answer a series of 6 questions derived from the published diagnostic test EBM user's guide. After they had assessed the study, the subjects were asked to apply the results of the study by revising the probability of disease given a specified pre-test probability and a test result. To assist with this calculation subjects were provided with calculators.
The articles
The fourth year medical students, who were participating in their Psychiatry clerkship, were asked to assess an article about a questionnaire to aid in the diagnosis of Major Depression and Panic Disorder [6]. The Internal Medicine residents read an article about a blood test to aid in the diagnosis of congestive heart failure [7]. While the articles focused on different diseases and diagnostic tests, the studies were similar from the critical appraisal perspective. Both articles described research on a new diagnostic test that had been undertaken in carefully selected clinical settings to avoid spectrum bias. Clinical experts who were blinded to the result of the test being evaluated were used as gold standards by both studies. The protocols used by both projects avoided referral bias. Lastly, both articles had been recently published by major medical journals.
Analysis
For this study, the calculated posttest probability was considered correct if it was ±5% of the correct answer derived from Bayes' Theorem. The performances of the medical students and the Internal Medicine residents were compared using Fisher's Exact Test for dichotomous outcomes and t-test for continuous outcomes. Cronbach's alpha was used to calculate the internal reliability of the 6-item critical appraisal worksheet for each group of subjects. The analyses were undertaken using NCSS 2004 (Kayesville, UT). An alpha of 0.05 was used and all tests were 2-tailed. Approval from the institutional review board was obtained for this project.