Case-based exercises fail to improve medical students' information management skills: a controlled trial

When today's medical students graduate, they will conduct patient encounters using multiple technology-enhanced decision support systems. Current medical student training in ambulatory settings may not prepare students for this type of practice. Students often learn from physicians who generate few (0.01 to 0.8) clinical questions per patient encounter and infrequently use information technology to answer clinical questions at the point of care [1–3] Clinicians have previously reported that answering clinical questions is too time consuming to be practical during clinical sessions[4, 5] However, that situation is changing. Improved information management tools, including personal digital assistants (PDAs) and Internet capable wireless computers, now allow rapid access to Web-based clinical information in ambulatory settings.

Medical students must develop information management skills as a routine, integral part of the ambulatory patient encounter. Information management skills include: asking and refining clinical questions; accessing, retrieving, integrating and applying information into a clinical situation; and managing the doctor/patient/technology interface. Managing information at the point of care requires different skills than traditional evidence-based medicine (EBM) as taught in most medical schools [6] In traditional EBM, the learner develops a clinical question, performs a literature search, selects and appraises an appropriate research study, and draws conclusions. The EBM process usually occurs remotely from the patient encounter and requires time, the ability to understand the source literature, and critical appraisal skills.

While EBM skills are important for medical learners, many clinical questions can be answered at the point of care without critical appraisal. For example, using an electronic drug database in a PDA to investigate potential drug interactions is information access and retrieval. Modifying the patient's medications to adjust for interactions is information integration and application. The learner accesses an information source he/she deems reliable, and finds the answer in approximately 20 seconds during the office visit.

Few studies have investigated students' ability to manage information in clinical settings. In a study by Bergus and colleagues, fourth year medical students evaluated a standardized patient (SP), then read an article about a diagnostic test relevant to the patient's presentation. Most students appraised the article correctly, but few could apply the information to the individual patient[7] In contrast, Webershock and colleagues demonstrated that, following an EBM seminar, third-year students could both appraise an article and integrate that information into a paper case[8] Davidson and colleagues conducted a more complex skills assessment by having SPs ask a question of third year medical students. Students then formulated a clinical question, performed a Medline search, selected and appraised a journal article and transmitted results to the patient. Students did well in this applied EBM exercise, averaging 3.7–4.0 on a scale of 1 (poor) to 5 (superior) for each task as evaluated by faculty and librarians[9] In summary, after a non-clinical EBM course, graduating students demonstrated EBM knowledge but had difficulty applying EBM information in the clinical setting. However, third year students applied EBM skills adequately to a paper case and standardized patient encounter given structured directions and sufficient time

A 2005 literature search yielded no previous studies investigating students' abilities to access, retrieve, integrate and apply information in real time patient encounters. In this study, we investigated whether case-based discussions with information management exercises improved students' information management skills as evaluated on a standardized patient case. We compared (1) intervention and control groups on two different clerkships and (2) first and third groups from both clerkships in academic year 2005–6.

Fifty-five students (26 intervention and 29 control) completed the standardized patient case and 42 (19 intervention and 23 control) consented to have their information included in the analysis (76%). There was a difference between consenters and non-consenters on total scores. Students who consented scored lower than those who did not. (14.0, 15.5, p = 0.05)

Percentages of consenting students completing each checklist item are shown in Table 2. Ninety percent of students accessed the Internet and 62% used a PDA during the encounter. Ninety-eight percent of students offered medication for malaria prophylaxis. Forty-eight percent asked if the patient would be working with animals, but only 10% inquired if she would be providing health care. Although the SP questioned every student about medication interactions, only 38% confirmed a potential interaction between doxycycline and antacids and of these 17 students, 71% (12) discussed how to take the medication to avoid the interaction.

There was no difference in total scores between intervention and control group students (13.4, 14.4, p = 0.12). However, students in both the intervention and control groups on the third rotation scored significantly higher than first rotation students. (13.6, 15, p = 0.038). However, when we examined the 13 technology items, there were no differences between intervention and control group scores (7.5, 7.9, p = 0.25) or between first and third rotation group scores (7.3, 8.3, p = 0.08).

Intervention group students who received case-based training in information management scored no higher on the SP case than students who received no such training. Possible explanations include that the intervention did not adequately teach information management skills or that the SP case did not accurately evaluate those skills. Also, students may have failed to apply/transfer the information management skills from the case-base discussions to the SP experience because the learning context changed from paper cases to a mock clinical encounter. In addition, students do not learn the clinical content (travel medicine) during any third year rotation, so we were testing on a content area that we had not formally taught. However, we deliberately chose this content because we considered that the lack of formal teaching 'leveled the playing field' for finding travel information that is easily retrieved from the Internet. Third rotation students performed better than first rotation students on the total score, but not on the technology sub-score. Third rotation students' better performance may be explained by their increased clinical experience, or by the fact that they learned the case expectations from students in the previous groups. All groups' mean scores were lower than the usual average score of 75% seen across multiple validated SP cases in our setting.

All but one student accessed a reputable site (e.g. the Center for Disease Control) and retrieved travel information for Botswana, and most integrated and applied information pertaining to malaria prophylaxis (but not health care delivery or working with animals). Thus, if we group information management skills into two levels – Level 1 (access and retrieval) and Level 2 (integration and application), our students did demonstrate some skills on both levels. However, their performance on Level 2 skills (integration and application) was generally poor. Even in our best-performing group only 67% consulted their PDAs to investigate a potential drug interaction. Students' performance may have been adversely affected by the time pressure of a 20-minute encounter, although we consider this unlikely. On our SP case feedback form, we routinely question students on whether or not they had adequate time for the case. For most cases, 2–3 students per group report having insufficient time. For the travel case, no student reported having insufficient time.

Even when questioned directly about medication interactions, only 38% of all students consulted their PDAs to investigate a possible interaction. Most students wrongly informed the standardized patient that there was no potential interaction between doxycycline and antacids without consulting their PDA, a task that requires minimal time (20 seconds or less). Two plausible explanations for this lack of PDA use are that students did not know how to use the drug database program or they incorrectly assumed that they knew the answer. In our institution, all third year students are given PDAs and receive formal training on logging patient encounters, but they receive no formal training on using medical programs such as Epocrates^©. Our experience with this highly Internet savvy generation is that most students quickly master the capabilities of their PDAs, but we have cannot report any empiric data on the numbers of students who could not operate the drug database versus those who chose not to consult it.

Our study has several limitations. We conducted it with a small number of learners, at a single institution, using a single SP case for evaluation. Global student competence cannot be reliably assessed by a single standardized patient case. However, in this pilot study, we were not evaluating individual students' global performance. Rather, we were measuring group performance on one particular skill set. We accounted for the suspected variation among student performance and the study had sufficient power to determine if one group of students performed better than another.

Managing clinical information at the point of care is an important skill for future physicians, yet the literature is sparse on teaching information management in ambulatory settings. Unfortunately, our curricular intervention did not impact students' information management performance as measured by an SP case. Student performance did improve with increasing clinical experience. One important finding was that few students accessed an available PDA database to investigate potential drug interactions even when directly prompted by the SP. More concerning was the finding that most students indicated incorrectly that there was no potential interaction. Wrongly answering patients' questions without using readily available information systems is an important and preventable source of medical errors. Medical educators must teach learners to develop information management habits that reduce medical errors, including prescribing errors. At our institution, we will re-evaluate our teaching methods, with the specific aim of teaching students to check for medication interactions at every encounter. Future studies must investigate other educational interventions that demonstrate improvements in students' information management skills.