Gender plays no role in student ability to perform on computer-based examinations

Delivery of examinations via computer (on-line testing) is becoming more and more prevalent in medical education. Since 1998, students have taken the United States Medical Licensing Examination (USMLE) Step 1, Step 2 and Step 3 on-line. In the very near future, the Medical College Admissions Test (MCAT) will be administered on-line only. The National Board of Medical Educators is now developing on-line Subject Examinations. Current thinking is that on-line testing is the same as paper-and-pencil administration [1–5].

Given the sweeping changes that are now occurring in medical school testing methods, it is important to understand the potential performance differences in students taking paper-and-pencil examinations compared to on-line examinations and to implement on-line examinations so that students have the best opportunity to show their level of proficiency [6, 7].

This study involves first year medical students at the University of Illinois at Urbana-Champaign. During Academic Year 2004–05 the biochemistry and neuroscience courses each began administering one of their major examinations via computer. The purpose of the study was to detect any differences in student achievement between traditional paper-and-pencil and computer-based exams, and furthermore, to detect any gender-based differences in student achievement using these different testing modalities.

We decided include an analysis of gender on our study because of several studies over the past thirteen years suggest that there is a 'technological gender gap' between males and females, with female subjects falling behind their male peers in use of computers [8–13]. In 1992 Canada and Brusca found that females tend to perceive themselves as less equipped to deal with computers. Eleven years later, in 2003, Lee reported results from surveys of college-age students regarding computer use. He found that, "...nearly half of females respondents...rated themselves as someone with 'limited experience' compared to slightly more than one-third of the males (in regard to computer use.)" Interestingly, Fallows found that as students progress in their studies the male/female technology gap appears to widen. Although more recent studies suggest that females have made some gains in computer use, they nevertheless conclude that in many significant ways females lag behind males in their use of computers and specifically Internet use [8].

The authors hypothesized students would perform less well on computer-based exams than on traditional pen-and-paper exams, and that much of this difference would reflect poorer performance by females on computer-based exams.

During Academic Year 2004–2005 178 students took M-1 examinations at the University of Illinois College of Medicine at Urbana-Champaign. The questions consisted primarily of Multiple Choice Questions (MCQ), most examinations were traditional paper-and-pencil tests, with students marking on the examination booklet and ultimately making their answer choice on a bubble scan sheet. For computer-based examinations, students selected their answer choice by clicking a web-page "radio button." It is important to note that during all computer-based examinations, students are given a scratch paper for notes. Also the on-line examinations were designed according to the best practices as outlined in the literature [2, 7].

Table 1 also describes the analyses performed in the study. It is broken into two sections that describe the metric used to analyze data comparing Academic Year performance over years 1, 2 and 3. The section outlines the analysis of student performance within the disciplines (courses) that employ a mix of on-line and paper-and-pencil examinations. The control discipline only employed paper-and-pencil examinations to measure student performance.

After Institutional Review Board approval, the population studied included all students enrolled in biochemistry, medical statistics and neuroscience in the first year medical school curriculum at the University of Illinois College of Medicine at Urbana-Champaign.

Conceived and created by Benjamin Williams, Ph.D. in Academic Year 1999–00, the University of Illinois College of Medicine at Urbana-Champaign has carefully and methodically developed a computer based testing software package originally designed to test images in the histology course for first year medical students. Since 1999 histology students have taken their final examinations on-line using the software developed by Dr. Williams. During the past two years, a significant effort to expand the software has resulted in a flexible, powerful and secure program that completely manages all aspects of assessment in the first year medical curriculum in Urbana. The on-line examinations were designed taking into consideration best practice as outlined in the literature [2, 7].

Further, all students become very well acquainted with the computerized format long before taking examinations through the use of on-line practice examinations and on-line grade reporting. Thus, giving all students, and especially females, a level of comfort and familiarity with the on-line format long before an on-line examination is administered.

During Academic Year 2004–05 faculty in biochemistry and neuroscience elected to have their final examinations administered in the new on-line format. The purpose of this study was to determine if students were placed at a disadvantage when taking on-line examinations in lieu of paper-and-pencil examinations and to determine if gender played a role in performance.

Analysis revealed that no difference in overall performance related to gender was found.

No difference in overall class performance was found, closer examination revealed no difference in performance when comparing males and females, regardless of the test format and regardless of discipline. Both the multi-factor ANOVA and the specific t-Tests, performed within each discipline revealed no performance difference between males and females in either one-tail or two-tail distributions.

Despite overwhelming evidence in the literature regarding gender differences in computer use and attitude toward computer use, females are not placed at a disadvantage when administered on-line examinations at the University of Illinois College of Medicine at Urbana-Champaign. M-1 students do not perform more poorly on on-line examinations compared to traditional paper-and-pencil examinations.

As the formats of course examinations and standardized examinations change from paper-and-pencil to the on-line format, it is imperative that testing design pay close attention to the best practice suggestions outlined in the literature and give students an opportunity to familiarize themselves with the test delivery software.

None