Imbalanced specialty representation of USMLE and NBME test writers

Abstract

Purpose

The United States Medical Licensing Examination (USMLE) is an examination series required for allopathic physician licensure in the United States (US). USMLE content is created and maintained by the National Board of Medical Examinations (NBME). The specialty composition of the USMLE and NBME taskforce members involved in the creation of examination content is currently unknown.

Methods

Using the 2021 USMLE and 2021 NBME Committees and Task Forces documents, we determined each member’s board-certified primary specialty and involvement in test material development committees who we dubbed “test writers”. Total active physicians by primary specialty were recorded from the 2020 Physician Specialty Data Report published by the Association of American Medical Colleges (AAMC). Descriptive statistics and chi-square analysis were used to analyze the cohorts.

Results

The USMLE and NBME test writer primary specialty composition was found to be significantly different compared to the US active physician population (USMLE χ²[32] = 172, p < .001 and NBME χ²[32] = 200, p < .001). Only nineteen specialties were represented within USMLE test writers, with three specialties being proportionally represented. Two specialties were represented within NBME test writers. Obstetrics and Gynecology physicians were proportionally represented in USMLE but not within NBME test writers. Internal Medicine (IM) accounts for the largest percentage of all USMLE test writers (60/197, 30%) with an excess representation of 31 individuals.

Conclusions

There is an imbalance in the specialty representation of USMLE and NBME test writers compared to the US active physician population. These findings may have implications for the unbiased and accurate portrayal of topics in such national examinations; thus, future investigation is warranted.

Introduction

The United States Medical Licensing Examination (USMLE) is an examination series required for allopathic physician licensure in the United States (US) and for international medical graduates wishing to practice in the US [1]. The USMLE series is composed of the Step 1, Step 2 Clinical Knowledge (CK), and Step 3 examinations which are landmark assessments throughout the continuum of medical training of basic science, clinical science, and internship [1]. USMLE content is, in part, maintained by the National Board of Medical Examinations (NBME) [1, 2]. The National Board of Medical Examiners (NBME) also creates the Clinical Science Subject Exam (‘shelf exams’), which are standardized exit examinations of the core clinical clerkship rotations for many US medical students [3].

Given the universality and importance of the USMLE as an assessment in the allopathic medical training coupled with breadth of topics covered on the examination [4], careful evaluation of the members responsible for the examination questions and content is indicated. USMLE test questions are derived from various USMLE test committees which, per the USMLE, represent a “national faculty of medicine” [1]. Likewise, the NBME claims that the test development committees responsible for subject examinations are composed of “health professionals who are experts in the area” [2]. To evaluate these claims, we compared these test writer populations to the US actively practicing physician population.

Methods

The study was exempt by the SUNY Upstate Medical University Institutional Review Board. Listed individuals from USMLE Test Material Development Committees of the 2021 USMLE Committees and Task Forces [5] and two NBME subject examination task forces of “Family Medicine”, and “Obstetrics and Gynecology” from the 2021 NBME Committees and Task Forces [6] were included. A general web search on the individual was conducted for a board-certified specialty, and if one existed, the individual was added to the “USMLE Test Writers” or “NBME Test Writers” cohort, respectively. Individuals already consented to be listed on the directories and furthermore were deidentified during our study. As such, no consent to participate declarations were collected.

In order to conduct a chi-square goodness of fit analysis, a control population was constructed from the 2020 Physician Specialty Data Report [7] published by Association of American Medical Colleges (AAMC), containing the number of total active physicians by predefined 47 unique specialties. To create a control population that is congruent in primary specialty division, and thus have a meaningful comparison to the USMLE and NBME test writers, we made modifications to create our general physician population. We combined all pediatrics subspecialties (“Pediatrics Anesthesiology, “Pediatrics Cardiology”, “Pediatrics Critical Care Medicine”, “Neonatal-Perinatal Medicine”, “Pediatrics Hematology/Oncology”) and counted them all as “pediatrics” specialty. Similarly, “Child and Adolescent Psychiatry” was combined with, and counted under “Psychiatry”. “Pain medicine/Pain Management” and “Sports Medicine” were omitted given unclear alternative primary specialty placement. Because no urologist appeared in the USMLE and NBME taskforce, urologists were likewise removed from the control population. This generated a total of 33 unique primary specialties. This control population was dubbed “Modified Active US Physicians by Primary Specialty” (Supplemental Table 1).

Two separate chi-square goodness of fit tests were conducted to examine whether the observed proportions by specialty for USMLE and NBME test writers differed from expected proportions based on active physician specialties. Analyses were performed in Stata 18 (StataCorp LLC, College Station, TX, US) with two-tailed tests at a 0.05 significance level. Due to small expected frequencies (N < 5) in several cells from the chi-square, we also performed sensitivity analyses through exact multinomial goodness of fit test (Monte Carlo method) using the Stata mgof module [8].

Results

There were 197 USMLE and 25 NBME test writers in 2021. There was no overlap of individuals between these two test writer cohorts. The USMLE test writer specialty composition was significantly different compared to the specialty composition in the US active physician population (χ²[32] = 172, p < .001). The NBME test writer specialty composition was also significantly different compared to that of the US actively practicing physician population (χ²[32] = 200, p < .001). Sensitivity analyses using exact tests for small samples also generated statistically significant differences (p < .001) for the USMLE and NBME test writers, indicating our findings are robust. A summary of US actively practicing physicians by specialty compared to USMLE and NBME test writer representation can be found in Tables 1 and 2, respectively.

Table 1 Primary Specialty representation of USMLE Test writers. Captures the relative representation of the 33 unique specialties among USMLE test writers and modified active US practicing physician population. The difference column is denoted with + for specialties in which an excess of members is present and denoted - for specialties lacking proportional representation

Table 2 Primary Specialty representation of NBME Test writers. Captures the relative representation of the 33 unique specialties among NBME test writers and modified active US practicing physician population. The difference column is denoted with + for specialties in which an excess of members is present and denoted - for specialties lacking proportional representation

Nineteen specialties were represented within USMLE test writers, with three specialties including neurology, physical medicine and rehabilitation, and neurosurgery were found to be proportionally represented. Despite only accounting for 14.8% of actively practicing physicians, Internal Medicine physicians account for the largest percentage of USMLE (29.1%) with a representation with an excess of 31 USMLE test writers.

Only two specialties were represented in NBME test writers within our scope of analysis of shelf examinations: Family Medicine (9/25, 36%) and Obstetrics and Gynecology (16/25, 64%). Family medicine physicians account for 14.8% while Obstetrics and Gynecology physicians account for 5.3% of all actively practicing US physicians.

Discussion

To our knowledge, this is the first study to investigate the primary specialty composition of the USMLE and NBME members involved in the creation of examination content. We found that USMLE and NBME test writers’ primary specialty composition did not reflect the US actively practicing physicians. This suggests certain medical specialties may not be appropriately represented while writing standardized medical examination content. Potential consequences include introducing bias, imbalance, and possible inaccuracies in standardized test questions [9, 10]. When reviewing USMLE examination questions regarding obesity, the Obesity Society of America and American Board of Obesity Medicine cited a lack of obesity experts as a contributing to the detected bias in such questions [9]. Critical care topics accounted for 19% of all sampled USMLE Step 2 questions [10], despite our analysis citing 1.6% of US active practicing physicians in the specialty. Although “critical care” could potentially encompass multiple specialties and subject topics, there could still possibly be an imbalanced content-to-writer ratio.

Our study also raises the question regarding the size of a specialty as a justification for exclusion from aspects of general medical examination creation. For example, cancer is found throughout the USMLE content outline [4]. Oncology treatments are often multidisciplinary, to which Radiation Therapy serves a critical role. A commonly provided reason for the lack of Radiation Oncology (RO) representation in undergraduate medical education and medical licensing examinations is that RO is too small of a field to justify inclusion. Our findings do demonstrate that Radiation Oncologists are a small proportion of active US physicians (0.7%). However, many of other fields of medicine are comparable, compromising < 1% of the active physician population: Geriatric (0.7%), Medicine-Pediatrics (0.7%), Allergy and Immunology (0.6%), Pulmonology (0.6%), Neurological Surgery (0.7%), Rheumatology (0.8%), Vascular Surgery (0.5%), and Thoracic Surgery (0.5%). For comparable-sized specialties of 0.7%, only neurology surgery was equally underrepresented in the USMLE test writers. When investigating the USMLE taskforce committees [5] directly, only one Radiation Oncologist was identified and present on the Step 1 Standard Setting Committee. These findings suggest that Radiation Oncology expert review may be absent from USMLE Step 2CK and Step 3 questions. This raises concern regarding important topics in oncology that need to be covered for all providers and should be further investigated.

One of the core missions of the USMLE “to develop and improve assessment for licensure with intent of assessing physicians more accurately and comprehensively” [1]. Our findings demonstrate a broad imbalance of specialty representation, often regardless of specialty size, which highlights an area for improvement. When NBME was called to integrate nutritional content into Step 1 and Step 2, they explicitly included a nutritional task force composed of multidisciplinary experts in the field when constructing the new examination questions [11]. It would not be unreasonable to reformat current task forces and properly balance expert opinion of tested topics.

It should be noted that NBME subject examinations are more specific to the specialty tested unlike the USMLE Step examination series, which reflects a general medical examination. As such, these NBME subject examinations may not require as rigorous balancing since the tested content may usually reflects a specific specialty. This may explain our findings of only Family Medicine and OBGYN specialty representation on the 2021 NBME Taskforce committee directories. However, there is a previous precedent to include a multidisciplinary panel for creations of tests questions [11]; thus, NBME could have a similar approach to review questions to confirm little to no bias is present in the line of questions.

A limitation of the study is the reliance on the 2021 USMLE and NBME Taskforce and Committees directories [5, 6] as a single reference point. Future studies comparing the taskforce committee compositions longitudinally are needed to determine if this specialty imbalance is a systemic finding. It should also be noted possible error regarding calculations of comparing observed vs. expected test writers given the small counts of individuals within the committees and smaller specialties. Board certification status may also not accurately reflect true specialty practice. For example, if a family medicine or internal medicine physician is boarded in geriatrics but primarily practices outside the field, are they still considered geriatricians? The authors of this paper did not contact the NBME directly on reading assistance on how to accurately assess the number and type of specialties on the various test materials development committees. Likewise, this study did not investigate any official USMLE or NBME examination content, nor the knowledge acquisition of students from test preparation. More studies are necessary to determine if there are biased portrayals of topics within these examinations, particularly in the underrepresented fields of medicine within medical education.

Conclusions

From our analysis, the primary specialty composition of USMLE and NBME test writers is not reflective of that of the active practicing physician population. Our findings demonstrate a broad imbalance of specialty representation, often regardless of specialty size, which highlights an area for improvement. These findings may have implications on the unbiased and accurate portrayal of topics in such national examinations. More studies are necessary to determine if there are biased portrayals of topics within these examinations, particularly in the underrepresented fields of medicine within medical education.