We examined 13 pre- and post-admissions academic performance variables to identify associations with matching into first-choice residency and into highly competitive specialties. Our study has the advantage of a large sample size as well as of examining both institution-specific (undergraduate science and overall GPA, pre-clinical GPA) and standardized (MCAT, COMLEX and USMLE scores) performance factors, with the latter being generalizable and thus potentially applicable to students at other institutions in the United States. Due to differences in selection criteria used for and the process of entering into postgraduate medical education between the United States and other countries such as the United Kingdom and Canada, our findings can only be generalized to other institutions in the United States rather than globally [17, 18].
Student performance parameters associated with matching into first-choice residency
We found that 53.9% of our graduates matched into their first-choice residency over the eight-year period of the study, which is comparable to the 53% of US allopathic seniors for the 2016 match . When examining factors associated with matching into first choice, we identified an association between pre-clinical GPA and matching into first choice. However, the effect size was small, with < 1-point difference (on 100-point GPA scale) between students that matched into their first choice and those that did not. This may be due to the fact that, unlike standardized licensing exams which allow the comparison of applicants from different institutions, grades can have different meanings at different institutions and, therefore, are not an important selection criterion for residency. We observed a similar effect of pre-clinical GPA using multivariate analysis. Specifically, for each 5-point increment in GPA, the odds of matching into first-choice residency increased by 22%. However, when additional performance factors (mean highest MCAT score, COMLEX Level 2 CE score and passing Level 2 PE) were included in the analysis, this effect was no longer statistically significant. This may be because the large effect of passing COMLEX Level 2 PE on matching into first choice (see below) outweighs the impact of pre-clinical GPA. It is also consistent with residency program directors not citing pre-clinical grades as a criterion for selecting applicants to interview and to rank them post-interview .
We also found that passing COMLEX Level 2 PE (a pass/fail exam) on the first attempt is associated with matching into first choice. Furthermore, multivariate analysis demonstrated that passing COMLEX Level 2 PE is the strongest predictor for matching into first-choice residency, increasing the odds of matching by 232% when considered alone and by 268% when accounting for the presence of additional academic performance factors. This is not surprising, since this exam is utilized by program directors to select applicants for interview as well as to rank them post-interview (relative importance 4.2/5 for both) . Additionally, we observed an association of COMLEX Level 2 CE and USMLE Step 2 CK scores, with students that matched into their first choice having higher mean scores (14.1 points for Level 2 CE and 3.9 points for Step 2 CK). A similar effect of COMLEX Level 2 CE score was observed in multivariate analysis, wherein for every 5-point increment in score there was a small effect (1%) of increasing both the unadjusted and adjusted odds of matching into one’s first-choice. Again, this may be expected, since Level 2 CE/Step 2 CK scores are also used by program directors both as interview selection (relative importance 4.1/5) and as post-interview applicant ranking criteria (relative importance 4.1/5) . Surprisingly, we did not identify a similar effect for Level 1 or Step 1 scores. This may be explained by the fact that students select a first-choice residency program (or revise their previous selection) using their Level 1/Step 1 scores as a measure of their competitiveness. Subsequently, they may strive to improve their Level 2 CE/Step 2 CK scores to maximize their chances of a successful match into this program.
Not surprisingly, passing COMLEX Level 2 CE on the first attempt was associated with matching into first-choice residency. Again, this is consistent with “any failed attempts in USMLE/COMLEX” being used as a criterion to select applicants for interview (relative importance 4.6/5) and rank them post-interview (relative importance 4.5/5) . Interestingly, we did not observe a similar effect for passing USMLE Step 2 CK on the first attempt. This may be explained by the fact that passing this exam is not a graduation requirement for our students, rather they opt to take it to increase their competitiveness in the matching process. Furthermore, all AOA-accredited as well as a large number of ACGME-accredited programs use Level 2 CE pass/fail or target scores in applicant selection . Therefore, students that fail Step 2 CK but pass Level 2 CE on their first attempt, can select a first-choice program, and still successfully match solely using their Level 2 CE performance.
Among the pre-admissions academic performance factors tested, we found a statistically significant association of highest MCAT score with matching into first choice. Interestingly, the mean score of students that did not match into their first choice was 0.4 points higher (on a 45-point scale). We observed the same effect in multivariate analysis. Specifically, the odds of matching into one’s first-choice residency decreased for every 5-point increase in mean highest MCAT score, both when considering it alone (19% decrease), and when adjusting for the presence of additional variables (5% decrease). This further confirms the results of our univariate analysis. A previous study has shown that taking the MCAT several times reduces the predictive value of the exam . This, taken together with the fact that the majority of our students have taken the MCAT more than once could account for our observation that the effect is in the opposite direction one would expect.
Student performance parameters associated with matching into highly competitive specialties
Matching into a highly competitive specialty is a different question. It does not solely involve a student’s subjective assessment of competitiveness or fit with the program, or other criteria such as student geographic preferences, which have been shown to play a role in student ranking of different programs [1, 20, 21]. A large body of work has demonstrated that USMLE Step 1 and/or Step 2 CK scores play an important role in matching into competitive specialties [6,7,8,9]. Our findings are consistent with these studies. Specifically, we observed a 62.7-point difference in mean COMLEX Level 1 and a 50.5-point difference in mean Level 2 CE scores of students that matched into specialties of high versus low competitiveness. For USMLE Step 1 and Step 2 CK, we found a 13.6-point and a 6-point difference in mean scores respectively. This difference in mean Level 2 CE and Step 2 CK scores was more pronounced for matching into highly competitive specialties than for matching into first-choice residency. Specifically, it was approximately 3.6-fold larger for Level 2 CE and 1.5-fold larger for Step 2 CK. This difference in magnitude suggests that licensing exam scores are more heavily weighted for matching into highly competitive specialties, than for matching into one’s first choice.
The importance of COMLEX Level 1 and Level 2 CE scores was further confirmed by multivariate analysis. When considered alone, Level 1 and 2 CE scores were associated with an increase in the odds of matching into highly competitive specialties (respectively 4 and 3% per 5-point increment in score). Upon adjusting for the presence of additional variables, COMLEX Level 1 score remained significantly associated, with a 4% increase in the odds of matching per 5-point increment in score. Although this may not appear to be a large effect, given that Level 1 is scored on a 990-point scale, as little as a 30-point increase will result in 25% higher odds of matching into a highly competitive specialty. The fact that COMLEX Level 1 score is the single most important predictor of matching into highly competitive specialties is supported by several lines of evidence. First, it is the only licensing exam score that students are required to submit with their initial residency applications, and since our institutional deadline for taking the exam is September 1st of the fourth year of studies, scores are not available by the September 15th deadline for submission of initial residency applications. Additionally, since, based on our experience, students typically submit their Level 2 CE score by November or in December (in the middle or towards the end of interview season), Level 1 is frequently the only score available to program directors when they select applicants to interview and during the interview process. Accordingly, COMLEX Level 1 is almost universally used by program directors to select candidates to interview (93% cite it as an important factor) and remains one of the most commonly used criteria to rank candidates post-interview (78% of program directors cite it as important) . If candidates’ Level 2 CE scores are available during the interview selection process they are also used, albeit not as frequently as Level 1 scores (83% of program directors cite Level 2 CE scores as an important factor) . Finally, since COMLEX Level 2 CE scores are required by programs to rank candidates they have already interviewed (most programs have a December deadline for submission of Level 2 CE scores which is towards the end of interview season), they play a secondary role in the matching process, with Level 1 scores being the most important determinant. Taken together, these observations also suggest an explanation as to why we did not observe a statistically significant association of Level 2 CE scores with the adjusted odds of matching into highly competitive specialties.
Furthermore, passing licensing exams on the first attempt was a critical factor for matching into highly competitive specialties. We found that 0% of students that failed COMLEX Level 1 and only 4.4% of those that failed Level 2 CE matched into highly competitive specialties. In comparison, 12.6% of students who passed Level 1 and 12.4% of those who passed Level 2 CE on the first attempt matched into specialties of high competitiveness. Similarly, for USMLE, only 1.4% of students that failed Step 1 and 4.8% of those that failed Step 2 on the first attempt matched into highly competitive specialties, whereas 15.7 and 15.3% of those that passed Step 1 and Step 2 CK on the first attempt respectively, did so. These findings are consistent with the 2016 NRMP Program Director survey where (on average) 96% of program directors in highly competitive specialties indicated that they would never/seldom consider applicants that failed USMLE Step 1 or Step 2 CK on the first attempt. In addition, 80.5 and 88.5% (on average) of program directors in low-competitiveness specialties stated they would never/seldom consider applicants that failed Step 1 or Step 2 CK respectively .
In contrast to our findings regarding matching into first-choice residency, we did not identify a statistically significant association of passing COMLEX Level 2 PE (a pass/fail exam) on the first attempt with matching into highly competitive specialties. A likely reason for this observation is that Level 2 PE is considered as a minimum competency exam by program directors of highly competitive specialties.
In addition to licensing exams, we identified a statistically significant association of pre-clinical GPA with matching into highly competitive specialties. However, the effect size was small with a 2.4-point difference between the mean GPA of students that matched into specialties of high versus low competitiveness. The fact that the effect size of pre-clinical GPA was small compared to that of licensing exam scores suggests that, since grades are not standardized among different medical schools, they are not as useful in comparing applicants to residency programs. In contrast, scores on standardized licensing exams allow program directors to compare diverse applicants from across the country. The results of multivariate analysis further support this, since, although pre-clinical GPA affected the unadjusted odds of matching into highly competitive specialties (66% increase per 5-point increment in GPA), there was no statistically significant association with the adjusted odds, indicating that pre-clinical GPA is not a predictor of matching, particularly when considered in context with COMLEX Level 1 score.
Finally, using univariate analysis, we did not observe an association between any pre-admissions academic performance variables, such as highest MCAT score or science GPA, with matching into highly competitive specialties. This is not surprising, since, although undergraduate GPA and MCAT scores allow for successful admission to medical school, they are not part of residency applications and thus are not available to program directors. When highest mean MCAT score was included in the multivariate analysis, it was associated with a 19% increase in the unadjusted odds of matching into highly competitive specialties per 5-point increment in score. However, when adjusting for the presence of additional performance factors (COMLEX Level 1, Level 2 CE and pre-clinical GPA), this effect was no longer statistically significant. A large body of work has demonstrated a small to medium predictive validity of MCAT scores for licensing exam performance, particularly for USMLE Step 1 [22, 23], and since USMLE Step 1 is a key determinant for matching into highly competitive specialties (as is COMLEX Level 1 according to the present study) [2, 7, 9, 24], this may account for the observed association of MCAT score with the increased unadjusted odds of matching into highly competitive specialties. However, since the effect of MCAT score on matching is indirect via its effect on Step 1/Level 1 scores, it is no longer statistically significant when considered in the context of COMLEX Level 1, which we found to be the single strongest predictor of matching into highly competitive specialties.
Limitations of the study
This study has a number of limitations. First, the relative competitiveness of specialties may slightly change from year to year, in part due to the variability in the number of applicants and positions available . Our study population consisted of students that matched into residency from 2010 to 2017, however, we used the 2017 Careers In Medicine® competitiveness ratings.. To estimate the effect of this limitation, publicly available match data from the National Resident Matching Program (NRMP) annual residency match reports 2010–2017, as well as Electronic Resident Application Service (ERAS) data, that provide detailed information for each specialty on the number of available positions, number of applicants, number of positions filled and number of positions filled by US seniors, was used to determine the specialty fill rate by US seniors, which is a measure of its competitiveness . This analysis revealed small year-to-year fluctuations in fill rate for the specialties included in this study during the eight-year study period. However, these fluctuations were not sufficient to result in any changes in competitiveness levels for any of these specialties during the study period.
A second limitation is that our students can apply for, and thus, match into either AOA- or ACGME-accredited residency programs, which may differ in competitiveness within the same specialty. However, this difference is not sufficient to cause a given specialty to move from one competitiveness category to another depending on whether the program is AOA- or ACGME-accredited. Since licensing exam scores are a key determinant in matching , if ACGME-accredited programs are more competitive than AOA programs within the same specialty, one would expect licensing exam scores of students matching into the former to be higher than those matching into the latter. This will result in an increase in mean exam scores than if it were the other way around. Since approximately the same percentage of students matched into ACGME versus AOA-accredited programs in the high and medium competitiveness categories (high: 53.8% ACGME, 44.2% AOA; medium: 51.9% ACGME, 42.5% AOA), mean licensing exam scores in each group will increase by the same extent, and the magnitude of the statistically significant differences we observed between these two groups is not affected by whether students matched into ACGME or AOA programs. In the low competitiveness category, a higher percentage of students matched into ACGME-accredited programs (62% ACGME, 31.9% AOA). Therefore, mean licensing exam scores in this group will be higher and this will attenuate the magnitude of the difference we observed between the high and low competitiveness, as well as between the medium and low competitiveness groups.
Third, whether students matched into their first-choice residency program is self-reported and has not been independently validated. However, students have no incentive or reward to falsify this data. The Electronic Residency Application System (ERAS) which holds the information on first, second or third choice match, does not officially release this information, thus all medical schools in the United States must rely on self-reported data . Additionally, in the present study, data on the specific specialty and residency program that students matched into was also self-reported. However, this was also officially validated by the Office of the Registrar at our institution, and we did not identify any inconsistencies between the self-reported and Registrar data. Thus, there is also no reason to doubt the validity of the self-reported data on first-choice match.
An additional limitation consists of the fact that students may revise their original first-choice residency selection using their COMLEX Level 1 or USMLE Step 1 score as a measure of their competitiveness, in order to maximize their chances of a successful match. This would result in an increased number of students matching into their first-choice program. However, this would not be unique to our institution, rather it would likely occur in osteopathic and allopathic institutions throughout the U.S., particularly in light of the fact that students are constantly advised of the importance of Level 1/Step 1 scores for matching into residency.
The fact that there is significant heterogeneity in the pre-clinical grading systems currently employed in U.S. medical schools (pass/fail, honors/pass/fail, numeric grades among others) limits the applicability of our findings with regards to the effect of pre-clinical GPA on matching into first-choice residency and competitive specialties. However, since there is no national standard for grading systems, this limitation applies to any study examining pre-clinical GPA and is not exclusive to our study.
Since this study focuses exclusively on students from one institution, the effect of medical school quality or competitiveness on matching into residency could not be assessed. A number of studies have shown that medical school quality (as measured by the widely accepted “U.S. News and World Report” rankings which, among other factors, capture the acceptance rate or competitiveness of a medical school) plays a role in matching into certain highly competitive specialties such as ophthalmology, otolaryngology and orthopedic surgery [7,8,9, 28]. Additionally, “graduate of a highly-regarded U.S. medical school” is used as a criterion by program directors for selecting applicants to interview as well as for ranking applicants that have already interviewed . Furthermore, less competitive medical schools tend to attract and admit applicants with lower MCAT scores. Since MCAT scores have been shown to possess a small to moderate predictive validity for licensing exam scores, particularly for USMLE Step 1, students in less competitive medical schools are more likely to score lower on their licensing exams, which will in turn decrease their likelihood of matching into highly competitive specialties [22, 23]. The most current “U.S. News and World Report” rankings place our school in the bottom 25% of the 120 ranked allopathic and osteopathic medical schools, therefore, given all of the above, one would expect fewer of our students to match into highly competitive specialties. In fact, during the eight-year study period, only 12% of our graduates did so. However, the fact that our school is in the bottom 25% of ranked schools does not affect the validity of our findings. A study from the University of Minnesota Medical School, which is a top 50 ranked institution, identified an association of licensing exam scores and residency specialty match, with students matching into specialties such as dermatology (highly competitive) having the highest mean scores and students matching into family medicine (low competitiveness) having the lowest mean scores, which is consistent with our findings .
Finally, we considered a large and comprehensive number of academic variables (which represent “hard skills”), however we did not consider any non-academic factors (“soft skills”), such as interpersonal skills and performance on the interview, which are much more difficult to quantitate but which play an important role in programs ranking applicants and can thus affect matching .