Importance of residency applicant factors based on specialty and demographics: a national survey of program directors

Abstract

Background

With the transition away from traditional numerical grades/scores, residency applicant factors such as service, research, leadership, and extra-curricular activities may become more critical in the application process.

Objective

To assess the importance of residency application factors reported by program directors (PDs), stratified by director demographics and specialty.

Method

A questionnaire was electronically distributed to 4241 residency PDs in 23 specialties during spring 2022 and included questions on PD demographics and 22 residency applicant factors, including demographics, academic history, research involvement, and extracurricular activities. Responses were measured using a Likert scale for importance. Descriptive statistics and Chi-square and Fisher exact test analysis were performed.

Results

767 questionnaires were completed (19% response rate). Across all specialties, the factor considered most important was the interview (99.5%). When stratified by specialty, surgical PDs were more likely to characterize class rank, letters of recommendation, research, presenting scholarly work, and involvement in collegiate sports as extremely important/very important (all p < 0.0001). In contrast, primary care PDs favored the proximity of the candidate’s hometown (p = 0.0002) and community service (p = 0.03). Mean importance of applicant factors also differed by PD age, gender, and ethnicity.

Conclusion

We have identified several residency application factors considered important by PDs, stratified by their specialty, demographics, and previous experiences. With the transition away from numerical grades/scores, medical students should be aware of the factors PDs consider important based on their chosen specialty. Our analysis may assist medical students in understanding the application and match process across various specialties.

Introduction

Several changes in medical school performance assessments have occurred over the past several years. The United States Medical Licensing Exam (USMLE) Step 1 transitioned from a numeric score to a pass/fail outcome on January 26, 2022 [1]. The score on this exam was previously considered one of the most important factors for choosing which residency applicants to interview, especially in applicants applying to competitive specialties [2]. Many medical school curriculums are graded as pass/fail during the preclinical years; thus, preclinical grades cannot be uniformly used to assess applicants. During the COVID-19 pandemic, numerous medical schools also changed their clinical year grading systems to pass/fail [3]. Given limited quantifiable data, these recent pass/fail reporting changes raise questions about how residency applicants should be objectively assessed.

The National Resident Matching Program (NRMP) Program Director Survey provides a wealth of information regarding factors that program directors value in selecting applicants to interview and ranking applicants in the Match. This survey is sent to residency program directors (PDs) of 23 specialties biennially, with the most recent report published in 2022 [4]. In 2021 and 2022, several items were deleted from the NRMP program director survey to allow space for more questions about holistic review and the virtual residency application experience during the pandemic, while decreasing respondents’ burden. Deleted items included questions about factors considered in decisions about which applicants to interview and rank and ratings of the importance of each factor. This survey compared the frequency with which programs interviewed and ranked specific applicant groups, including US MD Senior, US DO Senior, US MD Graduate, US DO Graduate, US IMG, and Non-US IMG. The survey also delved into the elements encompassed within the programs’ holistic review process and their significance. These factors include test scores, personal attributes, interests, interpersonal skills, ethics and professionalism, personal experiences, and geographic preferences.

While the NRMP survey results provide data regarding the mean importance of these holistic review factors, each of these factors is quite broad and does not characterize what specific attributes, interests, and experiences are of interest to program directors. In 2022, the NRMP stated that future iterations of the program director survey will re-introduce the questions about factors considered in decisions about which applicants to interview and rank. However, we felt it prudent to assess current opinions regarding which applicant factors are most important to program directors as programs continue to rely on a holistic review of applications due to decreasing objective data such as Step 1 scores.

Recent studies have surveyed program directors after the change to a pass/fail Step 1 score. One study found that approximately 40% of program directors replied that meaningful research participation would become more critical when choosing whom to offer interviews and that competitiveness of the specialty correlates with the reported importance of research [5]. Another study found that compared to non-procedural specialties, PDs of procedural specialties stated that they would place more emphasis on USMLE examinations such as Step 2 after transitioning to a pass/fail Step 1 [6]. While these studies, along with the NRMP data, provide substantial information regarding applicant factors that program directors emphasize when choosing whom to interview, the objective of our study was to analyze additional factors that may be emphasized in a holistic approach to applicant evaluation, such as athletic experience, military experience, medical scribe experience, and previous career prior to medicine. We evaluated the influence of athletic history on residency ranking given reported findings from other studies suggesting that prior participation in athletics may predict success in medical school and residency [7,8,9]. Medical scribe experience has also been postulated to be associated with academic success in medical school [10]. Military experience is associated with an increased likelihood of selecting a candidate for medical school interviews [11]. Secondarily, we assessed the importance of residency application factors stratified by PD demographics and specialty.

Methods

Following Institutional Review Board approval at Penn State College of Medicine, a questionnaire was electronically distributed to 4241 residency PDs within 23 specialties during Spring 2022. PD contact information was collected for every specialty through the FREIDA (Fellowship and Residency Electronic Interactive Database Access) website. All PDs provided informed consent when they agreed to participate in the survey. The survey included seven questions on PD demographics and a list of 22 residency applicant factors with an associated Likert scale rating system where program directors would rank the importance of each factor, as shown in Appendix 1. This survey was developed by the authors, with no collection of validity evidence. Three reminder emails were sent at one-week intervals. Responses were stratified by specialty [surgical (general surgery, neurological surgery, obstetrics and gynecology, orthopaedic surgery, urology, otolaryngology, plastic surgery, and vascular surgery), primary care (family medicine, internal medicine, pediatrics, medicine-pediatrics), all other (anesthesiology, child neurology, dermatology, diagnostic radiology/nuclear medicine, emergency medicine, interventional radiology, neurology, pathology, physical medicine and rehabilitation, radiation oncology, and psychiatry)] and demographics of the PD (age, sex, ethnicity, previous extra-curricular experiences). Extremes on each end of the Likert scale were combined for analyses as the differences between “not at all important” and “slightly important” are unlikely to affect decision-making processes significantly. The same applies to the difference between “very important” and “extremely important.”

Percentages and 95% Confidence Intervals were calculated for categorical variables. Differences between groups for categorical variables were characterized using contingency table analysis; significance levels were determined by Pearson’s chi-square statistic and Fisher’s Exact Test. P-values less than 0.05 were considered significant. The statistical analysis was performed using SAS software, version 9.4.

Results

Analysis was performed on 767 completed questionnaires (19% usable response rate). Most respondents were male (55.7%) and Caucasian (78.6%). 38.0% of respondents were between ages 41 and 50. (Table 1). Specialties of respondents included 189 surgical (25%), 253 primary care (33%), and 323 other (42%) (Fig. 1). The survey template is shown in Appendix 1. Across all specialties, the percentage of respondents who characterized the following factors as extremely important/very important were: interview (99.5%), passing USMLE examinations (88.2%), core clerkship grades (79.1%), demonstrating leadership (70%), letters of recommendation (69.4%), personal statement (64.2%), dean’s letter (49.4%), community service (40%), class rank (29.8%), specialty-specific research (19.3%), close proximity of candidate’s hometown (15.7%), research publications (15.1%), presenting their research at a scientific assembly (10.4%), ethnicity of candidate (9%), non-specialty-specific research (7.2%), previous involvement in collegiate sports (6.9%), previous career prior to medicine (5.5%), previous military experience (4.8%), previous involvement in global health (4.2%), sex of candidate (1.9%), previous involvement as a scribe (1.6%), and age of the applicant (1.4%) (Table 2). P-values less than 0.05 are bolded in Tables 2, 3, and 4 to distinguish significant findings.

Table 1 Demographics of responders (Percentage, 95% Confidence Interval)

Fig. 1

Respondents by specialty type (Primary Care, Surgical, Other)

Table 2 Importance of applicant factors based on all responders and then stratified by specialty (Percentage, 95% Confidence Interval)

When stratified by specialty, surgical PDs were more likely to characterize the following factors as extremely important/very important: class rank (p < 0.0001), letters of recommendation (p < 0.0001), specialty-specific research (p < 0.0001), non-specialty-specific research (p < 0.0001), presenting their research at a scientific assembly (p < 0.0001), and previous involvement in collegiate sports (p < 0.0001). Primary care PDs were more likely to characterize the following factors as extremely important/very important: close proximity to the candidate’s hometown (p = 0.0002) and community service (p = 0.03) (Table 2). Data regarding the importance of residency applicant factors stratified by each specialty grouped within “all others” is displayed in the Supplemental File without p-values due to small sample sizes.

When stratified by demographics, PDs ≤ 50 years old were more likely than PDs ≥ 50 years old to characterize ethnicity (p = 0.04) and letters of recommendation (p = 0.03) as extremely important/very important. Male PDs were more likely to characterize class rank (p = 0.0007) as extremely important/very important. In contrast, female PDs were more likely to characterize ethnicity (p = 0.008) and community service (p = 0.005) as extremely important/very important. Non-Caucasian PDs were more likely to characterize proximity of hometown (p = 0.007), dean’s letter (p = 0.0003), letters of recommendation (p = 0.0003), involvement in global health (p = 0.001), and specialty-specific research (p = 0.0006) as extremely important/very important. PDs with previous collegiate sports experience were more likely to characterize leadership (p = 0.004) and previous involvement in collegiate athletics (p = 0.003) as extremely important/very important compared with PDs without athletics experience. PDs with prior global health experience were more likely to characterize previous involvement in global health (p = 0.009) and community service involvement (p = 0.002) as extremely important/very important compared to PDs without global health experience (Tables 3 and 4).

Table 3 Importance of applicant factors based on age, sex, and ethnicity of the responders (Percentage, 95% Confidence Interval)

Table 4 Importance of applicant factors based on responders’ previous experience with the military, collegiate varsity sports, and global health (Percentage, 95% Confidence Interval)

Discussion

This survey analysis discovered numerous specific residency application factors considered important by PDs, stratified by their specialty, demographics, and previous experience.

Although several survey questions were similar to questions from the 2022 NRMP Program Director survey, our survey explores several applicant factors and their mean importance ratings that were asked in the 2020 NRMP Program Director Survey but not included in the 2022 NRMP Program Director Survey after Step 1 transitioned to pass/fail scoring. First, our survey uniquely stratifies responses by respondent demographics, allowing us to investigate how program director characteristics influence residency applicant ranking. Second, our survey uniquely expands on several residency applicant factors and supplements the 2022 NRMP survey as these aforementioned factors were removed to decrease respondent burden and focus on virtual aspects of residency applications during the pandemic, such as shifting to virtual recruitment and interviewing. Third, our survey focused on the importance of specific applicant experiences in residency ranking, including military experience, involvement in collegiate sports, experience as a scribe, and involvement in global health. Finally, our survey expanded on the importance of applicant demographics in residency ranking including age of candidate, sex of candidate, ethnicity of candidate, and proximity of candidate’s hometown to your program.

Several factors from the 2020 NRMP Program Director Survey overlapped with factors included in our survey. Across all specialties, factors in the NRMP survey were assessed for their importance in ranking applicants. Overlapping factors from the 2020 survey included several components of the interview, such as interpersonal skills (percent citing factor, average rating on Likert Scale with 1 being not at all important, 5 being very important) (95%, 4.8), interactions with faculty (89%, 4.8), and interactions with housestaff (89%, 4.8), letters of recommendation (70%, 4.1), leadership qualities (60%, 4.3), Dean’s Letter (58%, 4.0), personal statement (54%, 3.8), other life experience (47%, 4.0), any failed attempt in USMLE (46%, 4.4), grades in required clerkships (42%, 4.0), volunteer/extracurricular experiences (35%, 3.9) and demonstrated involvement and interest in research (28%, 3.8) [2]. Our results revealed similar emphasis on the importance of the interview, letters of recommendation, leadership qualities, Dean’ Letter, and the personal statement. Notably, in our survey conducted after the transition of Step1 scoring to pass/fail, 88.2% of respondents characterized passing USMLE examinations as extremely important/very important compared to 46% of respondents from the 2020 NRMP Program Director survey who cited any failed attempt in USMLE as an important factor in ranking applicants with an average rating of 4.4 on a Likert Scale of 1–5. This suggests that the importance of passing Step 1 on the first attempt has increased in importance since the transition to pass/fail scoring.

Our results also complement the 2022 NRMP Program Director survey. Program directors cited the following broad categories as important in holistic review of applicants: applicant personal attributes (88%, 4.4) (percent citing factor, average rating on Likert Scale with 1 being not at all important, 5 being very important), applicant interests (85%, 4.1), applicant interpersonal skills, ethics, and professionalism (81%, 4.6), and applicant personal experiences (81%, 3.9) [4].

Understanding our survey results may allow medical students to be better equipped in their decision-making regarding prospective residency programs in the medical specialty of their choice, knowing both the program director’s demographics and the subspecialty to which they are applying.

The decision to make the USMLE Step 1 exam pass/fail will have implications for both medical students and residency programs during the application process [6]. With this transition, in addition to many schools adopting the pass/fail curriculum for the didactic and clerkship years, students may find it harder to determine their competitiveness in a given specialty when relying solely on subjective feedback [12, 13]. In addition, without a 3-digit-score cutoff or student decision-making not to apply to specific programs due to a particular 3-digit score, the increasing number of applications each program receives will directly conflict with the holistic application screening because of the burden placed on educators to review many more applications. This may lead to using other objective measures, such as the USMLE Step 2, to become the screening tool of choice [14].

Given the limited amount of objective data, such as USMLE test scores on residency applications, it is essential for students to understand a residency program director’s perspective in the residency application process due to this consequential change. Unsurprisingly, program directors across all specialties emphasized application components, including interviews, passing USMLE examinations, core clerkship grades, demonstrating leadership, and letters of recommendation.

Our results indicate that PDs for surgical specialties will emphasize class rank, letters of recommendation, research, and previous involvement in collegiate sports. This is consistent with other studies analyzing factors that have an implication in the residency rank process [6, 15,16,17]. Thus, prospective candidates may glean insight into trends in applicants at specific programs by understanding the demographics of the program director of that residency program. The differences in applicant preference based on the demographics of PDs, including their age, sex, ethnicity, and athletic history, suggest implicit biases may influence the decision-making process for residency applicants. However, we feel that these data should not be used to guide students towards or away from certain programs based solely on PD demographics, but rather students should be aware that both implicit/explicit bias may affect applicant ranking during the residency application process. Additional studies are warranted to elucidate this potential relationship further.

There were several limitations to this study. First, we obtained program directors’ emails via FREIDA and found that many emails for program directors were not current, as we received undeliverable emails and responses that some PDs had left their positions. Another limitation is that our survey focused on USMLE scores rather than Comprehensive Osteopathic Medical Licensing Examination (COMLEX) scores, which osteopathic medical schools use. Future studies may include more emphasis on the role that COMLEX scores will play in residency applicant ranking.

There are several additional survey questions that we did not include but would supplement the findings of this study. For example, collecting demographic information about the size and rank/competitive nature of programs would help applicants better understand how program size and reputation affect the perceived importance of applicant factors. Our survey asked about the importance of factors on a Likert scale of 1–5. However, future iterations of surveys could include assessing if the presence or absence of factors, such as failure of a Step 1 exam, are critical in ranking applicants (i.e., a program may not rank an applicant with any USMLE failures). Another limitation is the omission of a free response text box for PDs to include additional comments of important aspects of candidate selection that were not included in our survey. We suggest a free response section be included in future PD surveys.

Our response rate of 19% is comparable to the 18% response rate for the 2020 NRMP Program Director survey, but lower than the 33.1% response rate for the 2022 NRMP Program Director survey. Although the data and generalizability of the results may be limited by the low response rate and sampling bias of those who choose to participate in the survey, we hope that our analysis of the importance of residency application factors assists medical students in understanding the application and match process across various specialties. Furthermore, we hope our data provides insight into possible implicit biases across specialties and how that may be implicated in the match process. Future research should involve determining whether medical students who excel or participate in these identified factors lead to success during the application and match process.

Conclusion

We have identified several residency application factors considered important by PDs, stratified by their specialty, demographics, and previous experiences. With the transition away from numerical grades/scores, medical students should be aware of the factors PDs consider important based on their chosen specialty. Our analysis may assist medical students in understanding the application and match process across various specialties.