Study population
All applications to the University of Michigan’s PCCM fellowship in 2021 were included in the sample.
The applicant’s self-reported race/ethnicity, sex, number of publications, number of presentations, chief medical status (CMR), Alpha Omega Alpha (AOA), and international medical graduate (IMG) status were identified from their applications and added into a de-identified electronic database, REDCap [22]. The race/ethnicity and sex of the applicant were coded to match the groupings used by the ERAS application. We did not adjust for the type of residency training program (e.g., community vs university) as we did not expect the type of program to be associated with gender bias in the letters of recommendation, and letter writers’ hospital affiliations did not always align with those of applicants.
Applicants were identified as an IMG if they completed medical school outside of the U.S. or Canada [4]. Applicants were identified as URiM per the definition used by the Association of American Medical Colleges as “any U.S. citizen or permanent resident who self-identified as one or more of the following race/ethnicity categories (alone or in combination with any other race/ethnicity category): American Indian or Alaska Native; Black or African American; Hispanic, Latino, or of Spanish Origin; or Native Hawaiian or other Pacific Islander.” [23].
The gender of the letter of recommendation writer was identified by how the letter writer was identified on university, hospital, and professional websites (e.g., Doximity and LinkedIn) where the author’s pronouns were available [24]. If no pronouns were able to be identified, the author’s gender was listed as unknown. Since the author’s sex could not be identified, we used pronouns which more likely reflected the author’s gender.
Data dictionary
In line with previous research, we used the Linguistic Inquiry and Word Count program (LIWC2015; Pennebaker Conglomerates, Inc., Austin, Texas). This program is a word-count based, text analysis program that quantifies language metrics. It has been previously used in multiple studies and fields to study the language used in letters of recommendation [9,10,11,12, 20, 25].
The data dictionary we used was based on our prior work, and captures the various adjectives commonly used in letters of recommendation including communal, social-communal, ability, grindstone, positive and negative agentic, research, and standout words [11, 12, 17, 20, 26]. A composite outcome measuring the degree of support was created encompassing grindstone, ability, research, standout, and positive agentic words (Supplemental Table 1).
The letters of recommendation were cleaned and deidentified using Adobe Acrobat Pro DC (Adobe, San Jose, CA). All names, salutations, dates, letterheads, and signatures were removed before processing by LWIC2015.
Statistical analysis
We used multivariable linear regression to identify if IMG applicants had shorter letters of recommendation as compared to USMG applicants, adjusting for sex, ethnicity, total number of publications, presentations, and CMR status. We did not include AOA status as not all international medical schools have this award.
We also used a multivariable linear regression to identify if IMG applicants had less supportive letters of recommendation, based on the composite outcome as compared to USMG applicants while adjusting for sex, ethnicity, total number of publications, presentations, and CMR status.
We conducted all statistical analysis with Stata software 15.1 (StataCorp).