Differences in international medical graduates’ letters of recommendation by gender in pulmonary and critical care medicine: a cohort analysis
BMC Medical Education volume 23, Article number: 58 (2023)
International Medical Graduates (IMGs) encounter barriers as they seek to match into fellowship programs in the United States (US). This study’s objective is to determine if there are differences in letters of recommendation written for IMGs compared to U.S. Medical Graduates (USMGs) applying to pulmonary and critical care medicine (PCCM) fellowship programs.
All applications submitted to a PCCM fellowship program in 2021 were included in this study. The applicant demographics and accomplishments were mined from applications. The gender of letter writers was identified by the author’s pronouns on professional websites. Word count and language differences in the letters were analyzed for each applicant using the Linguistic Inquiry and Word Count (LWIC2015) program. Multivariable linear regressions were performed controlling for applicant characteristics to identify if IMG status was associated with total word count and degree of support, measured by a composite outcome encompassing several categories of adjectives, compared to USMG status.
Of the 573 applications, most of the applicants were USMGs (72%, N = 334/573). When adjusting for applicant characteristics, IMG applicants had shorter letters of recommendation (87.81 total words shorter 95% CI: − 118.61, − 57.00, p-value < 0.01) and less supportive letters (4.79 composite words shorter 95% CI: − 6.61, − 2.97, p-value < 0.01), as compared to USMG applicants. Notably, female IMG applicants had the biggest difference in their word counts compared to USMG applicants when the letter writer was a man.
IMG applicants to a PCCM fellowship received shorter and less supportive letters of recommendation compared to USMG applicants.
Diversity, equity, and inclusion (DEI) efforts in academic medicine have focused on addressing the systemic barriers and challenges encountered by historically underrepresented in medicine (URiM) groups by offering solutions . While attention has predominantly focused on gender and racial minorities, international medical graduates (IMGs) [2, 3], defined as physicians who completed medical school outside of the United States (U.S.) or Canada , are often overlooked. Despite encountering numerous barriers and challenges to acquiring a training position in a U.S. program, IMGs make up 25% of the physician workforce and are more likely to care for patients in areas with healthcare shortages and provide care for more vulnerable patient populations [5,6,7]. These challenges may be implicitly reflected in their letters of recommendation.
Letters of recommendation play a critical role in the application process for trainees and how the letters of IMGs differ from U.S. medical graduates (USMGs) is unknown. Recent work has focused on evaluating letters of recommendation for applicants for systemic gender biases [8,9,10,11,12,13,14]. These studies have shown mixed results with women applicants sometimes having longer and more supportive letters as compared to men applicants [15,16,17]. Additionally, recent studies have suggested that there may be racial disparities in letters of recommendation with URiM applicants having shorter and less supportive letters of recommendation as compared to applicants who identified as white [18,19,20,21]. However, these studies did not identify differences for IMGs.
To fill this gap in knowledge, we investigated if letters of recommendation for IMG applicants to a pulmonary and critical care medicine (PCCM) fellowship program differed in length or language as compared to USMGs. Furthermore, we evaluated if the sex differences found in the general applicant population persisted among IMG applicants and if the gender of the letter writer contributed to these differences . We hypothesized that 1) IMG applicants would have shorter letters of recommendation compared to their U.S. counterparts, and 2) female IMG applicant letters would be longer and more supportive as compared to male IMG applicants.
All applications to the University of Michigan’s PCCM fellowship in 2021 were included in the sample.
The applicant’s self-reported race/ethnicity, sex, number of publications, number of presentations, chief medical status (CMR), Alpha Omega Alpha (AOA), and international medical graduate (IMG) status were identified from their applications and added into a de-identified electronic database, REDCap . The race/ethnicity and sex of the applicant were coded to match the groupings used by the ERAS application. We did not adjust for the type of residency training program (e.g., community vs university) as we did not expect the type of program to be associated with gender bias in the letters of recommendation, and letter writers’ hospital affiliations did not always align with those of applicants.
Applicants were identified as an IMG if they completed medical school outside of the U.S. or Canada . Applicants were identified as URiM per the definition used by the Association of American Medical Colleges as “any U.S. citizen or permanent resident who self-identified as one or more of the following race/ethnicity categories (alone or in combination with any other race/ethnicity category): American Indian or Alaska Native; Black or African American; Hispanic, Latino, or of Spanish Origin; or Native Hawaiian or other Pacific Islander.” .
The gender of the letter of recommendation writer was identified by how the letter writer was identified on university, hospital, and professional websites (e.g., Doximity and LinkedIn) where the author’s pronouns were available . If no pronouns were able to be identified, the author’s gender was listed as unknown. Since the author’s sex could not be identified, we used pronouns which more likely reflected the author’s gender.
In line with previous research, we used the Linguistic Inquiry and Word Count program (LIWC2015; Pennebaker Conglomerates, Inc., Austin, Texas). This program is a word-count based, text analysis program that quantifies language metrics. It has been previously used in multiple studies and fields to study the language used in letters of recommendation [9,10,11,12, 20, 25].
The data dictionary we used was based on our prior work, and captures the various adjectives commonly used in letters of recommendation including communal, social-communal, ability, grindstone, positive and negative agentic, research, and standout words [11, 12, 17, 20, 26]. A composite outcome measuring the degree of support was created encompassing grindstone, ability, research, standout, and positive agentic words (Supplemental Table 1).
The letters of recommendation were cleaned and deidentified using Adobe Acrobat Pro DC (Adobe, San Jose, CA). All names, salutations, dates, letterheads, and signatures were removed before processing by LWIC2015.
We used multivariable linear regression to identify if IMG applicants had shorter letters of recommendation as compared to USMG applicants, adjusting for sex, ethnicity, total number of publications, presentations, and CMR status. We did not include AOA status as not all international medical schools have this award.
We also used a multivariable linear regression to identify if IMG applicants had less supportive letters of recommendation, based on the composite outcome as compared to USMG applicants while adjusting for sex, ethnicity, total number of publications, presentations, and CMR status.
We conducted all statistical analysis with Stata software 15.1 (StataCorp).
Of the 573 applications received in 2021, 72% (N = 334/573) of the applicants were USMGs of which the majority were male (64%, N = 214/334) and white (57%, N = 191/334). Among the IMG applicants, the majority were male (67%, N = 161/239) and Asian (46%, N = 111/239) (Table 1). There were 2184 letters of recommendation reviewed of which 1280 were for USMGs and 904 were for IMGs. Letter writers were predominantly men (73%, N = 1580/2150) for both USMGs and IMGs (Table 1).
USMG applicants received longer letters of recommendation (total word count: 565, standard deviation [SD]: 360) as compared to IMG applicants (total word count: 463, SD: 277). IMG applicants received shorter letters of recommendation from both men and women letter writers as compared to USMG applicants and their letters included fewer adjectives in all categories used in the LWIC dictionary (Table 2).
When adjusting for applicant demographics (e.g., sex, race/ethnicity) and accomplishments (e.g., CMR status, number of publications and presentations), IMG applicants had shorter letters of recommendation (87.81 total words shorter 95% CI: − 118.61, − 57.00, p-value < 0.01) and less supportive letters (4.79 composite words shorter 95% CI: − 6.61, − 2.97, p-value < 0.01) as compared to USMG applicants (Table 3).
To our knowledge, no prior studies have evaluated the differences in letters of recommendation for IMGs as compared to USMGs applying to PCCM fellowship programs. We found that IMG applicants had shorter and less supportive letters of recommendation as compared to USMGs. Notably, female IMG applicants had the biggest difference in the length of their letters of recommendation as compared to USMG applicants when the letter writer was a man. Lastly, women letter writers wrote longer and more supportive letters as compared to men.
Relationship to previous studies
Our work begins to shed light on the differences that exist in the letters of recommendation for IMGs as compared to USMGs. Our findings demonstrate that IMGs have less supportive and shorter letters as demonstrated by the less frequent use of every adjective in our LWIC dictionary. While attention has focused on reducing biases in female applicant letters of recommendation, less attention has been given to differences in letters of recommendation for IMGs. These differences could suggest structural bias against IMGS, resulting in career development delays with downstream implications on the physician workforce [27,28,29].
The growing attention to address systematic barriers that impact DEI efforts in medicine has resulted in several studies evaluating bias in letters of recommendation. Historically this work was focused on the gender and URiM status of applicants, examining length of letters, discussions of research, and types of adjectives used by letter writers [20, 30]. While more recent work suggests that the trend may be changing for women applicants to PCCM, radiation oncology, and surgery, no work has evaluated the role of letters of recommendation in exacerbating the challenges faced by IMGs as they pursue specialty training and how the language in their letters may hinder them in their pursuits [15, 20, 21].
Research aimed at meeting academic medicine’s stated goal of improving DEI also needs to focus on addressing the existing biases within letters of recommendation for IMGs. IMGs make up approximately 25% of the physician workforce in the U.S. and are more likely to care for patients in underserved areas . IMGs are also less likely to match into residency programs . One step toward improving the representation of women, IMGs, and URiM fellows is addressing the disparities in language within letters of recommendation and educating letter writers of these biases.
Historically disadvantaged groups have had to work “twice as hard and twice as long” to even be considered acceptable for a position [13, 33]. This phenomenon has been described in the challenges women and other URiM physicians have encountered in medicine throughout their academic career and has been a focus of why the content of letters of recommendation matters so much more . Our study sheds light on the unfortunate reality of how disparate the letters of recommendation are for IMGs. Based on the implicit biases held against disadvantaged groups, IMGs would not only need similar length and supportive letters of recommendation, but their letters of recommendation would need to be longer and more supportive for them to be considered “acceptable” [27,28,29, 35, 36]. Until we establish clear, comprehensive evaluations of candidates that are not heavily dependent on letters of recommendation, personal and systematic implicit biases will continue to impact residents’ ability to match into the specialty of their choice and hinder efforts to improve DEI within academic medicine.
One step toward reducing the impact of systematic biases in letters of recommendation for fellowship applicants is for programs to move away from using letters of recommendation to evaluate applications.
Strength and limitations
There are several limitations to our study. First, the LWIC2015 program relies on word counts and does not offer insight into the context in which the words are being used. Second, this is a single site study and only includes applicants to an academic PCCM program, making it difficult to generalize beyond the subfield. Third, we were unable to distinguish between U.S. citizens who attended medical school abroad and IMGs who do not have U.S. citizenship. Fourth, it remains unknown if the length of the letters of recommendation truly matters in the selection process of applicants. However, having letters which encompass more supportive adjectives likely reads more favorably to selection committees. Finally, we did not adjust for university versus community-affiliated status of the residency programs – since IMGs are more likely to train at community-affiliated programs , it is possible that the training and experience of faculty writing the letters of recommendation might influence the content and length of the letters.
IMGs applicants as compared to USMG applicants to an academic PCCM program received shorter and less supportive letters of recommendation. Further work is needed to determine if these findings vary based on the type of residency program or the applicant’s immigration status.
Availability of data and materials
The datasets used and analyzed during the current study are available from the corresponding author on reasonable requests.
International medical graduate
United States medical graduate
Pulmonary and critical care medicine
Diversity, equity, and inclusion
Underrepresented in medicine
Chief medical resident
Alpha Omega Alpha
Westring A, McDonald JM, Carr P, Grisso JA. An integrated framework for gender equity in academic medicine. Acad Med. 2016;91(8):1041–4.
Ahmed AA, Hwang W-T, Thomas CR Jr, Deville C Jr. International medical graduates in the US physician workforce and graduate medical education: current and Historial trends. J Grad Med Educ. 2018;10(2):214–8.
American Medical Association. How IMGs have changed the face of American medicine. Chicago (United States): American Medical Association; 2021. Available from: https://www.ama-assn.org/education/international-medical-education/how-imgs-have-changed-face-american-medicine. Accessed 15 May 2022.
American Medical Assocation. Support for IMGs Practicing in the U.S. Chicago (United States): American Medical Association. Available from: https://www.ama-assn.org/system/files/cme-issue-brief-support-imgs.pdf. Accessed 15 May 2022.
Chen PG-CC, Ann L. Professional challenges of non-U.S.-born international medical graduates and recommendations for support during residency. Acad Med. 2011;86(11):1383–8.
Fiscella K, Frankel R. Overcoming cultural barriers: international medical graduates in the United States. JAMA. 2000;283(12):1751.
Morris AL, Phillips RL, Fryer GR Jr, Green LA, Mullan F. International medical graduates in family medicine in the United States of America: an exploration of professional characteristics and attitudes. Hum Resour Health. 2006;4:17.
Hoffman A, Ghoubrial R, McCormick M, Matemavi P, Cusick R. Exploring the gender gap: letters of recommendation to pediatric surgery fellowship. Am J Surg. 2020;219(6):932–6.
Filippou P, Mahajan S, Deal A, Wallen EM, Tan HJ, Pruthi RS, et al. The presence of gender Bias in letters of recommendations written for urology residency applicants. Urology. 2019;134:56–61.
Lin F, Oh SK, Gordon LK, Pineles SL, Rosenberg JB, Tsui I. Gender-based differences in letters of recommendation written for ophthalmology residency applicants. BMC Med Educ. 2019;19(1):476.
Madera JM, Hebl MR, Martin RC. Gender and letters of recommendation for academia: agentic and communal differences. J Appl Psychol. 2009;94(6):1591–9.
Schmader T, Whitehead J, Wysocki VH. A linguistic comparison of letters of recommendation for male and female chemistry and biochemistry job applicants. Sex Roles. 2007;57(7–8):509–14.
Trix FP, C. Exploring the color of glass: letters of recommendation for female and male faculty. Discourse Soc. 2003;14(2):191–200.
Turrentine FE, Dreisbach CN, St Ivany AR, Hanks JB, Schroen AT. Influence of gender on surgical residency Applicants' recommendation letters. J Am Coll Surg. 2019;228(4):356–65 e3.
French JC, Zolin SJ, Lampert E, Aiello A, Bencsath KP, Ritter KA, et al. Gender and letters of recommendation: A linguistic comparison of the impact of gender on general surgery residency applicants(). J Surg Educ. 2019;76(4):899–905.
Chapman BV, Rooney MK, Ludmir EB, De La Cruz D, Salcedo A, Pinnix CC, et al. Linguistic biases in letters of recommendation for radiation oncology residency applicants from 2015 to 2019. J Cancer Educ. 2022,37:965-72.
Darvall JN, Pilcher D, Bellomo R, Zampieri FG, Iwashyna TJ, Viglianti EM, et al. Discussion about “Association of frailty with short-term outcomes, organ support and resource use in critically ill patients”. Intensive Care Med. 2018;44(11):2014–6.
Powers A, Gerull KM, Rothman R, Klein SA, Wright RW, Dy CJ. Race-and gender-based differences in descriptions of applicants in the letters of recommendation for orthopaedic surgery residency. JBJS Open Access. 2020;5(3):e20.00023.
Zhang N, Blisset S, Anderson D, O'Sullivan P, Qasim A. Race and gender bias in internal medicine program director letters of recommendation. J Grad Med Educ. 2021;13(3):335–44.
Grimm LJ, Redmond RA, Campbell JC, Rosette AS. Gender and racial Bias in radiology residency letters of recommendation. J Am Coll Radiol. 2020;17(1 Pt A):64–71.
Viglianti E, Sheffield V, Vita A, Choudhuri I, Goldberg R, Byrd KM, et al. Gendered language in letters of recommendation for applicants to pulmonary critical care fellowships. ATS Sch. 2022;3(3):413–24.
Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.
Colleges AoAM. FACTS glossary Washington, DC 2021 Available from: https://www.aamc.org/data-reports/students-residents/interactive-data/facts-glossary. Accessed 15 Jun 2022.
Jagsi R, Guancial EA, Worobey CC, Henault LE, Chang Y, Starr R, et al. The “gender gap” in authorship of academic medical literature--a 35-year perspective. N Engl J Med. 2006;355(3):281–7.
Li S, Fant AL, McCarthy DM, Miller D, Craig J, Kontrick A. Gender differences in language of standardized letter of evaluation narratives for emergency medicine residency applicants. AEM Educ Train. 2017;1(4):334–9.
Women TUoACotSo. Avoiding gender bias in reference writing. Tuscon: The University of Arizona. Available from: https://csw.arizona.edu/sites/default/files/avoiding_gender_bias_in_letter_of_reference_writing.pdf. Accessed 15 June 2022.
Garcha N. Foregin-trained doctors file B.C. human rights complaint, claiming systemic discrimination. Global News 2020.
Chen PG, Nunez-Smith M, Bernheim SM, Berg D, Gozu A, Curry LA. Professional experiences of international medical graduates practicing primary care in the United States. J Intern Med. 2010;25(9):947–53.
Pemberton M, Gnanapragasam S, Bhugra D. International medical graduates: challenges and solutions in psychiatry. BJPsych Int. 2022;19(2):30–3.
Horwood C, McDermott S, Gennell T, Pawlik TM, Grignol VP, Hughes TM. Letters of recommendation for surgical fellowships - does gender make a difference? Am J Surg. 2021;221(1):90–4.
Cohen J. The role and contributions of IMGs: a U.S. Perspect Acad Med. 2006;82(12):385–90.
Henry TA. Match again sees record numbers. Take a peek behind the data. Chicago (United States): American Medical Association; 2022. Available from: https://www.ama-assn.org/residents-students/match/2022-match-again-sees-record-numbers-take-peek-behind-data. Accessed 15 June 2022.
Bates C, Gordon L, Travis E, Chatterjee A, Chaudron L, Fivush B, et al. Striving for gender equity in academic medicine careers: A call to action. Acad Med. 2016;91(8):1050–2.
Rojek AE, Khanna R, Yim JWL, Gardner R, Lisker S, Hauer KE, et al. Differences in narrative language in evaluations of medical students by gender and under-represented minority status. J Gen Intern Med. 2019;34(5):684–91.
Desbiens N, Vidaillet H. Discrimination against international medical graduates in the United States residency program selection process. BMC Med Educ. 2010;10:5.
Advance LU. Gender Bias calculator: letters of recommendation. Bethlehem: Lehigh University; 2016. Available from: https://advance.cc.lehigh.edu/content/gender-bias-calculator-letters-recommendation. Accessed 15 June 2022.
Ashley Shen and Rida Qureshi.
This work was supported by grants NHLBI K23 HL157364 (EMV).
Ethics approval and consent to participate
This study was approved by the University of Michigan IRB on November 21, 2021 the number is HUM00207734, which granted this project a category 4 exemption for informed consent. The work was conducted following the Declaration of Helsinki.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Byrd, K.M., Jain, S., Choudhuri, I. et al. Differences in international medical graduates’ letters of recommendation by gender in pulmonary and critical care medicine: a cohort analysis. BMC Med Educ 23, 58 (2023). https://doi.org/10.1186/s12909-023-04042-5