Skip to main content
Download PDF

Differences in international medical graduates’ letters of recommendation by gender in pulmonary and critical care medicine: a cohort analysis

BMC Medical Education volume 23, Article number: 58 (2023) Cite this article

Abstract

Background

International Medical Graduates (IMGs) encounter barriers as they seek to match into fellowship programs in the United States (US). This study’s objective is to determine if there are differences in letters of recommendation written for IMGs compared to U.S. Medical Graduates (USMGs) applying to pulmonary and critical care medicine (PCCM) fellowship programs.

Methods

All applications submitted to a PCCM fellowship program in 2021 were included in this study. The applicant demographics and accomplishments were mined from applications. The gender of letter writers was identified by the author’s pronouns on professional websites. Word count and language differences in the letters were analyzed for each applicant using the Linguistic Inquiry and Word Count (LWIC2015) program. Multivariable linear regressions were performed controlling for applicant characteristics to identify if IMG status was associated with total word count and degree of support, measured by a composite outcome encompassing several categories of adjectives, compared to USMG status.

Results

Of the 573 applications, most of the applicants were USMGs (72%, N = 334/573). When adjusting for applicant characteristics, IMG applicants had shorter letters of recommendation (87.81 total words shorter 95% CI: − 118.61, − 57.00, p-value < 0.01) and less supportive letters (4.79 composite words shorter 95% CI: − 6.61, − 2.97, p-value < 0.01), as compared to USMG applicants. Notably, female IMG applicants had the biggest difference in their word counts compared to USMG applicants when the letter writer was a man.

Conclusions

IMG applicants to a PCCM fellowship received shorter and less supportive letters of recommendation compared to USMG applicants.

Peer Review reports

Background

Diversity, equity, and inclusion (DEI) efforts in academic medicine have focused on addressing the systemic barriers and challenges encountered by historically underrepresented in medicine (URiM) groups by offering solutions [1]. While attention has predominantly focused on gender and racial minorities, international medical graduates (IMGs) [2, 3], defined as physicians who completed medical school outside of the United States (U.S.) or Canada [4], are often overlooked. Despite encountering numerous barriers and challenges to acquiring a training position in a U.S. program, IMGs make up 25% of the physician workforce and are more likely to care for patients in areas with healthcare shortages and provide care for more vulnerable patient populations [5,6,7]. These challenges may be implicitly reflected in their letters of recommendation.

Letters of recommendation play a critical role in the application process for trainees and how the letters of IMGs differ from U.S. medical graduates (USMGs) is unknown. Recent work has focused on evaluating letters of recommendation for applicants for systemic gender biases [8,9,10,11,12,13,14]. These studies have shown mixed results with women applicants sometimes having longer and more supportive letters as compared to men applicants [15,16,17]. Additionally, recent studies have suggested that there may be racial disparities in letters of recommendation with URiM applicants having shorter and less supportive letters of recommendation as compared to applicants who identified as white [18,19,20,21]. However, these studies did not identify differences for IMGs.

To fill this gap in knowledge, we investigated if letters of recommendation for IMG applicants to a pulmonary and critical care medicine (PCCM) fellowship program differed in length or language as compared to USMGs. Furthermore, we evaluated if the sex differences found in the general applicant population persisted among IMG applicants and if the gender of the letter writer contributed to these differences [21]. We hypothesized that 1) IMG applicants would have shorter letters of recommendation compared to their U.S. counterparts, and 2) female IMG applicant letters would be longer and more supportive as compared to male IMG applicants.

Method

Study population

All applications to the University of Michigan’s PCCM fellowship in 2021 were included in the sample.

The applicant’s self-reported race/ethnicity, sex, number of publications, number of presentations, chief medical status (CMR), Alpha Omega Alpha (AOA), and international medical graduate (IMG) status were identified from their applications and added into a de-identified electronic database, REDCap [22]. The race/ethnicity and sex of the applicant were coded to match the groupings used by the ERAS application. We did not adjust for the type of residency training program (e.g., community vs university) as we did not expect the type of program to be associated with gender bias in the letters of recommendation, and letter writers’ hospital affiliations did not always align with those of applicants.

Applicants were identified as an IMG if they completed medical school outside of the U.S. or Canada [4]. Applicants were identified as URiM per the definition used by the Association of American Medical Colleges as “any U.S. citizen or permanent resident who self-identified as one or more of the following race/ethnicity categories (alone or in combination with any other race/ethnicity category): American Indian or Alaska Native; Black or African American; Hispanic, Latino, or of Spanish Origin; or Native Hawaiian or other Pacific Islander.” [23].

The gender of the letter of recommendation writer was identified by how the letter writer was identified on university, hospital, and professional websites (e.g., Doximity and LinkedIn) where the author’s pronouns were available [24]. If no pronouns were able to be identified, the author’s gender was listed as unknown. Since the author’s sex could not be identified, we used pronouns which more likely reflected the author’s gender.

Data dictionary

In line with previous research, we used the Linguistic Inquiry and Word Count program (LIWC2015; Pennebaker Conglomerates, Inc., Austin, Texas). This program is a word-count based, text analysis program that quantifies language metrics. It has been previously used in multiple studies and fields to study the language used in letters of recommendation [9,10,11,12, 20, 25].

The data dictionary we used was based on our prior work, and captures the various adjectives commonly used in letters of recommendation including communal, social-communal, ability, grindstone, positive and negative agentic, research, and standout words [11, 12, 17, 20, 26]. A composite outcome measuring the degree of support was created encompassing grindstone, ability, research, standout, and positive agentic words (Supplemental Table 1).

The letters of recommendation were cleaned and deidentified using Adobe Acrobat Pro DC (Adobe, San Jose, CA). All names, salutations, dates, letterheads, and signatures were removed before processing by LWIC2015.

Statistical analysis

We used multivariable linear regression to identify if IMG applicants had shorter letters of recommendation as compared to USMG applicants, adjusting for sex, ethnicity, total number of publications, presentations, and CMR status. We did not include AOA status as not all international medical schools have this award.

We also used a multivariable linear regression to identify if IMG applicants had less supportive letters of recommendation, based on the composite outcome as compared to USMG applicants while adjusting for sex, ethnicity, total number of publications, presentations, and CMR status.

We conducted all statistical analysis with Stata software 15.1 (StataCorp).

Results

Of the 573 applications received in 2021, 72% (N = 334/573) of the applicants were USMGs of which the majority were male (64%, N = 214/334) and white (57%, N = 191/334). Among the IMG applicants, the majority were male (67%, N = 161/239) and Asian (46%, N = 111/239) (Table 1). There were 2184 letters of recommendation reviewed of which 1280 were for USMGs and 904 were for IMGs. Letter writers were predominantly men (73%, N = 1580/2150) for both USMGs and IMGs (Table 1).

Table 1 Applicant demographics by IMG status, University of Michigan Pulmonary Critical Care Medicine fellowship cohort, 2021
Full size table

USMG applicants received longer letters of recommendation (total word count: 565, standard deviation [SD]: 360) as compared to IMG applicants (total word count: 463, SD: 277). IMG applicants received shorter letters of recommendation from both men and women letter writers as compared to USMG applicants and their letters included fewer adjectives in all categories used in the LWIC dictionary (Table 2).

Table 2 Average word counts by the sex of the applicant and the gender of the letter writer, by IMG status
Full size table

When adjusting for applicant demographics (e.g., sex, race/ethnicity) and accomplishments (e.g., CMR status, number of publications and presentations), IMG applicants had shorter letters of recommendation (87.81 total words shorter 95% CI: − 118.61, − 57.00, p-value < 0.01) and less supportive letters (4.79 composite words shorter 95% CI: − 6.61, − 2.97, p-value < 0.01) as compared to USMG applicants (Table 3).

Table 3 Fellowship applicant demographics and accomplishments associated with letter of recommendation length and degree of support
Full size table

Discussions

Key findings

To our knowledge, no prior studies have evaluated the differences in letters of recommendation for IMGs as compared to USMGs applying to PCCM fellowship programs. We found that IMG applicants had shorter and less supportive letters of recommendation as compared to USMGs. Notably, female IMG applicants had the biggest difference in the length of their letters of recommendation as compared to USMG applicants when the letter writer was a man. Lastly, women letter writers wrote longer and more supportive letters as compared to men.

Relationship to previous studies

Our work begins to shed light on the differences that exist in the letters of recommendation for IMGs as compared to USMGs. Our findings demonstrate that IMGs have less supportive and shorter letters as demonstrated by the less frequent use of every adjective in our LWIC dictionary. While attention has focused on reducing biases in female applicant letters of recommendation, less attention has been given to differences in letters of recommendation for IMGs. These differences could suggest structural bias against IMGS, resulting in career development delays with downstream implications on the physician workforce [27,28,29].

The growing attention to address systematic barriers that impact DEI efforts in medicine has resulted in several studies evaluating bias in letters of recommendation. Historically this work was focused on the gender and URiM status of applicants, examining length of letters, discussions of research, and types of adjectives used by letter writers [20, 30]. While more recent work suggests that the trend may be changing for women applicants to PCCM, radiation oncology, and surgery, no work has evaluated the role of letters of recommendation in exacerbating the challenges faced by IMGs as they pursue specialty training and how the language in their letters may hinder them in their pursuits [15, 20, 21].

Study implications

Research aimed at meeting academic medicine’s stated goal of improving DEI also needs to focus on addressing the existing biases within letters of recommendation for IMGs. IMGs make up approximately 25% of the physician workforce in the U.S. and are more likely to care for patients in underserved areas [31]. IMGs are also less likely to match into residency programs [32]. One step toward improving the representation of women, IMGs, and URiM fellows is addressing the disparities in language within letters of recommendation and educating letter writers of these biases.

Historically disadvantaged groups have had to work “twice as hard and twice as long” to even be considered acceptable for a position [13, 33]. This phenomenon has been described in the challenges women and other URiM physicians have encountered in medicine throughout their academic career and has been a focus of why the content of letters of recommendation matters so much more [34]. Our study sheds light on the unfortunate reality of how disparate the letters of recommendation are for IMGs. Based on the implicit biases held against disadvantaged groups, IMGs would not only need similar length and supportive letters of recommendation, but their letters of recommendation would need to be longer and more supportive for them to be considered “acceptable” [27,28,29, 35, 36]. Until we establish clear, comprehensive evaluations of candidates that are not heavily dependent on letters of recommendation, personal and systematic implicit biases will continue to impact residents’ ability to match into the specialty of their choice and hinder efforts to improve DEI within academic medicine.

One step toward reducing the impact of systematic biases in letters of recommendation for fellowship applicants is for programs to move away from using letters of recommendation to evaluate applications.

Strength and limitations

There are several limitations to our study. First, the LWIC2015 program relies on word counts and does not offer insight into the context in which the words are being used. Second, this is a single site study and only includes applicants to an academic PCCM program, making it difficult to generalize beyond the subfield. Third, we were unable to distinguish between U.S. citizens who attended medical school abroad and IMGs who do not have U.S. citizenship. Fourth, it remains unknown if the length of the letters of recommendation truly matters in the selection process of applicants. However, having letters which encompass more supportive adjectives likely reads more favorably to selection committees. Finally, we did not adjust for university versus community-affiliated status of the residency programs – since IMGs are more likely to train at community-affiliated programs [2], it is possible that the training and experience of faculty writing the letters of recommendation might influence the content and length of the letters.

Conclusion

IMGs applicants as compared to USMG applicants to an academic PCCM program received shorter and less supportive letters of recommendation. Further work is needed to determine if these findings vary based on the type of residency program or the applicant’s immigration status.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable requests.

Abbreviations

IMG:

International medical graduate

USMG:

United States medical graduate

PCCM:

Pulmonary and critical care medicine

DEI:

Diversity, equity, and inclusion

URiM:

Underrepresented in medicine

CMR:

Chief medical resident

AOA:

Alpha Omega Alpha

SD:

Standard deviation

References

  1. Westring A, McDonald JM, Carr P, Grisso JA. An integrated framework for gender equity in academic medicine. Acad Med. 2016;91(8):1041–4.

    Article  Google Scholar 

  2. Ahmed AA, Hwang W-T, Thomas CR Jr, Deville C Jr. International medical graduates in the US physician workforce and graduate medical education: current and Historial trends. J Grad Med Educ. 2018;10(2):214–8.

    Article  Google Scholar 

  3. American Medical Association. How IMGs have changed the face of American medicine. Chicago (United States):  American Medical Association; 2021. Available from: https://www.ama-assn.org/education/international-medical-education/how-imgs-have-changed-face-american-medicine. Accessed 15 May 2022.

    Google Scholar 

  4. American Medical Assocation. Support for IMGs Practicing in the U.S. Chicago (United States): American Medical Association. Available from: https://www.ama-assn.org/system/files/cme-issue-brief-support-imgs.pdf. Accessed 15 May 2022.

  5. Chen PG-CC, Ann L. Professional challenges of non-U.S.-born international medical graduates and recommendations for support during residency. Acad Med. 2011;86(11):1383–8.

    Article  Google Scholar 

  6. Fiscella K, Frankel R. Overcoming cultural barriers: international medical graduates in the United States. JAMA. 2000;283(12):1751.

    Article  Google Scholar 

  7. Morris AL, Phillips RL, Fryer GR Jr, Green LA, Mullan F. International medical graduates in family medicine in the United States of America: an exploration of professional characteristics and attitudes. Hum Resour Health. 2006;4:17.

    Article  Google Scholar 

  8. Hoffman A, Ghoubrial R, McCormick M, Matemavi P, Cusick R. Exploring the gender gap: letters of recommendation to pediatric surgery fellowship. Am J Surg. 2020;219(6):932–6.

    Article  Google Scholar 

  9. Filippou P, Mahajan S, Deal A, Wallen EM, Tan HJ, Pruthi RS, et al. The presence of gender Bias in letters of recommendations written for urology residency applicants. Urology. 2019;134:56–61.

    Article  Google Scholar 

  10. Lin F, Oh SK, Gordon LK, Pineles SL, Rosenberg JB, Tsui I. Gender-based differences in letters of recommendation written for ophthalmology residency applicants. BMC Med Educ. 2019;19(1):476.

    Article  Google Scholar 

  11. Madera JM, Hebl MR, Martin RC. Gender and letters of recommendation for academia: agentic and communal differences. J Appl Psychol. 2009;94(6):1591–9.

    Article  Google Scholar 

  12. Schmader T, Whitehead J, Wysocki VH. A linguistic comparison of letters of recommendation for male and female chemistry and biochemistry job applicants. Sex Roles. 2007;57(7–8):509–14.

    Article  Google Scholar 

  13. Trix FP, C. Exploring the color of glass: letters of recommendation for female and male faculty. Discourse Soc. 2003;14(2):191–200.

    Article  Google Scholar 

  14. Turrentine FE, Dreisbach CN, St Ivany AR, Hanks JB, Schroen AT. Influence of gender on surgical residency Applicants' recommendation letters. J Am Coll Surg. 2019;228(4):356–65 e3.

    Article  Google Scholar 

  15. French JC, Zolin SJ, Lampert E, Aiello A, Bencsath KP, Ritter KA, et al. Gender and letters of recommendation: A linguistic comparison of the impact of gender on general surgery residency applicants(). J Surg Educ. 2019;76(4):899–905.

    Article  Google Scholar 

  16. Chapman BV, Rooney MK, Ludmir EB, De La Cruz D, Salcedo A, Pinnix CC, et al. Linguistic biases in letters of recommendation for radiation oncology residency applicants from 2015 to 2019. J Cancer Educ. 2022,37:965-72.

  17. Darvall JN, Pilcher D, Bellomo R, Zampieri FG, Iwashyna TJ, Viglianti EM, et al. Discussion about “Association of frailty with short-term outcomes, organ support and resource use in critically ill patients”. Intensive Care Med. 2018;44(11):2014–6.

    Article  Google Scholar 

  18. Powers A, Gerull KM, Rothman R, Klein SA, Wright RW, Dy CJ. Race-and gender-based differences in descriptions of applicants in the letters of recommendation for orthopaedic surgery residency. JBJS Open Access. 2020;5(3):e20.00023.

    Article  Google Scholar 

  19. Zhang N, Blisset S, Anderson D, O'Sullivan P, Qasim A. Race and gender bias in internal medicine program director letters of recommendation. J Grad Med Educ. 2021;13(3):335–44.

    Article  Google Scholar 

  20. Grimm LJ, Redmond RA, Campbell JC, Rosette AS. Gender and racial Bias in radiology residency letters of recommendation. J Am Coll Radiol. 2020;17(1 Pt A):64–71.

    Article  Google Scholar 

  21. Viglianti E, Sheffield V, Vita A, Choudhuri I, Goldberg R, Byrd KM, et al. Gendered language in letters of recommendation for applicants to pulmonary critical care fellowships. ATS Sch. 2022;3(3):413–24.

    Article  Google Scholar 

  22. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.

    Article  Google Scholar 

  23. Colleges AoAM. FACTS glossary Washington, DC 2021 Available from: https://www.aamc.org/data-reports/students-residents/interactive-data/facts-glossary. Accessed 15 Jun 2022.

    Google Scholar 

  24. Jagsi R, Guancial EA, Worobey CC, Henault LE, Chang Y, Starr R, et al. The “gender gap” in authorship of academic medical literature--a 35-year perspective. N Engl J Med. 2006;355(3):281–7.

    Article  Google Scholar 

  25. Li S, Fant AL, McCarthy DM, Miller D, Craig J, Kontrick A. Gender differences in language of standardized letter of evaluation narratives for emergency medicine residency applicants. AEM Educ Train. 2017;1(4):334–9.

    Article  Google Scholar 

  26. Women TUoACotSo. Avoiding gender bias in reference writing. Tuscon: The University of Arizona. Available from: https://csw.arizona.edu/sites/default/files/avoiding_gender_bias_in_letter_of_reference_writing.pdf. Accessed 15 June 2022.

  27. Garcha N. Foregin-trained doctors file B.C. human rights complaint, claiming systemic discrimination. Global News 2020.

  28. Chen PG, Nunez-Smith M, Bernheim SM, Berg D, Gozu A, Curry LA. Professional experiences of international medical graduates practicing primary care in the United States. J Intern Med. 2010;25(9):947–53.

    Google Scholar 

  29. Pemberton M, Gnanapragasam S, Bhugra D. International medical graduates: challenges and solutions in psychiatry. BJPsych Int. 2022;19(2):30–3.

    Article  Google Scholar 

  30. Horwood C, McDermott S, Gennell T, Pawlik TM, Grignol VP, Hughes TM. Letters of recommendation for surgical fellowships - does gender make a difference? Am J Surg. 2021;221(1):90–4.

    Article  Google Scholar 

  31. Cohen J. The role and contributions of IMGs: a U.S. Perspect Acad Med. 2006;82(12):385–90.

    Google Scholar 

  32. Henry TA. Match again sees record numbers. Take a peek behind the data. Chicago (United States): American Medical Association; 2022. Available from: https://www.ama-assn.org/residents-students/match/2022-match-again-sees-record-numbers-take-peek-behind-data. Accessed 15 June 2022.

    Google Scholar 

  33. Bates C, Gordon L, Travis E, Chatterjee A, Chaudron L, Fivush B, et al. Striving for gender equity in academic medicine careers: A call to action. Acad Med. 2016;91(8):1050–2.

    Article  Google Scholar 

  34. Rojek AE, Khanna R, Yim JWL, Gardner R, Lisker S, Hauer KE, et al. Differences in narrative language in evaluations of medical students by gender and under-represented minority status. J Gen Intern Med. 2019;34(5):684–91.

    Article  Google Scholar 

  35. Desbiens N, Vidaillet H. Discrimination against international medical graduates in the United States residency program selection process. BMC Med Educ. 2010;10:5.

    Article  Google Scholar 

  36. Advance LU. Gender Bias calculator: letters of recommendation. Bethlehem: Lehigh University; 2016. Available from: https://advance.cc.lehigh.edu/content/gender-bias-calculator-letters-recommendation. Accessed 15 June 2022.

Download references

Acknowledgements

Ashley Shen and Rida Qureshi.

Funding

This work was supported by grants NHLBI K23 HL157364 (EMV).

Author information

Authors and Affiliations

  1. Department of Internal Medicine Division of Pulmonary and Critical Care, University of Michigan, 2800 Plymouth Road, NCRC building 16, Ann Arbor, MI, 48109, USA

    Kaitland M. Byrd, Jakob I. McSparron & Elizabeth M. Viglianti

  2. Department of Internal Medicine, Section of Pulmonary, Critical Care, and Sleep Medicine, Yale School of Medicine, New Haven, CT, USA

    Snigdha Jain

  3. Internal Medicine Resident in the Department of Internal Medicine, University of Pittsburgh, Pittsburgh, PA, USA

    Irada Choudhuri

  4. Division of Pulmonary, Critical Care and Sleep Medicine, University of Washington, Seattle, WA, USA

    Başak Çoruh

  5. Veterans Affairs Center for Clinical Management Research, HSR&D Center for Innovation, Ann Arbor, MI, USA

    Elizabeth M. Viglianti

  6. Institute of Health Policy and Innovation, University of Michigan, Ann Arbor, MI, USA

    Elizabeth M. Viglianti

Authors
  1. Kaitland M. Byrd
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Snigdha Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Irada Choudhuri
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Başak Çoruh
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Jakob I. McSparron
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Elizabeth M. Viglianti
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

K.M.B. designed the study, compiled the manuscript, and is accountable for all aspects of the work. SJ, IC, BÇ, and JIM interpreted the results, and provided critical revisions for the manuscript. E.M.V. designed the study, performed the statistical analyses, interpreted the results, and provided critical revisions for the manuscript. The author(s) read and approved the final manuscript.

Authors’ information

Disclaimers: This work does not necessarily represent the views of the U.S. Government or Department of Veterans Affairs. This material is the result of work supported with resources and use of facilities at the Ann Arbor VA Medical Center.

Corresponding author

Correspondence to Kaitland M. Byrd.

Ethics declarations

Ethics approval and consent to participate

This study was approved by the University of Michigan IRB on November 21, 2021 the number is HUM00207734, which granted this project a category 4 exemption for informed consent. The work was conducted following the Declaration of Helsinki.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1: Table 1.

Data Dictionary for LWIC2015

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and Permissions

About this article

Verify currency and authenticity via CrossMark

Cite this article

Byrd, K.M., Jain, S., Choudhuri, I. et al. Differences in international medical graduates’ letters of recommendation by gender in pulmonary and critical care medicine: a cohort analysis. BMC Med Educ 23, 58 (2023). https://doi.org/10.1186/s12909-023-04042-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12909-023-04042-5

Keywords

  • Letters of recommendation
  • bias
  • Graduate medical education
  • Pulmonary and critical care
  • International medical graduates

BMC Medical Education

ISSN: 1472-6920

Contact us