- Research article
- Open Access
- Open Peer Review
Associations between resident physicians’ publications and clinical performance during residency training
BMC Medical Educationvolume 16, Article number: 22 (2016)
Both research and clinical medicine requires similar attributes of efficiency, diligence and effective teamwork. Furthermore, residents must succeed at scholarship and patient care to be competitive for fellowship training. It is unknown whether research productivity among residents is related to broad measures of clinical achievement. Our goal was to examine associations between the quantity of internal medicine residents’ publications and validated measures of their knowledge, skills and multi-source evaluations of performance.
This was a longitudinal study of 308 residents graduating from Mayo Clinic from 2006 to 2012. We identified peer-reviewed articles in Ovid MEDLINE between July of each resident’s match year and the end of their graduation. Outcomes included American Board of Internal Medicine (ABIM) certification examination scores, mini clinical examination (mini-CEX) scores, and validated assessments of clinical performance by resident-peers, faculty and non-physicians. Performance assessments were averaged to form an overall score ranging from 1 to 5. Associations between quantity of resident publications – and ABIM, mini-CEX and performance assessment scores – were determined using multivariate linear regression.
The residents published 642 papers, of which 443 (69.0 %) were research papers, 198 (30.8 %) were case reports, and 380 (59.2 %) were first-authored. On adjusted analysis, multi-source clinical performance evaluations were significantly associated (beta; 99 % CI; p-value) with the numbers of research articles (0.012; 0.001–0.024; 0.007), and overall publications (0.012; 0.002–0.022; 0.002).
To our knowledge, this is the first study to demonstrate that scholarly productivity based on journal publication is associated with clinical performance during residency training. Our findings suggest that residents who invest substantial efforts in research are not compromised in their abilities to learn medicine and care for patients.
To become reflective and competent doctors, resident physicians must be trained to interpret the literature, apply evidence to patient care and demonstrate competence in research methods . Therefore, the Accreditation Council for Graduate Medical Education (ACGME) requires that residents participate in scholarly activities .
Dedicated research time during medical school and residency is associated with increased publication by learners . The Lincoln Hospital found that that 21 % of residents who participated in a research elective published, compared to only 5 % of residents who did not have the elective . Likewise, a Mayo Clinic study demonstrated that internal medicine residents, all of whom engage in research rotations, publish at significantly higher rates than similar residents at other institutions .
Students and residents who receive dedicated research time and publish are more likely to select careers in academic medicine and ultimately achieve higher academic rank [3, 5–12]. For example, Johns Hopkins medical students with research experience were nearly three times more likely to pursue an academic medical career compared to those without such experience . Similar findings have been demonstrated among resident physicians in the fields of urology , general surgery , neurosurgery , pediatrics , and emergency medicine .
Although research experience during medical school and residency appears to be correlated with future careers in academia, few studies have addressed potential associations between scholarly productivity and performance during residency, and research has failed to demonstrate a connection between publishing during medical school and subsequent performance during residency training [14–16]. For instance, researchers at one academic medical center found that the quantity and quality of publications during medical school had no apparent impact on future resident performance in terms of supervisory ratings and standardized test scores .
It remains unknown whether research productivity among resident physicians is related to broad measures of their clinical performance during residency training. Substantial time investment is required to be successful at research [18–22], which suggests that residents who excel in scholarship might suffer in their abilities to care for patients [23–25]. However, both research and clinical medicine require similar traits like efficiency, diligence, and effective teamwork, and residents must be successful at both scholarship and patient care to compete for fellowship training. Consequently, we hypothesized that there would be an association between residents’ publications, knowledge acquisition, and clinical accomplishment. Our goal was to utilize a longitudinal study design to examine potential associations between the quantity of internal medicine residents’ peer-reviewed publications and diverse measures of their clinical performance using validated measures of knowledge (ABIM certification examination scores), skills (mini-clinical examination scores) and multi-source evaluations of clinical performance.
Resident research curriculum
The Mayo Clinic internal medicine resident research curriculum extends across three years of training and includes the following topics: project planning, protocol design, basic principles of biostatistics and epidemiology, human subjects’ protection, and scientific writing. Educational resources are also available through the Mayo School of Graduate Medical Education, Mayo Center for Clinical and Translational Science, and biannual scientific writing workshops. Each year, residents may engage in month-long research electives, which require completion of a standardized application form, a mentor’s letter of support, and final approval by the associate program director for resident scholarship. Following their electives, residents receive evaluation and feedback from their mentors.
Independent measure: resident publications
To quantify resident scholarship in terms of amount (counts), authorship role (first author, yes or no), and category (original research and review articles versus case report), we identified peer-reviewed articles indexed in Ovid MEDLINE between July of each resident’s match year and the end of their graduation calendar year. Articles were found by searching author last name, first initial, and middle initial when available. The resulting references were then examined manually for full-name matches. For references without full first names or where multiple authors were listed under the same name, we checked the Electronic Residency Application Service record. We excluded non-journal articles and any publication dated before July 1 of the resident’s match year. For each resident, we recorded the total number of peer-reviewed articles, case reports, and first-authored publications.
Validity of outcome measures and co-variables
The measures used in this study are supported by validity evidence. Research has demonstrated predictive validity of the ABIM certification examination . Several studies have established the validity of in-training examination (ITE) scores, including correlation with resident conference attendance and self-directed reading, and no correlation with empathy [27–29]. Validity and reliability of the mini clinical evaluation exercise (mini-CEX) has been verified by previous studies at the Mayo Clinic and elsewhere [30–36]. We use a traditional version of the mini-CEX; however, the items are on 5-point, as opposed to 9-point, scales. Clinical performance assessments of Mayo Clinic internal medicine residents are completed by peers, senior medical residents, faculty, and non-physician professionals. Validity of these assessments is based on elements from previously published instruments, input from experts with experience in scale design, factor analysis showing multiple dimensions, and excellent internal consistency reliability .
Study design and data analysis
This was a longitudinal study of 308 post-graduate-year three (PGY-3) residents graduating from Mayo Clinic Rochester from 2006 to 2012. Associations of resident publications – with the outcomes of ABIM average percent correct, mini-CEX and clinical performance assessment scores – were determined using multivariate linear regression models. Scores for the clinical performance assessment items were averaged across year-three within assessor group to form an overall score ranging from 1 to 5. Co-variables adjusted for included sex (male; female), medical school (U.S. versus international medical graduate), track (categorical; clinician-investigator), PhD (yes; no), age, and percent correct on ITE examination. The threshold for statistical significance was set conservatively at 0.01 to account for multiple comparisons. This study of 308 residents provided 90 % power to detect a medium Cohen’s f2 effect size of 0.25 for adjusted associations between the number of publications and each outcome variable. Statistical analyses were performed using SAS version 9.3 (SAS Institute Inc., Cary, NC). The Mayo Clinic Institutional Review Board deemed this study exempt under 45 CFR 46.101, on the basis of existing data that were recorded in such a manner that the subjects cannot be identified.
Of the 308 residents, 197 (64.0 %) were male, 259 (84.1 %) were U.S. medical graduates, 279 (90.6 %) were categorical, and 11 (3.6 %) had PhD degrees. Their average age was 29.7 years. The residents published 642 papers, of which 443 (69.0 %) were research papers, 198 (30.8 %) were case reports, and 380 (59.2 %) were first-authored (Table 1).
Upon adjusted analysis, multi-source evaluations of clinical performance were significantly associated (beta; 99 % CI; p-value) with the number of overall publications (0.012; 0.002–0.022; 0.002), and research articles (0.012; 0.001–0.024; 0.007 [Table 2]). In other words, each additional publication was associated with a 0.012-point increase in the expected mean PGY-3 multisource evaluation score after adjusting for all covariates. There were no statistically significant associations between quantity of publications and ABIM certification examination or mini-CEX scores.
To our knowledge, this is the first study to explore relationships between residents’ scholarly activity in terms of publications and broad measures of their clinical performance during residency training. We found that residents’ overall numbers of publications in peer-reviewed journals were associated with validated, multi-source assessments of their clinical performance. These findings suggest that a potential overlap exists between the skills required for research and medical practice. Additionally, these findings indicate that residents who invest time in research may not be compromised in their abilities to care for patients and contribute to healthcare teams.
Although there was a positive association between resident scholarship and clinical evaluation scores, it is noteworthy that there were no negative associations between resident scholarship and any aspect of clinical performance. In other words, our findings would suggest that research does not detract from clinical training, which is the main goal of residency training [25, 38, 39]. These findings support a study of residents who, despite logging fewer clinical cases than their colleagues who did not do research, performed better on oral examinations at graduation . Additionally, these findings underscore a previously-untested assumption that providing education in research during residency training likely improves residents’ skills across all of the ACGME core competencies .
Potential explanations for the association between resident scholarship and clinical performance include external and internal forms of motivation, as well as traits that are adaptive to both research and clinical practice . As for external motivation, it is recognized that selection criteria for competitive fellowship training programs and academic appointments includes a strong emphasis on peer-reviewed publications [40, 42, 43]. Examples of internally motivating factors are intellectual curiosity and a personal drive for excellence that would translate to all aspects of medicine, including scholarship and patient care. Similarly, it has been observed that achievement in research and clinical practice require the abilities of time management, efficiency, diligence and effective teamwork . It is possible that residents who are widely recognized for their research achievements enjoy a “halo effect,” which could render them attractive across diverse aspects of their work [45, 46]. Lastly, it has been shown that residents who participate in research have a higher satisfaction with residency training; this may enhance their clinical accomplishments and push them towards careers in academic medicine [47, 48].
This study expands upon previous research regarding scholarship among medical students and residents. One study did not identify a correlation between publications during medical school and subsequent in-training exam scores or global faculty-of-resident assessments during residency training , although this study may have been limited by low publication rates. Another study revealed that residents with research experience are more likely to pursue academic careers . One study demonstrated a correlation between research experience and clinical achievement during residency . However, the variables in that study were research elective and performance on an oral examination. Our study involved measured numbers of peer-reviewed publications by internal medicine residents, along with validated assessments of clinical performance, and the universally-important outcome measures of mini-CEX and ABIM certification scores.
This study has several limitations. It was conducted at a single institution, which may limit generalization of findings to other settings. Specifically, aspects of our residency research program that might be distinctive and are recognized as valuable for success include a dedicated research director, robust and longitudinal research curriculum, statistical resources, faculty with established research careers, and a strong culture of mentoring support [5, 39, 49–51]. Similarly, the assessment of clinical performance that was utilized in this study is unique to the Mayo Clinic; nonetheless, this assessment reflects the same ACGME competencies that are utilized by other institutions , and validity of the instruments’ scores include relations to other meaningful variables such as resident physician professionalism and well-being [52, 53].
This study showed that numbers of journal publications by residents were positively associated with their clinical performance assessment scores during residency training. These findings suggest that research does not interfere with residents’ abilities to care for patients and work in healthcare teams. Potential explanations for the study findings are that certain traits – such as discipline, critical thinking ability, organization, and ability to work in teams – are adaptive to excellence in both research and clinical medicine. To be selected for fellowship programs, residents must demonstrate both scholarly and clinical success. Therefore, additional research should determine whether there is a relationship between publication during residency and future accomplishments such as selection into competitive fellowship programs.
American board of internal medicine
Accreditation council for graduate medical education
Mini clinical evaluation exercise
Post-graduate year 3
Schon DA. The Reflective Practitioner. 1st ed. New York: Basic Books, Inc.; 1983.
Common Program Requirements Effective July 1, 2013. Accreditation Council for Graduate Medical Education. Available at: https://www.acgme.org/acgmeweb/Portals/0/PFAssets/ProgramRequirements/CPRs2013.pdf. Accessed December 8, 2013.
Dyrbye LN, Davidson LW, Cook DA. Publications and presentations resulting from required research by students at Mayo Medical School, 1976–2003. Acad Med. 2008;83(6):604–10.
Kanna B, Deng C, Erickson SN, Valerio JA, Dimitrov V, Soni A. The research rotation: competency-based structured and novel approach to research training of internal medicine residents. BMC Med Educ. 2006;6:52–60.
West CP, Halvorsen AJ, McDonald FS. Scholarship during residency training: a controlled comparison study. Am J Med. 2011;124(10):983–7.
Bhattacharya SD, Williams JB, De la Fuente SG, Kuo PC, Seigler HF. Does protected research time during general surgery training contribute to graduate' career choice? Am J Surg. 2011;77(7):907–10.
Holmes JF, Sokolove PE, Panacek EA. Ten-year experience with an emergency medicine resident research project requirement. Acad Emerg Med. 2006;13(5):575–9.
Yang G, Zaid UB, Erickson BA, Blaschko SD, Carroll PR, Breyer BN. Urology resident publication output and its relationship to future academic achievement. J Urol February. 2011;185(2):642–6.
Brancati FL, Mead LA, Levine DM, Martin D, Marolis S, Klag MJ. Early Predictors of career acheivment in academic medicine. JAMA. 1992;267(10):1372–6.
Burkhardt J, Kowalenko T, Meurer W. Academic career selection in American emergency medicine residents. Acad Emerg Med. 2011;18 suppl 2:S48–53.
McClelland III S. Pre-residency peer-reviewed publications are associated with neurosurgery resident choice of academic compared to private practice careers. J Clin Neurosci. 2010;17(3):287–9.
Rezek I, McDonald RJ, Kallmes DF. Pre-residency publication rate strongly predicts future academic radiology potential. Acad Radiol. 2012;19(5):632–4.
Ledley FD, Lovejoy Jr FH. Factors influencing the interests, career paths, and research activities of recent graduates from an academic pediatric residency program. Pediatrics. 1993;92(3):436–41.
Stohl HE, Hueppchen NA, Bienstock JL. Can medical school performance predict residency performance? Resident selection and predictors of successful performance in obstetrics and gynecology. J Grad Med Educ. 2010;2(3):322–6.
Dirschl DR, Dahners LE, Adams GL, Crouch JH, Wilson FC. Correlating selection criteria with subsequent performance as residents. Clin Orthop. 2002;399:265–71.
Carmichael KD, Westmoreland JB, Thomas JA, Patterson RM. Relation of residency selection factors to subsequent orthopaedic in-training examination performance. South Med J. 2005;98(5):528–32.
Cavalcanti RB, Detsky AS. Publishing history does not correlate with clinical performance among internal medicine residents. Med Educ. 2010;44(5):468–74.
Bland CJ, Schmitz CC. Characteristics of the succesful researcher and implications for faculty development. J Med Educ. 1986;61(1):22–31.
Goldman L. Blueprint for a research career in general internal medicine. J Gen Intern Med. 1991;6(4):341–4.
Bauchner H. Mentoring clinical researchers. Arch Dis Child. 2002;87(1):82–4.
Toto RD. How to launch a successful career in clinical research: tips on making the most of available resources. J Investig Med. 2008;56(6):843–6.
Rosier RN. Institutional barriers to the orthopaedic clinician-scientist. Clin Orthop. 2006;449:159–64.
Schott NJ, Emergick TD, Metro DG, Sakai T. The cost of resident scholarly activity and its effect on resident clinical experience. Anesth Analg. 2013;117(5):1211–6.
Robbins L, Bostrom M, Marx R, Roberts TS, Sculco TP. Restructing the orthopedic resident research curriculum to increase scholarly activity. J Grad Med Educ. 2013;5(4):646–51.
Kroenke K. Conducting research as a busy clinician-teacher or trainee. Starting blocks, hurdles, and finish lines. J Gen Intern Med. 1996;11(6):360–5.
Ramsey PG, Caroline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the american board of internal medicine. Ann Intern Med. 1989;110(9):719–26.
Garibaldi RA, Trontell MC, Waxman H, et al. The in-training examination in internal medicine. Ann Intern Med. 1994;121(2):117–23.
Garibaldi RA, Subhiyah R, Moore ME, Waxman H. The In-training examination in internal medicine: an analysis of resident performance over time. Ann Intern Med. 2002;137(6):505–10.
West CP, Huntington JL, Huschka MM, et al. A prospective study of the relationship between medical knowledge and professionalism among internal medicine residents. Acad Med. 2007;82(6):587–92.
Cook DA, Beckman TJ. Does scale length matter? A comparison of nine-versus five-point rating scales for the mini-CEX. Adv Health Sci Educ Theory Pract. 2009;14(5):655–64.
Cook DA, Beckman TJ. Reflections on experimental research in medical education. Adv Health Sci Educ Theory Pract. 2010;15(3):455–64.
Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: a method for assessing clinical skills. Ann Intern Med. 2003;138(6):476–81.
Kogan JR, Bellini LM, Shea JA. Feasibility, reliability, and validity of the mini-clinical evaluation exercise (mCEX) in a medicine core clerkship. Acad Med. 2003;78(10 suppl):S33–35.
Hatala R, Ainslie M, Kassen BO, Mackie I, Roberts JM. Assessing the mini-Clinical Evaluation Exercise in comparison to a national specialty examination. Med Educ. 2006;40(10):950–6.
Holmboe ES, Huot S, Chung J, Norcini J, Hawkins RE. Construct validity of the miniclinical evaluation exercise (miniCEX). Acad Med. 2003;78(8):826–30.
Durning SJ, Cation LJ, Markert RJ, Pangaro LN. Assessing the reliability and validity of the mini-clinical evaluation exercise for internal medicine residency training. Acad Med. 2002;77(9):900–4.
Beckman TJ, Mandrekar JN, Engstler GJ, Ficalora RD. Determining reliability of clinical assessment scores in real time. Teach Learn Med. 2009;21(3):188–94.
Applegate WB, Williams ME. Career Development in academic medicine. Am J Med. 1990;88(3):263–7.
Rothberg MB. Overcoming the obstacles to research during residency: what does it take? JAMA. 2012;308(21):2191–2.
Potti A, Mariani P, Saeed M, Smego Jr RA. Residents as researchers: expectations, requirements, and productivity. Am J Med. 2003;115(6):510–4.
Ryan RM, Deci EL. Intrinsic and Extrinsic Motivations: Classic Definitions and New Directions. Contemp Educ Psychol. 2000;25(1):54–67.
Souba WW, Tanabe KK, Gadd MA, Smith BL, Bushman MS. Attitudes and opinions toward surgical research. A survey of surgical residents and their chairpersons. Ann Surg. 1996;223(4):377–83.
Khan K, Levstik M. Ranking in Canadian gastroenterology residency match: what do residents and program directors want? Can J Gastroenterol. 2010;24(6):369–72.
Calhoun KH, Bailey BJ. Thriving in Academic Medicine. Otolaryngol Head and Neck Surg. 1997;117(6):575–9.
Iramaneerat C, Yudkowsky R. Rater errors in a clinical skills assessment of medical students. Eval Health Prof. 2007;30(3):266–83.
McLaughlin K, Vitale G, Coderre S, Violato C, Wright B. Clekship evaluation-- what are we measuring? Med Teach. 2009;31:2.
Takahashi O, Ohde S, Jacobs JL, Tokuda Y, Omata F, Fukui T. Residents' experience of scholarly activities is assocaited with higher satisfaction with residency training. J Gen Intern Med. 2009;24(6):716–20.
Zahtz G, Vambutas A, Hussey HM, Rosen L. Resident research experience and career path association: a national survey of recent otolaryngology graduates. Otolaryngol Head and Neck Surg. 2014;151(1):46–51.
Levine RB, Hebert RS, Wright SM. Resident research and scholarly activity in internal medicine residency training programs. J Gen Intern Med. 2005;20(2):155–9.
Alquire PC, Anderson WA, Albrecht RR, Poland GA. Resident research in internal medicine training programs. Ann Intern Med. 1996;124(3):321–8.
Schultz HJ. Reserach during internal medicine residency training: meeting the challenge of the Residency Review Committee. Ann Intern Med. 1996;124(3):340–2.
Beckman TJ, Reed DA, Shanafelt TD, West CP. Resident Physician well-being and assessments of their knowledge and clinical performance. J Gen Intern Med. 2012;2012(27):3.
Cullen MW, Reed DA, Halvorsen AJ, Wittich CM, Baumann Kreuziger LM, Keddis MT, et al. Selection criteria for internal medicine residency applicants and professionalism ratings during internship. The Mayo Clin Proc. 2011;33(1):197–202.
None of the authors have any competing interests.
LS, AW, CW, DR, AH and TB contributed to conception and design. LS, CW, AH and GE contributed to data acquisition. All authors contributed to literature review, interpretation of data, and drafting and revision of the article. All authors approved the final manuscript for publication. All authors are in agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.