Table 1 Brief summary of articles included in a systematic review of scholarship initiatives in graduate medical education, 2003-March 2017

Anandarajah, et al., 2016 [17]

Rhode Island, USA

Pre-post study using historical data as a comparison group

PGY3 Family medicine residents: 13 in comparison group (2008) vs. 12 in Primary Care Scholarly Development Program projects (2009)

1–year longitudinal PGY3 curriculum included three workshops regarding the planning of a project, six interactive seminars regarding research principles, one-to-one mentoring, and residents’ presentations at professional meetings.

No

Yes (faculty)

Increase in scholarship after implementation:

• Presentation at local/regional/ national conference or publication in a peer-reviewed journal increased from 0 in 2008 to 12 in 2009

N/A

Basu Ray et al., 2012 [18]

Louisiana, USA

Pre-post study using historical data as a comparison group

Internal medicine residents: Comparison group (2010–2011) vs. the Consolidated Academic and Research Exposition (CARE) (2011–2012)

The Consolidated Academic and Research Exposition (CARE) program included 4 core components: house staff mentoring and the Resident Career Development Program, journal clubs, medical writing instructions, and research engagements. In addition they implemented monthly research forums and mentor meetings to discuss research related issues. Particular emphasis was given to projects that could be completed within a 1-month period and result in publication.

Yes (Research elective: 1 month in PGY1 and PGY3 or up to 2 months in PGY2)

Yes (faculty)

Increase in scholarly activity from 2010 to 2011 to 2011–2012 by 110%.

• 6 submissions with 2 abstracts accepted for presentation at the Louisiana Chapter of the American College of Physicians Associates meeting (2010–2011) compared to 14 submissions, with 4 accepted for presentation (2011–2012)

• 4 submissions to the Southern Hospitalist Conference (2010–2011), which increased to 7 (2011–2012), with one receiving second place in the competition

N/A

Brackmann, et al., 2016 [19]

Michigan, USA

Pre-post study using historical data as a comparison group

Comparison group (27 months before) vs. (27 months after) workgroup discussions

Biweekly, voluntary (1–2 h) gynecologic oncology research workgroup. An informal, discussion-style format for brainstorming research ideas, formulating study protocols, and collaborating on institutional review board submissions. Additional aims included editorial feedback on abstracts and manuscripts as well as oral presentation preparation. Discrete goals are set for each project by those involved such as completing a literature search, compiling a table, or writing an abstract between meetings.

2 months of protected research time

Faculty and senior residents mentor during sessions

Increase in IRB submissions, abstracts, and manuscript submissions.

• IRB submissions increased from 2 to 3 after intervention

• Accepted abstracts increased from 0 to 9

• Manuscript submissions increased from 1 to 6 (p = 0.05)

A casual dinner is provided through faculty funds

Chang et al., 2013 [20]

Missouri, USA

Pre-post study using historical data as a comparison group

Otolaryngology residents: Comparison group (July 1997–June 2004) vs. Reward point system (July 2004–2011)

A monetary reward point system: rewarded residents for each progressive step along the research path from project design to acceptance for publication. Each point was worth $1, which was used for allowable educational expenses (maximum of 2500 points per year).

Yes (3 month dedicated research rotation: 2-months in PGY3 and 1 month in PGY4)

No

• Mean publication output per resident per year increased from 0.13 (95% CI, 0.03–0.23) before reward system to 0.43 (95% CI, 0.26–0.60) after (p = 0.004)

• Approved IRB projects increased from 0.47 (95% CI, 0.18–0.75) per resident per year (before reward system) to 1.29 (95% CI, 0.96–1.63) per resident per year (p = 0.007)

$2500/resident maximum cap

$500 dollars during PGY1, and $200 during PGY2 to assist start-up

Elliott, et al., 2009 [21]

California, USA

Pre-post study using historical data for intervention and comparison group

General Surgery residents: 42 residents in the mandatory research group (1990–1996) and 41 residents in the voluntary research group (1999–2005)

This program switched from a required to voluntary research requirement. They evaluated if there was a significant research output difference between the two cohorts.

Yes (Elective 1–2 years of research program, Most residents choose one year option.)

No

• Publications per resident per year were statically equivalent, (Early 2.0 +/− 0.4 versus late 2.6 +/− 0.5)

• Late group residents performed a higher percentage of basic science research 27.7% vs 24.4%. (P=NS)

• 33% of early cohort residents continued to fellowship vs 65% of late cohort residents

N/A

Farrokhyar et al., 2014 [22]

Ontario, Canada

Pre-post study using historical data as a comparison group

Residents in 12 surgical specialties: 126 Research Seminar series (RSS) comparison group (2004-June 2009) and 81 Surgical Research Methodology (SRM), intervention (July 2009–2013)

SRM Program: a 2-year educational research curriculum with an orientation session and 12 modules. Education modules included readings, slides, assignments, and quizzes. PGY1 residents attended an interactive and progressive course on the principles of clinical epidemiology and basics of biostatistics presented over one year. PGY2 residents presented either a proposal or a completed study. Each resident received a program grade, based on quizzes, presentation of research, and class participation.

Yes

Yes (research methodologist and attending surgeons)

Increase in productivity and performance of the SRM residents compared with RSS residents.

• SRM residents had more publications than the RSS residents (mean difference 1.0 (95% CI: 0.5–1.5; p < 0.001) with an effect size of 0.26

• SRM residents had more statistically significant presentations of higher evidence than the RSS residents (52.5% vs. 29%; p < 0.005)

• Research performance improved 11.0 points (95% CI: 8.5–13.5%; p < 0.001) with an effect size of 0.51 in favor of the SRM residents

N/A

Fisher et al., 2010 [23]

South Carolina, USA

Pre-post study using historical data as a comparison group

General surgery residents

Small, inexpensive program spanning 2 years. They informed residents about faculty research, opportunities for involvement, and compiled a list of project milestones while setting semi-annual deadlines. Lectures were incorporated into the Basic Science curriculum schedule, covering the institutional review board process, case design, and statistical analysis.

No

Yes (faculty)

After the intervention there were increases in compliance, publications, and presentations.

• Pre-intervention, 2 of 16 residents were compliant; post-intervention there was 100% compliance with all aspects of the program

• Case report rates fell from 65, to 30%, to 25% by the end of year 2

• Increase in publications from 3 (pre-intervention) to 13 by year 2

• Increase in presentations from 5 (pre-intervention) to 13 by end of year 2

They indicated that their initiative was inexpensive, but did not provide specific amounts spent.

Hoedebecke et al., 2014 [24]

North Carolina, USA

Pre-post study using historical data as a comparison group

Family medicine residents: Research curriculum comparison group(2011–2012) vs. a resident led Scholarly Activity (SA) process (2012–2013)

Implemented a resident led SA intervention with Resident Research Teams (RRT) consisting of a volunteer PGY2 and a PGY3 with the greatest research experience among their peers. They included regular public reminders of submission opportunities, journal clubs, monthly meetings to discuss goals as well as to share ideas, pairing of interns/students with senior mentors with similar interest, and publically recognize scholarly accomplishments.

Yes (4 h per month, 4 week PGY-1 research rotation)

Yes (senior residents and faculty)

After the resident driven process there were increases in presentations, publications, and involvement.

• Increase in the number of presentations from 3 (2011–2012) to 28 (2012–2013): 7 regional, 10 national, and 11 international presentations

• Increased resident peer-reviewed publications from 2 (2011–2012) to 6 (2012–2013)

N/A

Kanna, et al., 2006 [25]

New York, USA

Cohort study using concurrent data as comparison groups

Internal medicine residents: 38 non-rotators comparison group and 43 research program rotators intervention (May 2004)

A two-week structured research rotation included dedicated faculty and online resources. Intensive 2-h weekly sessions on research methodology were conducted for four weeks every year. Residents were provided dedicated time to attend didactic lectures. All residents were required to participate and complete at least one scholarly activity during their training period. The research director was given protected time to administer, teach, and evaluate all research components. The research director conducted research-in-progress seminars and had daily meetings with the residents during their research rotation.

Yes (2 week research rotation with no clinical obligation)

Yes (A qualified faculty member with a Masters of Public Health was appointed as research director)

There was an increase in total research projects, letters to the editor, and publications among rotators vs. non-rotators.

• The rotators group outperformed the non-rotators group in total research projects (published & non-published) (p < 0.001) and letters to editors (p < 0.001)

• The number of residents who were able to publish among the rotators was marginally significant compared with non-rotators (p = 0.053)

• Significantly more participation by rotator residents in scholarly activities (p < 0.001)

N/A

Khurana et al., 2015 [26]

Chandigarh, India

Pre-post study using historical data as a comparison group

Neurology residents: Pre-publication comparison group (June 2010–April 2012) vs. publication rotation intervention (June 2012–September 2013)

A publication rotation was created with mandatory participation of residents. There was no formal class on research development. Residents were encouraged to analyze landmark research trials, review articles, and send Letters to the Editor (correspondence) if they found flaws or had a differing opinion.

Yes (15 day rotation per 6 months)

Yes (faculty)

The number of publications by residents increased after the research publication rotation, with a significantly greater amount in high impact journals.

• Total resident publications increased from 10 (pre-rotation period) to 27 (post-rotation period)

• 1 paper was accepted by a high-impact journal (pre-rotation period) compared to 9 papers (post-rotation period)

• The total impact factor increased from 22.68 to 116.66 (p = 0.039)

• 11 clinical images were published (post-publication rotation) compared to 1 (pre-publication rotation)

• The number of original articles produced was the same in both periods

N/A

Kohlwes, et al., 2006 [27]

California, USA

Cohort study using some historical and concurrent data as comparison groups

32 internal medicine residents from 2000 to 2004 vs. historical comparison group (1990’s), as well as residents not in the Primary Medical Education (PRIME) program

The PRIME program was a two-year outpatient based internal medicine residency track. The program accepted 8 residents per year who divided their time evenly between the standard categorical inpatient rotations and the PRIME outpatient blocks. The PRIME curriculum consisted of didactic lecture, weekly journal club, work-in-progress sessions, and active mentoring. Didactic time was divided evenly between clinical outpatient topics, behavioral medicine, and epidemiology/ research methods training. Faculty lectured, facilitated work-in-progress sessions, and had monthly meetings with residents.

Yes (2 half days of didactic time, and 1–2 half days of research time)

Yes (faculty)

Increases in resident competency and local presentations were seen after implementation of PRIME.

• The overall clinical competence scores of the 32 PRIME residents were 8.23 on a 9.0-point scale vs. 8.09 for the rest of the internal medicine program (p < 0.001)

• 22 of 32 PRIME residents (2001–2004) produced 20 original research papers published or in-press in peer-reviewed journals, 2 clinical-review papers, 1 paper currently under review, and 2 book chapters

• Resident presentations at the Rector Symposium rose from 6 to 9 per year in the 1990’s to 13–21 (2001–2004)

N/A

Kohlwes, et al., 2016 [28]

California, USA

Cross-sectional survey

71 PRIME and

98 non-PRIME (comparison) internal medicine residents from 2001 to 2010

This track immerses residents in a clinical outcomes research curriculum and provides opportunities for participants to conduct research projects. The curriculum addresses several barriers to research productivity during residency. We ensure baseline knowledge of research methods through resident participation in a Designing Clinical Research class offered in person or online. Ongoing didactic and work-in-progress sessions continue year-round on ambulatory blocks to support resident research and help ensure success.

Protected time for research occurs during the 6 months of ambulatory medicine training time scheduled for every resident during the 2nd and 3rd years of residency.

Yes (faculty)

An individualized learning pathway enables residents to successfully publish manuscripts and access mentorship.

• PRIME alumni were more likely to publish their residency research projects compared to non-PRIME alumni (64% vs. 40%, p = 0.002)

• Strongly agree to adequate access to mentors during residency for PRIME vs. non-PRIME (4.4/5 vs. 3.6/5, p = 0.001)

• 63% of PRIME alumni agreed that their research experience during residency influenced their subsequent career choice versus 46% of non-PRIME alumni (p = 0.023)

N/A

Konstantakos et al., 2010 [29]

Ohio, USA

Pre-post study using historical data as a comparison group

Orthopedic residents: Pre-intervention comparison group (September 2000–August 2005) vs. a Research Team (RT) (September 2005–November 2009)

Creation of a RT that encompassed a research director, faculty, research residents, biostatistician, and other support staff that oversaw residents’ scholarly activity during monthly meetings. Mentors met weekly with, collaborated, guided, and evaluated the resident. Throughout the academic year, all residents were provided with dedicated time to attend a lecture series on planning and conducting research, biostatistics, and critical appraisal of the medical literature taught by public health and statistics faculty. Attention was also focused on seeking external funding. A manuscript that was ready for, or submitted to, a peer-reviewed journal was a requirement for graduation.

Yes (3 months of PGY2 with 2 half days of clinical duty per week and 2–3 days of night call per month)

Yes (faculty)

Scholarly activity increased substantially from the 5-year period before the implementation of the RT to the 4-year period after initiation of the RT.

• Peer-reviewed publications on which residents were authors increased from 1 to 10 per year

• Peer-reviewed national presentations at professional meetings increased from 6 to 11 per year

• Peer-reviewed local and regional presentations increased from 2 to 4 per year

Departmental Reimbursements based on FTE:

Biomechanical engineer 30%;

Statistician 10%;

Administrator 10%;

Basic scientist 10%; Research coordinator 100%

Krueger, et al., 2017 [30]

3 residencies from unknown locations in the USA

3 nonrandomized groups: 3 orthopedic residencies with varying amounts of protected research time in each residency, retrospectively evaluated between 2007 and 2014

Peer-reviewed publications from 3 residency programs were examined retrospectively from January 2007 through December 2014. All 3 programs shared the same research requirement—completing at least 1 publishable research project during residency.

Program 1: research year is mandatory for all residents and is completed between their PGY3–4 years.

Program 2: all residents have the opportunity to volunteer for an elective research year between their PGY2–3 years.

Program 3: no residents were given protected research time during their residency

No

• Residency programs with dedicated research time did not produce significantly (p > 0.05) more, or higher quality, peer-reviewed publications than residencies without dedicated research time.

• There was no significant difference in the number of publications produced between the institutions on a per-staff (p = 0.27) or per-resident (p = 0.80) basis.

• There were no significant differences between programs in terms of the SCImago Journal Ranking for the journals containing their publications (p = 0.135).

N/A

Kurahara et al., 2012 [3]

Hawaii, USA

Pre-post study using historical data as a comparison group

Pediatric residents: 63 pre-intervention comparison group (1994–2000) and 72 Residency Research Requirement and Program (RRRP) residents (2001–2007)

RRRP focused on increased resident productivity, faculty collaboration, and seeking fellowship training. Elements included didactic lectures on research and journal clubs for critical review of the literature. There was a research week for resident presentation. Most residents conducted research during PGY2 or PGY3 when more time was available. Faculty aided by re-writing articles and in fielding arguments in the peer-reviewed journals.

No

Yes (faculty)

There was a significant increase in publications, collaboration, and scholarly activity.

• Publications increased from 2 to 26 post intervention (p < 0.001)

• Collaboration between resident co-authors increased from 5 to 31 post intervention (p < 0.001)

• 8% of residents involved with published research pre-intervention to 36% during RRRP

• Pediatric faculty involvement in resident projects increased significantly from 1 to 48 (p < 0.001)

• Pre-intervention 18% of residents who went into fellowship published vs. 4% of those who went into general pediatrics; increased post-intervention to 63% of those who went into fellowship vs. 23% of those in general pediatrics

N/A

Lohr, et al., 2006 [31]

Ohio, USA

Pre-post study using historical data as a comparison group

General surgery residents: Pre-intervention comparison group (1989–1997) vs. a Research Team (RT) (1997–2003)

Each RT consisted of a faculty mentor, senior resident, mid-level resident, and an intern. Residents maintained their team assignment throughout their 5-year appointment. An annual 6–8 week structured lecture series, including critical thinking, evaluation of the literature, statistics, guidelines for research projects, literature searches, hypothesis development, and tools for data collection. Journal clubs were held to improve resident literature reading skills, analytical review, research methods, biostatistics, and epidemiology. Support staffs such as a research director, research nurses, epidemiologist, and research specialists were available to assist the RT. Residents were expected to submit a case report and present at a national or regional meeting.

No

Yes (faculty and senior residents)

The RT increased the amount of presentations, publications, and peer-reviewed publications though none met statistical significance.

• Pre-RT, 69 presentations and 60 publications compared to 92 presentations and 77 publications after the RT

• Peer-reviewed publications were 83% pre-RT compared with 92.5% of publications post-RT

• The RT resulted in a 33% increase in presentations and a 13% increase in publications

The Institute received an endowment from Dr. E. Kenneth Hatton in 1997. However, the amount used for resident research efforts was not specified.

Löwe, et al., 2007 [32]

Heidelberg, Munich, and Tubingen, Germany

Cohort study using concurrent data as comparison groups

Internal medicine, psychotherapy and psychosomatics, psychiatry, and psychology residents: 22 non-participant comparison group and 15 Resident Training Program (RTP) participants (2005)

Structured 1-year training program with three elements: 1) provision of a methodological research knowledge within the scope of a “Clinical Research Methods” course; 2) mentorship by an experienced researcher; and 3) an individual research project. The Clinical Research Methods course had 33 weekly lectures, each lasted 90 min.

No

Yes (faculty)

Residents within the training program had increased scholarly activity, presentations, and grant applications.

• Significantly more intervention subjects were currently working on a journal article (87% vs. 36%; p = 0.003)

• RTP residents had increased presentations of scientific results at research meetings (80% vs. 41%; p = 0.04)

• Increased RTP completion of 1 original paper as coauthor (60% vs. 18%; p = 0.01)

• RTP residents completed significantly more applications for funding as co-investigators (47% vs. 5%; p = 0.004)

• Significantly more RTP grant applications were accepted for funding (33% vs. 0%; p = 0.007)

Annual research money per full professor (438,000€, 458,000€, and 484,000€,)

Macknin, et al., 2014 [33]

Ohio, USA

Cohort study using concurrent data and retrospective data: 48 research track and 74 traditional track residents

Orthopedic residents: Compared the outcomes of 74 traditional track and 48 research track training from 1987 to 2006.

Created two research tracks for orthopedic residents. Research track residents were given 1 year for full-time basic science research after their intern year. Traditional residents completed a research project during residency with significantly less protected time and no long term protected time.

Traditional track—no

Research track: 1 year of dedicated research time

Yes (orthopedic scientist)

Residents in the research track were more likely to publish during residency and throughout their careers.

• Residency publications were higher for research track than traditional residents (71% vs. 41%, p < 0.01)

• Mean publications during residency for research residents were higher than traditional (1.8 vs. 0.9 publications)

• Publications after residency were higher for research residents than traditional (7.7 vs. 4.8 publications.)

• Residents who published in residency were more likely to continue publishing in their careers compared to those who did not (75% vs. 55%, p = 0.02)

N/A

Manring, et al., 2014 [34]

Ohio, USA

Pre-post study using historical controls as a comparison group

Orthopedic residents: 42 pre-intervention (2006–2008) and 46 orthopedic residents with a multifaceted publications program (2009–2012)

Appointed a research and curriculum director to increase output of research by requiring: 1) preparation of a review article or systematic review; 2) presentation of a clinical or laboratory research project; 3) preparation and submission of a manuscript based on a research project to a peer-reviewed journal; and 4) mentoring of a junior resident on research. A research editor assisted with publication and manuscript production. Faculty members mentored and delivered lectures on research methodology, statistical analysis, and regulations. Regular meetings were established to monitor topic developments, presentations, findings, and submissions to granting agencies that focus on orthopedic resident research.

No

Yes (faculty and senior residents)

There was a rise in authorship, publications, and presentations among residents.

• Resident publications in peer-reviewed journals increased from 6 (2009) to 53 (2012)

• Resident presentations at national or international conferences increased from 15 (2008–2009) to 24 (2009–2010), and were 19 (2011–2012)

Metropolitan area technical editor described as $50,000 to $60,000 per year.

Mayo, et al., 2015 [35]

Texas, USA

Pre-post study using historical data as a comparison group

Gastroenterology fellows: 29 Pre-program comparison group (2001–2007) and 43 research program intervention (2008–2014)

Creation of a structured research program that emphasized clear expectations, protected time, mentorship and oversight, support via a statistician, an educational curriculum, tracking of accomplishments, and accountability. They instituted a fellowship research committee that oversaw and ensured residents were completing research in a timely manner. Fellows met with the committee semi-annually. Fellows met with faculty mentors bimonthly. The educational curriculum consisted of online modules and 6 specific lectures:

1) How to select a mentor; 2) How to select a project; 3) How to get your article published; 4) Statistics for the trainee; 5) Database searches and electronic journals; and 6) Use of reference management software

Yes (1 month in year 1, 3 months in year 2, and 2 months in year 3)

Yes (faculty)

There was an increase in publication, scholarly activity, and pursuit of an academic career.

• Trainees presenting original research at scientific meetings or publishing their work in peer-reviewed journals increased from 33 to 100%

• Fellows meeting the scholarship requirement increased from 20 to 60% and by the 3rd year, the research metric was achieved in 100% of fellows

• The number pursuing careers in academic medicine increased approximately 3-fold, from 14 to 51%

Authors funded 10% of the statistician’s effort or if they were funded they could pay an hourly rate.

Mills, et al., 2011 [7]

North Carolina, USA

Retrospective cohort study using concurrent data as comparison groups

Pediatric residents and fellows: 295 non-exposed comparison group and 232 exposed to an Evening of Scholarship (EOS) (1985–2007)

A pediatric program created a voluntary EOS that allowed residents and fellows to present their research on an annual basis to the department. They sought to measure the future publication rate of residents who engaged with EOS against those who did not participate (1985–2007).

No

No

Residents and fellows involved with EOS were more likely to publish in the future than those that were not engaged.

• 69% of EOS residents had publications after graduation versus 34% of nonparticipants (p < 0.001)

• Participants in EOS were more likely to have previously published (31% vs. 15%; p < 0.001)

• EOS residents had more publications after training (p < 0.001)

This project was supported through grant funds. However, amounts used toward resident research efforts were not specified.

Papasavas, et al., 2013 [36]

Connecticut, USA

Pre-post study using historical data as a comparison group

General surgery residents: 60 pre-intervention comparison group (July 2008 to June 2010) and 58 research requirement residents (July 2010–June 2012)

Research program consisted of a research curriculum, an annual research day, research mentors, project repository, statistical support, a Director of Research, and data base mining. Monthly meetings and lectures incorporated into the overall resident core curriculum. They invited IRB staff; statisticians; and senior scientists involved in basic science, translational, and outcomes research to lecture residents. During these meetings, there were opportunities for the residents to discuss the design of their research project and get feedback from the faculty and fellow residents There was a requirement to submit an abstract 30 days before a meeting with specification similar to national meetings regarding abstract structure and word limit. There was also an alternate 2-year research fellowship available at the end of PGY2.

No

Yes (faculty)

With the creation of a research requirement there was an increase in poster and podium presentations.

• Increase from 9 of 60 (15%) residents with a podium or poster presentation to 23 of 58 (40%) (p < 0.01)

• Significant increase in the proportion of podium presentations at national/international vs. regional meetings (p < 0.01)

• Increase from 14 residents producing 31 publications in peer-reviewed journals to 17 residents producing 32 publications

• 88% of 58 podium and poster presentations post-intervention originated from residents who participated in the research requirement

• Residents in the 2-year research track produced significantly higher proportion of publications (62% vs. 19%; p < 0.05)

Various departments of surgery covered the expense of statistical support. However, amounts used toward resident research efforts were not specified.

Penrose, et al., 2012 [37]

Texas, USA

Pre-post study using historical data as a comparison group

Obstetrics and gynecology residents (12 total residents in residency): Pre-intervention comparison group (2007–2008) vs. Baby Steps Program (BPS) (2008–2010)

An obstetrics and gynecology program added dedicated research staff to facilitate and coordinate resident research projects, and support clinical faculty in research activities. Faculty concentrated their efforts on developing research ideas and mentoring resident researchers with the assistance of the post-doctoral researcher to coordinate research efforts.

No

Yes (faculty and research coordinator)

• Resident presentations rose from 2 regional/national to 8 regional and 4 national presentations

• 8 of 12 clinical faculty members were engaged as mentors in resident research compared with only 3 in past years

All faculty salaries are funded through

clinical efforts, not research dollars. The postdoctoral researcher position was created and filled in August 2009 using clinically generated Funds. However, amounts used toward resident research efforts were not specified.

Robbins, et al., 2013 [38]

New York, USA

Pre-post study using historical data as a comparison group

Orthopedic residents: 75 pre-curriculum comparison group (1998–2006) and 32 research curriculum intervention (2007–2010)

Structured program included research milestones for each training year, a built-in support structure, use of an accredited bio-skills laboratory, mentoring by National Institutes of Health–funded scientists, and protected time to engage in required research or prepare scholarly peer-reviewed publications. 8 h of lecture per year. Topics included research design, navigating the IRB process, critical appraisal, and basic research methods (i.e., statistical design). Residents had the option to pursue a research year free from clinical training to gain additional academic and/or research experience.

Yes (6–7 weeks or 960 h per year)

Yes (faculty and research coordinator)

• The total amount of grants awarded increased from $15,000 for eight 2007 graduates to $380,000 for nine 2010 graduates

• Residents began to submit more than one research proposal, peaking in 2010 with 9 graduates submitting 17 applications

• The twelve 2005 graduates had a total of 16 publications from 2000 through 2006, whereas the nine 2010 graduates published 84 papers from 2005 through 2011

• 162 publications from the 1998–2006 graduates (9 years) and 341 publications from the 2007–2010 graduates (4 years)

The costs per year included $19,000 (0.3 FTE) for an academic research coordinator; $16,000 for resident travel to professional meetings; reimbursement for 213 faculty hours; and funding for resident salaries while on the research rotation, paid by the hospital budget.

Robbins, 2017 [39]

New York, USA

Pre-post study using historical data as a comparison group

Neurology residents: 53 pre-program (2005–2009) vs. 57 post-program (2011–2015)

Components included an expanded journal club led by 2 investigators during which resident projects were discussed in workshop form, guided mentorship provided, a required grand rounds platform presentation before graduation, and the presentation of annual awards for the most scholarly and seminal research findings, as judged by a faculty awards committee. Required the production of peer-reviewed publications, presentations at scientific meetings, and authored book chapters or textbooks. The program was formally administered by an associate residency program director.

Could use electives to complete research project

Yes (faculty)

Research outcomes increased after the program was implemented.

• Percentage of first-authored abstract presentation or publication of residents increased from 30.2 to 71.9% (p < 0.0001)

• Abstracts per resident increased from 0.15 +/− 0.41 to 1.26 +/− 1.41 (p < 0.001) after intervention

• Peer-reviewed publications did not show a statistically significant increase pre- to post-program (p = 0.36)

N/A

Roth, et al., 2006 [40]

Alberta, Canada

Pre-post study using historical data as a comparison group

Pediatrics residents: 20 pre-curriculum comparison group (2002–2003) and 23 research curriculum intervention (2004–2005)

Research curriculum had three main components: 1) the resident research project; 2) a supportive training environment; and 3) accessible research funding. Twenty, 75-min academic sessions addressed the steps of a research protocol. Residents directed monthly journal clubs. During research blocks residents did not have daytime clinical duties and had reduced nighttime call responsibilities. Mentors guided the resident through study design, conduct and analysis, and obtaining funding. Medical librarians assisted with literature searches and departmental biostatisticians provided consultation to residents. Residents were required to present their work at least once during residency.

Yes (12 weeks during PGY1 and up to 8 weeks PGY2-PGY4)

Yes (faculty)

There was an increase in all measures from the comparison group to the intervention, but none of the observed differences were statistically significant.

• Research articles increased from 3 to 5

• Research grants increased from 3 to 7

• Conference presentations increased from 6 to 9

Departmental funds allocated to resident research grants include seed money ($250 CDN) allocated on a non-competitive basis to offset the administrative costs associated with preparation of a research proposal.

Residents could receive $2000 CDN per project per year to present at meetings.

Rothberg, et al., 2014 [41]

Massachusetts, USA

Pre-post study using historical data as a comparison group

Internal medicine residents: Pre-implementation comparison group (2001–2006) vs. implementation of a Resident Research Program (2006–2012)

The program consisted of evidence-based medicine training, 4 two-hour interactive workshops, to stimulate interest in research. Structural changes were made to support resident’s conduct of research including protected time during ambulatory blocks, a research assistant who aided with tasks such as institutional review board applications and data entry, a research nurse to assist with data collection, easily accessible biostatistical support, and a resident research director to provide mentorship.

Yes (1 day per week PGY2)

Yes (faculty and research director)

• Resident publications increased from 3 to 58 (p < 0.001)

• Residents accepted to subspecialty fellowships increased from 28 to 50 (33 to 49%; p = 0.04)

• Original research increased from 2 to 28

• Residents first authors increased from 1 to 33

0.25 FTE for research director; 0.5–1.0 FTE for master’s-level statistician; 10 h per week for research nurse; $750/ resident who presents at a national meeting; $750 in prizes; and $175 for catering

Ruiz, et al., 2011 [42]

North Carolina, USA

Pre-post study using historical data as a comparison group

Internal medicine residents: Pre-intervention comparison group (July 2006–June 2007) vs. Research Curriculum intervention (July 2007–June 2009)

Developed a comprehensive 3-year curriculum, appointed a chief resident for research and a faculty research director to coordinate all resident research activities. All residents were involved in a 3-year evidence-based medicine curriculum that covered 12 to 14 topics in research methodology, statistical methods, research design, and manuscript preparation. There were monthly research forums to discuss projects and future directives. Residents were required to submit an abstract and present a poster of their work at the annual departmental research day.

Yes (1–3 months of research elective time per year; not to exceed a total of 3 months)

Yes (faculty and research chief resident)

• Graduates with a peer-reviewed publication increased (7% vs. 32%; p = 0.04); publications increased from 7 (comparison group) to 11 (year 1) to 15 (year 2)

• Graduates with a presentation at a national meeting increased from 5 to 10 (4% vs. 29%; p = 0.02)

• Regional presentations increased from 2 to 9

The department committed to providing funding

for poster printing and meeting-related expenses, including

costs of transportation, lodging, and meals for regional and national meetings when not provided by the sponsoring subspecialty section or a third party. However, amounts used toward resident research efforts were not specified.

Sakai, et al., 2014 [43]

Pennsylvania, USA

Pre-post study using historical data and rank-to-match analysis as a comparison group

Anesthesiology residents: 54 Pre-initiative comparison group (2003–2006) and 65 post-initiative intervention (2007–2011)

An annual research introductory lecture was given (1-h lecture of basic grantsmanship, steps in research activity, and introduction of potential faculty research mentors), deadlines for abstracts and meetings were presented, scholarly achievements were announced on the department website, the Resident Research Rotation (RRR) director was appointed, a 90-min research problem–based learning discussion was developed, and an annual Trainees Research Day was formed. The following initiatives were implemented only for the PGY4 elective RRR: a formal application process for acceptance, mandated attendance at a weekly research meeting with the rotation director where weekly milestones were presented, and submission of an abstract to local and state resident research competition was strongly recommended.

Yes (6 months for senior residents)

Yes (faculty and research director)

• Resident peer-reviewed publications went from 16 pre- to 41 post-intervention

• The RRR residents (n = 25) published 36 articles whereas the non-RRR residents (n = 94) published 33 articles

• The RRR residents authored more original articles than the non-RRR residents (83.3% vs 33.3%, p < 0.0001)

• A greater number of post-initiative residents entered fellowships (42.6% vs. 70.8%; p = 0.002)

N/A

Schnapp, et al., 2009 [44]

Washington, USA

Pre-post study using historical controls as a comparison group

Pulmonary and critical care medicine (PCCM) fellows: Pre-intervention comparison group (1995–2000) vs. Translational Research Training Program (TRTP) intervention (2001–2006)

Increased collaboration between clinical and basic science researchers. Research training in the primary research discipline of a trainee (basic science or clinical), cross-training in the alternate research discipline, development of a research project that included a translational research component, and enhancement of the research environment to emphasize translational research. All fellows are required to complete a research project under the direction of a faculty mentor and the mentoring committee. There was a 9-week course that covered basic research methods, a laboratory workshop, and statistical approaches. Modified journal clubs evaluated literature from a basic science and clinical standpoint.

No

Yes (faculty)

• The average number of authors per manuscript increased from 3.79 in 1995 to 5.54 in 2006 (p < 0.05)

• The total number of publications from PCCM Division members increased from 43 (1995) to 52 (2000) and 83 (2006)

• Average number articles published per fellow prior to institution of TRTP (1995–2000) was 2.5, and after institution of TRTP (2001–2006) it increased to 4.7

• Publications that included PCCM Division members from more than one training track increased from 8% in 2000 to 32% in 2006 (p < 0.005)

• The percent of translational articles was similar in 1995 and 2000 (9 and 11%, respectively), but increased to 22% in 2006 (p > 0.05)

N/A

Seehusen, et al., 2009 [45]

Georgia. USA

Pre-post study using historical data as a comparison group

Family medicine residents: Pre-point system comparison group (2002-June 2006) vs. Point System intervention (July 2006–July2007)

Research point system tabulated resident’s research productivity. Residents had to accumulate 10 scholarly activity points. The research director, program director, and faculty mentor determined point allotments. The most efficient way to accumulate points was through a research project and subsequent presentation. Residents were allowed to collaborate with one another or faculty on projects.

No

Yes (faculty)

• 4 peer-reviewed medical journal publications pre-intervention compared to 4 during the post-intervention period

• Regional, national, and/or international podium/poster presentations increased from 5 to 8

• Increase from 1 book section or chapter to 5 with resident authors

N/A

Torres, et al., 2015 [46]

Texas, USA

Pre-post study using historical data as a comparison group

Orthopedic residents: 24 pre-implementation comparison group (2001–2006) and 27 dedicated resident research program intervention (2007–2012)

The dedicated resident research program included: 1) the requirement for the number of original research projects per resident increased from one to two; 2) each project required at least one faculty mentor; 3) a project proposal had to be reviewed by the newly established departmental research committee and revised as needed; 4) the resident presented the project proposal to the entire departmental faculty for majority approval before it was accepted as an official project; 5) once the project was approved, the research committee monitored the project’s progress; and 6) project completion was achieved by manuscript submission for peer-reviewed publication.

No

Yes (faculty)

• Post-intervention residents published more papers during residency than the comparison group [1.15 vs 0.79 publications per resident; 95% CI (0.05,0.93); p = 0.047]

• Journal impact factor increased after implementation [1.25 versus 0.55 per resident; 95% CI (0.2,1.18); p = 0.005]

• Trainees after implementation more often continued education in a subspecialty fellowship (81.5% versus 45.8%; p = 0.008)

N/A