Graduate medical education scholarly activities initiatives: a systematic review and meta-analysis

Background According to the Accreditation Council for Graduate Medical Education residents “should participate in scholarly activity.” The development of a sustainable, successful resident scholarship program is a difficult task faced by graduate medical education leadership. Methods A medical librarian conducted a systematic literature search for English language articles published on scholarly activities initiatives in Graduate Medical Education (GME) between January 2003 and March 31 2017. Inclusion criteria included implementing a graduate medical education research curriculum or initiative designed to enhance intern, resident, or fellow scholarly activities using a control or comparison group. We defined major outcomes as increases in publications or presentations. Random effects meta-analysis was used to compare the rate of publications before and after implementation of curriculum or initiative. Results We identified 32 relevant articles. Twenty-nine (91%) reported on resident publications, with 35% (10/29) reporting statistically significant increases. Fifteen articles (47%) reported on regional, national, or international presentations, with only 13% (2/15) reporting a statistically significant increase in productivity. Nineteen studies were eligible for inclusion in the meta-analysis; for these studies, the post-initiative publication rate was estimated to be 2.6 times the pre-intervention rate (95% CI: 1.6 to 4.3; p < 0.001). Conclusions Our systematic review identified 32 articles describing curricula and initiatives used by GME programs to increase scholarly activity. The three most frequently reported initiatives were mentors (88%), curriculum (59%), and protected time (59%). Although no specific strategy was identified as paramount to improved productivity, meta-analysis revealed that the publication rate was significantly higher following the implementation of an initiative. Thus, we conclude that a culture of emphasis on resident scholarship is the most important step. We call for well-designed research studies with control or comparison groups and a power analysis focused on identifying best practices for future scholarly activities curricula and initiatives. Electronic supplementary material The online version of this article (10.1186/s12909-018-1407-8) contains supplementary material, which is available to authorized users.


Background
The Accreditation Council for Graduate Medical Education (ACGME) mandates that residents "should participate in scholarly activity" and that " [t]he sponsoring institution and program should allocate adequate educational resources to facilitate resident involvement in scholarly activities" [1]. Such broadly-defined requirements leave individual residencies to interpret and execute scholarly activities within their program in varying ways. However, this can lead to a wide diversity in residency curricula, programs, outcomes, and experiences between residency programs. As such, it is important to identify high-yield practices of successful programs so that they may be tailored to other residency programs.
It has been shown that increased research exposure and experience leads to increased fellowship acceptance and opportunities [2,3]. Providing residents with the tools to succeed in their scholarly activities promotes the long-term benefit of producing well-rounded clinicians. Even residents who choose not to pursue academic careers will benefit from an improved ability to critically assess medical literature [4].
Given the stated importance of resident scholarly activities, medical educators are faced with the difficult task of implementing curricula and initiatives supporting residents through their scholarly experience. The availability and promotion of scholarship vastly differs between residency programs, even within the same subspecialty [5][6][7]. Differing opportunities for time allotted, faculty involvement, and relevant curricula can make for highly varied experiences and outcomes for residents. Therefore, it is important to describe the efforts of successful programs.
Our initial review of the literature, identified one systematic review published in 2003, focused on resident research curricula only [8]. Hebert and colleagues identified 41 articles and summarized instructional methods, goals, and objectives, as well as obstacles encountered in implementing resident research curricula [8]. They concluded that the lack of detailed developmental information and meaningful evaluations hinders educators interested in adopting a new research curriculum [8]. We set out to conduct a systematic review of the literature to extend these results beyond 2003 and to include all initiatives designed to increase intern, resident, or fellow scholarly activity.

Literature search
A medical librarian (ILV), who has participated in multiple systematic reviews, conducted a comprehensive literature search for English language articles published on research curricula and scholarly activities initiatives in Graduate Medical Education (GME) between January 1, 2003 and March 31, 2017 in PubMed (National Library of Medicine), EMBASE (Elsevier), and Scopus (Elsevier) databases. We chose relevant controlled vocabulary and keywords to locate GME articles focused on scholarly activities and research curricula (Additional file 1).
From the searches, 2980 unique articles were obtained. Two anesthesiology-trained authors (WW and PK) independently reviewed all titles and abstracts when available. The percent agreement on initial independent selection of articles for further review was 98.9%. Inter-rater reliability using Cohen's Kappa was κ = 0.897, p < 0.001.
When either reviewer selected an article, the full text was ordered for further review by trained research assistants (JM and AM). Research assistants also checked the reference sections of all included articles to identify other relevant studies. Using this strategy, 197 articles were obtained and reviewed for possible inclusion. All (197) articles were independently reviewed by the trained research assistants (JM and AM) to determine eligibility for inclusion. In rare cases where a disagreement occurred, the full text was reviewed by a third team member (LAR). This was followed by a team discussion of the article, where a final inclusion decision was made. After this review process, 32 articles were identified that met our protocol criteria (Fig. 1).

Inclusion and exclusion criteria
We developed a comprehensive systematic review protocol containing operational definitions and inclusion/exclusion criteria. The operational definitions included: 1) Scholarly Activities Curriculum: defined as instruction, teaching, didactic, seminars, or workshops developed and implemented with the goal of increasing scholarly or research outcomes/productivity; 2) Initiative: defined as any activity, tactic, or intervention (e.g., role models, mentors, protected time, journal club, or project funding) implemented to improve or increase scholarly or research outcomes/ productivity; 3) Major scholarly activities outcomes/productivity: defined as regional, national, or international presentation(s) and/or publication(s); 4) Participants: defined as interns, residents, or fellows in graduate medical education programs; 5) Strategies: defined as procedures or processes that the author describes as being imperative, key, or a major contributor to their study's success; 6) Barriers: defined as obstacles or problems the author(s) described as being a hindrance or impediment to their study's success.
Articles meeting the following criteria were eligible for review: published between January 1, 2003 and March 31, 2017; English language; studied interns, residents, or fellows of any graduate medical education discipline; implemented a curriculum or activity designed to increase scholarly activities outcomes/productivity; and presented results using a control or comparison group. Exclusion criteria included: letters to the editor, commentaries, editorials, or newsletter articles; articles that did not include a description of implementation with outcomes data; or did not measure presentations and/or publications.

Abstraction process
Two trained reviewers (JM and AM) individually evaluated all selected articles to ascertain the study's purpose, quality, and results. Information pertinent to the systematic review was independently abstracted, organized, and added to a spreadsheet for further assessment. Monthly meetings were held with a separate author (LAR) to review, revise, and validate the extracted data. During the course of the meetings a finalized document of abstractions was created. All abstraction disagreements were minor and were resolved during discussion between the reviewers.

Quality assessment
We used the Medical Education Research Study Quality Instrument (MERSQI) developed by Reed et al. [9] to assess article quality. It is an 18-point, 6-domain instrument designed specifically for medical education research. The 6 domains are study design, sampling, type of data, validity of assessment instruments' scores, data analysis, and outcomes evaluated. Since its introduction in 2007, multiple studies have shown evidence of its validity and reliability [9][10][11]. As described in its original use [9], the total MERSQI score was calculated as the percentage of total achievable points. This percentage was then adjusted to a standard denominator of 18 to allow for comparison of MERSQI scores across studies. One item on the MERSQI rates "type of data." The scoring choices are "subjective, assessment by study participant" = 1 and "objective measurement" = 3. If a study measured both subjective and objective data, it was given 3 points for objective data.

Types of data reported
The authors abstracted the following data from the selected articles: first author's last name, year published, study location; study design, sample, and participants; program/interventions; curricula (yes/no); protected time for scholarship (yes/no); mentors (yes/no); support (yes/ no); funding (yes/no); journal club (yes/no); grant writing assistance/guidance (yes/no); mandatory requirement, such as participation, attendance, completion of project, involvement in activities (yes/no); major outcomes; and price/cost. The authors defined major outcomes as publications authored or co-authored by participants or presentations at regional, national or international conferences made by participants. In addition, the authors compiled a list of strategies and barriers mentioned in the abstract or discussion sections of the included studies.

Qualitative analysis of barriers and strategies
Barriers and strategies mentioned in either the abstract or discussion sections of the included articles were abstracted and listed in phrase format. Reviewers (JM and AM) independently created their respective lists. Reviewers then met to discuss and come to consensus on final, comprehensive lists: one for barriers and a second for strategies.
Reviewers (JM and AM) then used an immersive iterative process of content analysis [12] to identify themes and create relevant category labels. Another author (LAR) and JM then used a second iterative process to finalize category and subcategory labels.

Statistical analysis
Descriptive statistics were used to report counts and percentages of initiatives. For each program, a publication rate, defined as the number of publications per participant per year, was estimated for the pre-and post-initiative periods. A publication rate ratio (PRR) was then calculated by dividing the post-initiative publication rate by the pre-initiative rate. A PRR greater than 1 indicates that the publication rate increased in the post-initiative period. Random effects meta-analysis [13] was then used to obtain a pooled estimate of the PRR. Meta-regression was also used to assess whether any specific initiatives were significantly associated with the publication rate.
Possible publication bias was assessed using the Egger test and with funnel plots [14]. A two-tailed p-value < 0.05 was considered statistically significant. All statistical analyses were performed by using R statistical software, version 3.0 [15,16].
Under half of the studies with curriculum (7/19, 37%) reported using needs assessment prior to the development of their research curriculum. Two conducted surveys [25], and one program held a faculty retreat [35] to construct their needs assessment. Another used both interviews and a committee [40]. The remaining 4 studies (21%) used a committee or team that discussed and developed the curriculum [17,34,38,42]. Ten (10/19, 53%) studies evaluated the curriculum they implemented via surveys [17,18,22,25,28,32,33,35,44], quizzes/exams [22,32] and/or interviews [17,35,38]. All but one of these articles reported results of the evaluation [38]. All surveys and Yes (faculty) The number of publications by residents increased after the research publication rotation, with a significantly greater amount in high impact journals.
• Total resident publications increased from 10 (prerotation period) to 27 (postrotation period) • 1 paper was accepted by a high-impact journal (prerotation period) compared to 9 papers (post-rotation period) • The total impact factor increased from 22.68 to 116.66 (p = 0.039) • 11 clinical images were published (post-publication rotation) compared to 1 (pre-publication rotation) • The dedicated resident research program included: 1) the requirement for the number of original research projects per resident increased from one to two; 2) each project required at least one faculty mentor; 3) a project proposal had to be reviewed by the newly established departmental research committee and revised as needed; 4) the resident presented the project proposal to the entire departmental faculty for majority approval before it was accepted as an official project; 5) once the project was approved, the research committee monitored the project's progress; and 6) project completion was achieved by manuscript submission for peer-reviewed publication.  [17,22,25,28,32,35,44], except one, which had mixed results [18]. The two studies testing knowledge [22,32] compared results of a control group to an intervention group (received curriculum) showing statistically significant results in favor of the curriculum. In addition, one of these studies used a pre-post design as well as the comparison to a control group [32]. Most included articles (31/32, 97%) used multiple interventions with the goal of increasing scholarly productivity [3,. The one article that did not use multiple interventions used an annual research day to stimulate an increase in research productivity [7]. The number of interventions ranged from 1 [7] to 8 [40] [42], (mean 4.0 ± 1.7; median 4).

Meta-analysis
Nineteen of the 32 articles (59%) provided enough detail to calculate a publication rate ratio [3, 7, 21, 23-25, 28-30, 32-34, 36, 38, 40-43, 46]. Two studies reported the percentage of participants who published, but not the overall number of publications; for these studies we assumed one publication per participant [7,33]. The PRR for these studies ranged from 0.6 to 25, with eight studies having PRR significantly greater than one, indicating that for these programs, the post-intervention publication rates were significantly higher than in the pre-intervention period [21,23,24,28,[40][41][42][43]. Overall, the publication rate was significantly higher following implementation of initiatives (p < 0.001; Fig. 2); we estimate that the post-initiative publication rate was 2.6 times (95% CI: 1.6 to 4.3 times) the pre-intervention rate, or a 160% increase.
Of the eight initiatives (mentors, curriculum, protected time, a mandatory component, journal club, grant writing guidance/assistance, funding, and support staff) identified in the included studies, mentoring, curriculum, and protected time were offered most frequently. For this reason, these three initiatives were selected for inclusion in the meta-regression to compare the PRR for programs providing those initiatives to those that did not. Sixteen of the 19 (84%) included programs provided mentors [3, 23-25, 28, 29, 32-34, 36, 38, 40-43, 46]. For programs that provided mentors, the post-initiative publication rate was estimated to be 3.2 times (95% CI: 1.92 to 5.23 times; p < 0.001) the pre-initiative publication rate, while the pre-and postinitiative publication rates for programs not providing mentors did not significantly differ (p > 0.20) (Fig. 3). However, the difference in the publication rate ratios for these two groups did not reach statistical significance (p = 0.10). Programs that provided curriculum (12/19 or 63% of included studies) or protected time (63% of included studies) also did not have significantly higher PRR than programs that did not use these initiatives (p > 0.20, for both).

Publication Bias
The Egger test did not detect any evidence of publication bias (p > 0.20). However these results should be interpreted with caution, as it is likely that residency programs that did not see improvements in productivity, or saw a decline in productivity, might be less inclined to publish their data.

Discussion
This systematic review of GME scholarly activities initiatives identified 32 relevant articles published after 2006. All included articles demonstrated improvements in resident productivity in regards to publications or presentations. Unfortunately, many of these articles (66%) failed to specify whether or not their improvements were statistically significant.
Most included articles used multiple interventions (97%), with providing mentors reported most often (88%). Research curricula or protected time were both provided in over half of studies (59%). However, there was wide variability in both curricula and protected time provided.
Programs with curricula included workshops, a lecture series, or research seminars, with much variability in length and content of sessions. The most frequently taught topics were research design (75%) and statistics (69%). The remaining topics were covered by less than half of programs with research curricula. Programs providing protected time varied from a 2-week rotation to a year-long rotation. This wide range in time commitment dedicated to scholarly activities curriculum and protected time to complete projects makes it impossible to make direct comparisons between programs.
A systematic review of research curricula published over a decade ago concluded that "successful educational interventions should incorporate needs assessment, clearly defined learning objectives, and evaluation methods" [8]. ,p61 Despite this call published in 2003, we found little progress made in the inclusion of needs assessment, objectives, and curricular evaluation. Hebert and colleagues [8] found that only 27% of included articles had a needs assessment. In our review, we found that this percent had increased to 37%, which is an improvement but still falls short of ideal. Likewise, we found a lack of curricular evaluation, with only half (53%) providing evaluation data. Similar to our findings of 11%, Hebert et al. [8], found that 12% of studies used an objective pre-post knowledge test.
Our primary outcomes were presentations and publications. Only 2 of the included studies reported statistically significant increases in presentations, while 10 (36%) reported statistically significant increases in publications. All studies with a research curriculum reported increases in presentations and/or publications. However, only 32% of studies with curricula reported statistically significant improvements. This is possibly due to small sample sizes and resulting lack of power to detect differences. In the future, more robust study designs with larger sample sizes are needed to definitively assess the importance of inclusion of a research curriculum.
Of the remaining interventions, funding was reported in a quarter of the studies. However, the amount of funding varied widely across studies. In fact, all initiative interventions varied a great deal across studies, including length of curriculum, amount of protected time and mentoring provided.
We identified nine categories of strategies commonly reported as important (Table 2). Structured research curricula, faculty mentorship, and providing departmental infrastructure and support were the most commonly cited strategies. Strategies that we identified were similar to those identified by Hebert et al. [8] in their 2003 systematic review of residency research curricula. In their article, they described common curriculum components such as educational goals and objectives, lectures, seminars or small groups; role models; and research mentors.
In addition, we identified barriers to research output noted in the reviewed articles. The most frequently cited obstacle when implementing changes to resident research was lack of time, due to clinical responsibility or the importance of educational curriculum besides research. Difficulties in providing clinical research mentors to residents, lack of resident interest in research, departmental funding towards research and challenges in providing adequate training and support were also identified in our review. Hebert et al. [8] identified obstacles encountered based on learner (e.g., resident resistance, lack of motivation), faculty (e.g., resistance, time/intensity demands, lack of motivation) and institutional (e.g., lack of time, financial barriers, lack of critical support staff). Our review yielded much similar results of institutional barriers, yet very few articles mentioned learner or faculty barriers as described by Hebert. This may reflect a change in academic department attitudes towards research.
Hebert and collegues [8] noted that many articles failed to provide descriptions of feasibility, sustainability, or cost. We found 15 (47%) articles had some mention of cost and/or feasibility of their research initiatives. However, many provided vague statements that would yield very little concrete assistance in determining actual cost or feasibility and no article directly addressed sustainability.
It appears that after implementing research initiatives in a residency, a majority of programs saw an increase in resident publications or presentations. However, we were unable to identify a particular intervention that was associated with statistically significant improvements. Whether it was faculty mentor participation, scheduled research instruction, or other initiative, it appears that any departmental dedication to resident research may increase scholarly productivity. It may be that as the overall culture within a residency moves toward supporting resident research and scholarly activity, resident publications and presentations will increase.

Limitations
As recently noted in an editorial, medical education reviews are difficult to conduct [47]. Many aspects of published medical education research vary, including study design, operational definitions, educational interventions, subjects, sample size, and outcome measures. All of these differences prevent easy aggregation of data [47]. Despite these limitations, we were able to identify 19 (59%) studies with enough detail to include in a meta-analysis of publication rate ratios. As with any systematic review, the results are limited by the search strategy and methods used. We addressed these issues by developing a detailed protocol with operational definitions and by using multiple trained reviewers throughout the study process. Our search included 3 databases and was conducted by an experienced medical librarian. In addition, the reference sections of all included articles were reviewed for possible additional articles. Although all of these strategies improve the quality of our systematic review, we may have missed some relevant articles.
As with all systematic reviews of the literature, there exists the possibility of a publication bias against negative studies, resulting in few studies published that did not demonstrate improvements. In addition, we were only able to analyze interventions that were themselves published. There likely exist many residency programs throughout the country that have implemented or updated their resident research/scholarly activity initiatives while not explicitly publishing data on the changes and their results.
Although our meta-analysis concluded that the postinitiative publication rate was significantly higher than the pre-initiative publication rate, this result should be interpreted with caution. The design of the studies included in the analysis were varied, and it is possible that changes in the publication rate may have been due to factors outside of the implementation of an initiative. Meta-analysis results can also be sensitive to publication bias, which is likely to be present for this study.

Recommendations
When implementing or updating a resident research curriculum it is important to consider all aspects of curriculum development, including conducting a needs assessment, developing goals and objectives, and designing a robust mechanism for curriculum assessment. Further, education leadership should consider using freely available, peer-reviewed, online resources, such as MedEdPORTAL. A brief search conducted by our team yielded two teaching resources devoted to research curricula [48,49] and another two focused on scholarly activities and research mentor resources [50,51]. There are likely many more such resources available.
It is vital to address barriers to outcomes early and often, to avoid stagnation or poor utilization of valuable resources. In addition, future studies should provide data on cost, feasibility, and sustainability of initiatives used to improve resident scholarly activities.

Conclusions
While specific interventions designed to improve resident scholarly activity cannot be individually tied to an increase in resident productivity, it appears that a culture of research emphasis is likely the most important factor in leading to improvements in resident research productivity. However, we call for prospective studies that include a power analysis; a control or comparison group; well defined, quantifiable parameters; and high-quality design to identify best practices for future scholarly activity initiatives. Without these studies it remains difficult for residency education leadership to design cost-effective interventions proven to increase resident scholarly activities (e.g., local, regional, and national presentations and peerreviewed publications). the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. JM made substantial contributions to analysis and interpretation of data; was involved in drafting the manuscript and revising it critically for important intellectual content; gave final approval of the version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. PK made substantial contributions to conception and design, and analysis and interpretation of data; was involved in drafting the manuscript and revising it critically for important intellectual content; gave final approval of the version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. ILV made substantial contributions to conception and design; was involved in drafting the manuscript and revising it critically for important intellectual content; gave final approval of the version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. CJM made substantial contributions to analysis and interpretation of data; was involved in drafting the manuscript and revising it critically for important intellectual content; gave final approval of the version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. AHZM made substantial contributions to analysis and interpretation of data; was involved in drafting the manuscript and revising it critically for important intellectual content; gave final approval of the version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. LAR made substantial contributions to conception and design, and analysis and interpretation of data; was involved in drafting the manuscript and revising it critically for important intellectual content; gave final approval of the version to be published; and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. All authors have read and approved the manuscript.