Making It Stick: Use of Active Learning Strategies in Continuing Medical Education Running Head: Active Learning Strategies in Continuing Medical Education


 BackgroundDespite the known benefits of active learning (AL), the predominate educational format in higher education is the lecture. The reasons for slow adaptation of AL in medical education are not well understood. The purpose of this survey was to determine knowledge, usage, attitudes, and barriers to AL use in academic Continuing Medical Education (CME).MethodA 20-item questionnaire was developed and sent with a link to an online questionnaire to the Society of Academic Continuing Medical Education (SACME) listserv of ~350 professionals representing academic medical centers, teaching hospitals, and medical specialty societies in the United States (U.S.) and Canada. Responses were collected with SurveyMonkey® from October-November, 2019. Data were analyzed using the SPSS®.ResultsResponses from 146 SACME members in 91 CME units yielded a ~42% survey response rate. Many respondents reported their self-perceived knowledge of AL as high. Advanced training (e.g., certificate, Master of Education degree) was positively correlated with AL knowledge. AL methods were reportedly used in half of the CME activities in the majority (80%) of institutions. Higher levels of self-perceived knowledge were correlated with an increased percentage of AL-related CME activities. Commonly perceived barriers to use of AL were presenters’ lack of familiarity and a need for more time-consuming preparation. Conclusions﻿More efforts are needed to increase innovation and incorporate evidence-based AL strategies in medical education, especially to foster learner engagement, critical thinking, and problem-solving ability.


Background
Teacher-centered lectures continue to be largely utilized in higher education overall. However, leaders in medical education have challenged the lecture format because active learning (AL) strategies have been shown to promote better retention and application of new knowledge than listening to passive, information-only imparting lectures [1,2,3]. Fostering curiosity and discovery while critically evaluating and discovering viable ways to address gaps in professional practice, change practice, and improve patient outcome are necessary for life-long learning in medicine. While biomedical information grows at an exponential rate and email inboxes are ooded with learning opportunities, learners seek relevant, e cient, and meaningful educational forums.
Active learning describes a broad range of learning activities (e.g., ipped classroom, think-pair-share, turn and talk, bulleted breaks during lecture) that require a learner to "construct", understand, and comprehend the knowledge derived from their educational experience "while simultaneously improving knowledge gain and recall abilities" [4]. Although AL strategies were introduced nearly 40 years ago, reports of use in medical education are limited, but the bene ts of AL have been well described. Increased content knowledge, critical thinking, problem-solving abilities, and positive attitudes towards learning have been noted when inquiry-based cooperative learning was compared to traditional lecture-based learning formats [5]. In addition, development of graduate capabilities such as critical and creative thinking, problem-solving, adaptability, communication and interpersonal skills were observed when AL strategies were used in graduate education [6]. Freeman et al. concluded, "If the experiments analyzed here had been conducted as randomized controlled trials of medical interventions, they may have been stopped for bene t-meaning that enrolling patients in the control condition might be discontinued because the treatment being tested was clearly more bene cial" [7].
Although we know that AL is effective, it's unclear why it isn't used more commonly. There is opportunity to learn more about knowledge, usage, attitudes and barriers to use of AL methods to improve content delivery that fosters knowledge retention and self-directed learning.
A knowledge gap about use of AL learning strategies extends across all areas of medical education, including continuing medical education (CME) [8]. These research efforts emerged from the results of a 2018 survey of academic CME unit leaders in the U.S. and Canada suggesting that the lecture is the most common educational delivery method in CME [9]. An earlier report in the CME literature highlighted the importance of using learner-centric AL formats to engage the learner with the teacher and audience to increase self-awareness and critical thinking [3]. As in other learner groups, use of AL methods in CME has been shown to reinforce content and promote changes in physician practice with improvement in patient outcome [10]. Given the importance of these methods in changing practice, there is a substantial gap in the literature about the reasons surrounding the limited use of these newer learning strategies in CME. By understanding more about knowledge, usage, attitudes and barriers to use of AL methods, the results of this research will provide CME programs with an opportunity to learn so that CME units can improve the learning environment, using their unique position within medical education to in uence physicians' practice. By creating greater awareness of AL methods across the educational continuum, there is an opportunity to engage learners by making lessons "stickier" (more comprehensible and memorable) and utilizing "lecture halls without lecture" [1]. CME is just one area on this continuum of medical education, however similar questions and survey tools can be used to extend our knowledge and awareness of the bene ts of AL in other areas of medical education.

Method
Potential survey respondents were leaders of CME units and members of Society of Academic Continuing Medical Education (SACME) at 144 U.S., 17 Canadian, and other CME medical education units (e.g., American College of Surgeons, American Academy of Pediatrics). SACME is the academic society for continuing medical and interprofessional education that advances continuing education through scholarship by promoting and advancing high-value healthcare. Respondents were identi ed through the SACME Communications Committee listserv.
While the 2018 Harrison Survey provided the AL construct for the study and reported that the lecture format is the most commonly used learning format in CME, the survey results provide a widesweeping "snapshot of the current structure and function of CME/CPD units at medical schools in the United States and Canada." [9] and did not speci cally address AL strategies. Therefore, the literature was searched extensively for publications about AL, but there have been no surveys published that represent the construct in its entirety. Questions were subsequently developed for this de novo survey based on 6 content areas that were identi ed through another comprehensive review of existing literature about AL use in medical education and higher education in general: 1) demographics; 2) knowledge; 3) usage; 4) attitudes about; 5) barriers to use; and 6) potential strategies to address those barriers [4,7,9,11]. AMEE Guide No. 87 [12], a guideline publication outlining best practices for survey reporting [13], and an expert in survey design and administration from the University of Colorado School of Public Health in uenced item development. Subsequently, 20 draft items consisting of multiple choice, matrix, yes/no, and freetext questions were pretested by content experts (including 15 members of the SACME communications and scholarship committees) as well as the expert in survey design. After this initial round of survey item revision, cognitive interviews were conducted with 7 randomly selected SACME members. This was followed by a nal round of survey item revision, instrument formatting, and re nement. Altogether, pretesting, cognitive interviews, and expert reviews followed by further revision helped to establish content and response process validity.
The Active Learning Survey in CME was conducted under an IRB-approved protocol (#19-1274) through the University of Colorado School of Medicine. An introductory email announcing the upcoming survey was distributed to ~ 350 SACME members by the Content Manager of SACME. Survey invitations were then sent by this individual directing individual members to a speci c SurveyMonkey® link (Active Learning Survey in CME). This email explained the purpose and objectives of the survey. They were informed that they were voluntarily consenting to participate in a research study when completing the survey. No compensation or incentives were provided. The 20-question survey took approximately 10 minutes to complete. Four email reminders were sent to non-respondents during the 6-week study period in October-November, 2019.
Demographic data were collected. Respondents were asked about their perceived level of knowledge about AL methods with a four-point Likert scale (very knowledgeable, mostly knowledgeable, somewhat knowledgeable, not at all knowledgeable). Respondents were also asked what type, if any training they had received in AL methods (e.g., certi cate or Master of Education). Their beliefs and the extent of agreement or disagreement (strongly agree, agree, neither agree or disagree, disagree, or strongly disagree) with attitudinal statements about AL were also assessed in the survey. CME unit leaders were asked about the use of AL compared to lecture in CME activities at their institution. AL strategies were de ned for participants in a linked online Teaching Practice Terminology Tip Sheet. If respondents reported that AL strategies were used in their unit, they were prompted to identify which type of strategy was used. Survey respondents were also asked to indicate the degree to which various factors made it more di cult to use AL in their CME unit. This was followed by a question about how helpful various methods would be in facilitating the use of AL strategies in their unit. Respondents were also invited to complete open-ended questions about best practices for use of AL at their institution. A link to the Association of American Medical Colleges (AAMC) Faculty Roster 2018 was provided for respondents to identify the size of faculty at their institution.
Non-responders were identi ed by review of medical school IP addresses not represented in the data set. Ten non-responders were contacted by phone to participate in a brief scripted follow-up telephone survey focused on a subset of key survey questions, including demographics, use of AL in their CME unit, and barriers to usage. This telephone survey was conducted by the principal investigator to identify potential differences and extent of non-response bias between responders and non-responders. Sample size for this survey was limited by the xed number of academic CME units in the U.S. and Canada.
Data from de-identi ed survey responses were collected electronically with SurveyMonkey® and analyzed using the IBM® SPSS® Statistics 26.0 (publication date: April 9, 2019; https://www.ibm.com/analytics/spss-statistics-software) software platform. The primary outcomes analyzed were the overall knowledge, attitudes about, use, barriers, and barriers to use of AL in CME. All demographic data were summarized using descriptive statistics (means, standard deviations, frequencies, percentages). Spearman's correlations were used to assess statistical associations between: 1) advanced training in AL (e.g., certi cate, Master, or Doctor of Education degree) and self-perceived knowledge of AL; 2) self-perceived knowledge of AL and percentage of CME activities in their institution using AL; and 3) advanced training in AL (e.g., Certi cate, Master's, or Doctor of Education degree) and percentage of CME activities using AL in their institution. Internal consistency and reliability of the survey scores were assessed with Cronbach's or Krippendorf's alpha coe cients which were calculated for each subconstruct measured.

Results
We were able to collect 146 responses. This represents ~ 42% of SACME unit leaders from 93 U.S. and Canadian medical schools as well as other medical specialty societies (e.g., American College of Surgeons, American Academy of Pediatrics). Since not all respondents answered all questions due to skip-logic survey methodology, not all data sets are complete. The number of full-time faculty in medical schools represented ranged from 150-8,000 with a mean of 1,406 full-time faculty members per responding institution. Seventy percent of the responding CME unit leaders were based in a medical school. More than 50% were CME Unit Directors and Assistant/Associate Deans of CME. Just over half of participating unit leaders had been in their positions for more than 6 years. Other demographics are described in Table 1. Sixty percent of the survey participants described themselves as very or mostly knowledgeable about AL (see Fig. 1). Approximately 80% of respondents had participated in national continuing education programs (e.g., annual conferences) that included information about AL methods. In addition, ~ 50% had participated in a continuing education program at their institution and/or had engaged in self-study that included information about AL. Fewer respondents had participated in on-line courses. More than onethird of the survey respondents had completed a certi cate, Master, or Doctor of Education degree that included information about AL methods. Spearman's correlation revealed that advanced training was positively correlated with self-perceived knowledge of AL (Spearman's correlation = 0.23 at P = .01).
More than 80% of respondents reported that AL methods were used in less than half of CME activities at their institutions. Simulation, audience response polling, case-based teaching, small group activities, and panel discussions were used most commonly (> 90%). Active learning strategies used in CME units, as reported by CME unit leaders are outlined in Table 2. Respondents also noted other examples of AL methods including gaming, group re ection with coaching between sessions, and role playing. While advanced training was not signi cantly correlated with an increased percentage of CME activities using AL (Spearman's correlation = 0.14 at P = .06), higher levels of self-perceived knowledge were correlated with an increased percentage of CME activities using AL in respondents' institutions (Spearman's correlation = 0.16 at P = .04). More than 90% of respondents agreed or strongly agreed that AL combines engagement with re ection. They also agreed that AL requires cooperation of the instructor to engage learners by working together to increase knowledge and recall (see Table 3).  (3) The learner is engaged so that both knowledge gained and recall are increased in AL 93.4% (113) 5.8% (7) 0.8% (1) Both the instructor and the learners work cooperatively in AL 90.9% (110) 5.8% (7) 3.3% (4) With AL, instructors are more concerned with eliciting re ective thoughts that apply knowledge to practice than merely conveying facts Most commonly perceived barriers to use of AL are lack of familiarity of faculty/presenters with AL methods in addition to increased preparation and planning for AL. Other barriers are described in Table 4. Respondents suggested that guidelines for use of AL strategies, faculty development programs to include training in AL, and coaching course directors to discuss AL methods, are examples of how CME unit leaders could increase AL content in CME activities. Other resources that were noted by respondents in assisting them in developing AL strategies in their institutions are presented in Table 5. Score validity and reliability of the survey instrument were assessed by applying Messick's ve sources of validity evidence [14]. While the new instrument was being developed, we collected evidence for content validity from an exhaustive literature review and consultation with content experts for question review and revision. Response process validity was also examined by testing the questions by experts and reviewers before distributing the survey. Since construct validity re ects the extent to which a measure 'behaves' the way that the underlying construct should behave in relation to other constructs, Spearman's correlations were used to establish this type of validity. Score reliability of the survey instrument is reported in Tables 2-5.
When addressing potential non-response bias with telephone interviews, we found results similar to the online survey. Most of the 10 non-respondents that were questioned by phone were CME Unit Directors or Program Managers and 45-64 years of age. Less than 50% of these telephone interviewees used AL in CME activities at their institutions. Barriers included a lack of familiarity with AL methods, more timeconsuming planning and preparation, and a lecture-based culture at their institutions.

Discussion
Increasing awareness of AL and it's lack of use as well as extending the use of this AL survey instrument and methodology to other areas of medical education were important goals in publication of this study. However, the primary goal of this survey was to extend the results of the 2018 Harrison survey and address unanswered questions about the knowledge, usage, attitudes, and barriers to usage of AL strategies in academic CME units in U.S. and Canada. The results suggest that many CME unit leaders are well-trained and have education in AL as well as positive attitudes towards the use of AL methods, but the change from the lecture format to implementation of AL methods has been minimal. While the mission of the Accreditation Council for Continuing Medical Education (ACCME) is to "assume and advance quality learning for healthcare professionals that drives improvements in patient care" [15], the translation of knowledge into practice that improves patient outcome is complex [16]. Although much has been written about the quality of health care delivered in the U.S. [17,18], the challenge for CME is to deliver impactful medical knowledge in a way that providers are able to retain and engage with the delivered knowledge in order to stay current and improve patient care.
In general, faculty members have not kept pace with pedagogical advancements and knowledge of medical education delivery methods [2]. Little is known about the actual training and knowledge of CME unit leaders in AL methods. In our study, we found that many of the CME unit leaders have both formal and informal training that included information about AL. More than half of the respondents consider themselves knowledgeable about AL methods. Learners in these institutions may have an edge since engagement and translation of the knowledge facilitates changes in practice and improved patient outcomes [10].
While the practice of medicine has changed dramatically during the last half century and knowledge and information have proliferated over the last several decades [19], medical classroom teaching has not changed substantially and is still driven by the lecture format [9]. Our study con rms that change in teaching methods comes slowly. Despite the fact that most of the respondents in our survey described themselves as knowledgeable about AL methods, 80% reported that AL is used in less than half of the CME activities at their institutions. When AL strategies were used, audience response polling, simulation, and group discussions were the most common. We asked survey respondents about the source of their responses to determine the accuracy of their reporting. They indicated that their answers were mostly based on factual CME unit data and frequent attendance at lectures, courses, or conferences at their institutions, validating the accuracy of our study ndings. Taken together, the positive correlation of advanced training and self-perceived knowledge with use of AL suggest that many CME unit leaders are trained to use and facilitate these teaching methods in their institutions.
There are many AL strategies for instructors to consider when they design courses, but negative attitudes and pedagogical barriers must be overcome [4]. When exploring these potential attitudes of CME unit leaders, our survey demonstrates that survey participants recognize to a large degree the importance of learner engagement to increase knowledge and recall. They also agree that instructors who use AL are more concerned with facilitating re ection and working cooperatively with the learner. As a result, instead of merely conveying facts to the learner, knowledge can be applied in practice. While more than 60% of survey participants indicate that lectures are not di cult to adapt to AL methods, instructors at their institutions, nevertheless, do not incorporate AL methods to a large degree into their CME activities. Our study con rms that the bene ts of AL are recognized but practical implementation still lags behind in many institutions.
While respondents have knowledge, positive attitudes, and perceptions about AL, we still needed to explore barriers to AL and describe resources needed to overcome these barriers to better understand the challenges of CME unit leaders in their institutions. In STEM disciplines, barriers to use of AL methods are described as "complex" [11] and include the triad of lack of time, training, and incentives to implement educational change. In medical education, however, barriers to use of AL have not previously been described. In our study, we found similar barriers identi ed by the STEM disciplines and additional resources are needed for preparation, planning, and faculty development. Other barriers include a lack of administrative support and change-averse cultures at institutions. Similar to the STEM disciplines, assistance from educational designers, institutional recognition, as well as additional training and faculty development in AL was found to be useful for increasing the use of AL in CME.
The strengths of this study are the methods used to develop the survey. Questions used to develop the construct, AL, were developed through an in-depth literature review of AL. Survey development was also guideded by Messick's ve sources of validity evidence [14]. Although content and response process validity are strong because of use of extensive pretesting and cognitive interviewing of the study population prior to dissemination, the lack of established criterion validity is a limitation of the study and will be an important next step in this area of research. Buy-in from SACME also facilitated survey distribution to ensure that all members of the society had an opportunity to respond to the survey.
Following survey distribution and data collection, we evaluated reliability of the survey instrument. Although one of our subconstructs, 'Active learning strategies currently used in CME units' had lower internal consistency reliability related to the yes/no question type and distribution of the data, the Likert scale survey questions demonstrated high internal consistency reliability as con rmed by Cronbach's alpha.
Limitations of the study include low response rates as observed with on-line surveys. Reasons for this include potential incorrect email addresses, survey fatigue of the study population, and membership changes within SACME, whose membership is dynamic. The varying number of SACME listserv members, directly impacts the denominator. Although voluntary responses improve the quality of the responses, non-response bias must also be considered. Since it could be inferred that individuals more interested in AL were more likely to complete the survey, we attempted to limit non-response by frequent reminders.
Since non-response bias also increases the likelihood of differences in the group of respondents compared to non-respondents, we followed up with telephone surveys of 10 randomly selected nonrespondents to validate the results of selected questions. Results of these telephone surveys were similar to online survey results, indicating a small effect of non-response bias on our results. Although some institutions may have been represented by several respondents, the anonymous nature of the survey and heterogeneity in responses limits this concern. The non-validated nature of self-reported items such as training, knowledge, and use of AL represent other recognized limitations. Finally, the use of AL cannot make up for poor event content, even if AL is used as the teaching method.
This study is the rst to expand one conclusion of the 2018 Harrison survey by describing the knowledge, usage, attitudes, and barriers to implementation of AL strategies in academic CME units in the U.S. and Canada. Our results suggest that increased levels of self-perceived knowledge but not advanced training are associated with increased use of AL. Despite CME unit leaders' training, education, and positive attitude toward AL, the use of AL is infrequent in many CME units, often as a result of limited resources including faculty development and institutional recognition for AL teaching. Future directions include extending the survey and its methodology to other areas of medical education such as undergraduate and graduate medical education, incorporating qualitative data (e.g., in-depth narrative data) to gain better depth of understanding of the AL construct, as well as follow-up with similar surveys done in tandem with other aspects of the Harrison survey to trend the data. We would welcome collaboration by the AAMC and SACME to create awareness of AL and the considerations of its use and implrmentation.

Conclusion
By improving dissemination of information about evidence-based AL strategies in medical education, the results of this study can be used to increase innovation more broadly and facilitate high-value education, especially to foster learner engagement, critical thinking, and problem-solving ability. Declarations Ethics approval and consent to participate: The Active Learning Survey in CME was conducted under an IRB-approved protocol (#19-1274) through the University of Colorado School of Medicine. Survey participants were informed that they were voluntarily consenting to participate in a research study in an introductory email prior to completing the online survey.
Consent for publication: Consent for publication was obtained in the introductory email and explained that the data obtained from the survey would be used for the purpose of publication.
Availability of data and material: Data used and/or analysed for the study are available from the corresponding author after a request.
Competing interests: The authors declare no con icts of interest.
Funding: This research was supported by departmental and institutional resources (survey development, data collection, statistical analysis) of the University of Colorado School of Medicine. It did not receive any grants from funding agencies.
Authors' contributions: All members of the research team contributed to the conception and development of the survey that was used to collect data. The Content Manager of SACME distributed the survey via a solicitation email to the SACME listserv. BB collected the data. BB, NA, JH and UK took part in the analysis and interpretation of the data collected. All members of the research team contributed to drafting of the nal manuscript. They have read and approved this nal version of the manuscript. In addition, they have reviewed the manuscript for accuracy and integrity.