- Research article
- Open Access
- Open Peer Review
A national training program for simulation educators and technicians: evaluation strategy and outcomes
https://doi.org/10.1186/s12909-016-0548-x
© Nestel et al. 2016
- Received: 20 March 2015
- Accepted: 14 January 2016
- Published: 22 January 2016
Abstract
Background
Simulation-based education (SBE) has seen a dramatic uptake in health professions education over the last decade. SBE offers learning opportunities that are difficult to access by other methods. Competent faculty is seen as key to high quality SBE. In 2011, in response to a significant national healthcare issue – the need to enhance the quality and scale of SBE - a group of Australian universities was commissioned to develop a national training program - Australian Simulation Educator and Technician Training (AusSETT) Program. This paper reports the evaluation of this large-scale initiative.
Methods
The AusSETT Program adopted a train-the-trainer model, which offered up to three days of workshops and between four and eight hours of e-learning. The Program was offered across all professions in all states and territories. Three hundred and three participants attended workshops with 230 also completing e-learning modules. Topics included: foundational learning theory; orientation to diverse simulation modalities; briefing; and debriefing. A layered objectives-oriented evaluation strategy was adopted with multiple stakeholders (participants, external experts), methods of data collection (end of module evaluations, workshop observer reports and individual interviews) and at multiple data points (immediate and two months later). Descriptive statistics were used to analyse numerical data while textual data (written comments and transcripts of interviews) underwent content or thematic analysis.
Results
For each module, between 45 and 254 participants completed evaluations. The content and educational methods were rated highly with items exceeding the pre-established standard. In written evaluations, participants identified strengths (e.g. high quality facilitation, breadth and depth of content) and areas for development (e.g. electronic portfolio, learning management system) of the Program. Interviews with participants suggested the Program had positively impacted their educational practices. Observers reported a high quality educational experience for participants with alignment of content and methods with perceived participant needs.
Conclusions
The AusSETT Program is a significant and enduring learning resource. The development of a national training program to support a competent simulation workforce is feasible. The Program objectives were largely met. Although there are limitations with the study design (e.g. self-report), there are strengths such as exploring the impact two months later. The evaluation of the Program informs the next phase of the national strategy for simulation educators and technicians with respect to content and processes, strengths and areas for development.
Keywords
- Simulation
- Program evaluation
- Health workforce
- Faculty development
Background
Educators may require unique and specific pedagogical skills to effectively support learning through simulation. As McGaghie et al. state: Simulation-based education (SBE) “is not easy or intuitive; clinical experience alone is not a proxy for simulation instructor effectiveness” [1]. As simulation is increasingly used in health professional education, there is a concomitant need for appropriately skilled teachers. While there are many faculty development programs in SBE, we could not locate any related theoretical or empirical published reports. From an international perspective, Navedo and Simon (2013) provide a descriptive account of thirteen multi-day continuing education simulation instructor courses [2]. Although it is unclear how the programs were identified (or sampled), they write, “Typically, these programs are on-site, intensive experiences with established and well-defined learning outcomes.” (p596) The authors report features common to the courses such as, “overviews in teaching and learning theory, introduction to simulation-based learning, orientation to the equipment, debriefing fundamentals, and management of common problems and offer hands-on opportunities to practice.” (p596) For simulation educators who seek further scholarly advancement, the authors describe eight award programs with key components on SBE. Zigmont et al. (2015) describe the content of basic and advanced courses for simulation educators [3] recommending Kern et al’s (1998) six steps for curriculum design [4] and mapping course content to the blueprint of the Certified Healthcare Simulation Educator standards of the Society for Simulation in Healthcare http://www.ssih.org/Certification/CHSE). Nestel et al. (2013) also offer a descriptive account but of a national strategic approach to SBE and the role of programmatic factors [5]. They argue for enabling connections of national, state and local initiatives such that programs are not seemingly ad hoc but coordinated and facilitated through development of communities of practice. Reviews of effective SBE cite the need for targeted training for faculty [1, 6].
In Australia, the healthcare simulation education community has experienced a period of particularly rapid growth. This is in part a response to the actions of Health Workforce Australia (HWA), a body “established to meet the future challenges of providing a health workforce that responds to the needs of the Australian community” [7]. Disestablished in late 2014, HWA was federally funded and closely linked with state and territory governments. Like most Australian bureaucracies, HWA faced the challenge of ensuring equity throughout a nation, which has a huge geographic area, but with the majority of the population concentrated in major coastal cities. Additionally, it had a broad focus, including issues such as enhancing clinical training, workforce planning analysis and supporting the role of international health professionals. Within its program of clinical training reform [7], HWA contributed more than $90 million over three years to enhance SBE within Australia, to develop the health workforce.
The AusSETT Program
Participants’ ratings of the extent to which they met learning objectives for the modules in the AusSETT program
Number | Mean | Median | SD | 95 % Confidence interval for mean | ||
---|---|---|---|---|---|---|
Lower | Upper | |||||
C1: Simulation-based education: Contemporary issues for healthcare professions | ||||||
1 Outline common definitions of simulation-based education (SBE) | 254 | 4.68 | 5.00 | 0.90 | 4.56 | 4.79 |
2 Describe various types of simulation activities | 254 | 4.72 | 5.00 | 0.95 | 4.60 | 4.84 |
3 Describe the role of a ‘safe’ learning environment for SBE | 254 | 4.65 | 5.00 | 0.95 | 4.53 | 4.77 |
4 Debate the benefits of teaching and learning with simulation | 254 | 4.62 | 5.00 | 0.96 | 4.50 | 4.74 |
5 Debate the limitations of teaching and learning with simulation | 253 | 4.40 | 4.00 | 0.99 | 4.28 | 4.53 |
6 Give examples of simulation modalities best suited for learning specific skills | 253 | 4.31 | 4.00 | 1.00 | 4.19 | 4.44 |
7 Access publicly available resources on SBE | 253 | 4.35 | 4.00 | 1.10 | 4.21 | 4.49 |
8 Identify professional development opportunities for simulation educators and technicians | 250 | 4.38 | 4.00 | 1.08 | 4.24 | 4.51 |
9 Outline key research findings on simulation as an educational method | 252 | 4.43 | 5.00 | 0.96 | 4.30 | 4.55 |
10 Outline theories that inform SBE | 251 | 4.51 | 5.00 | 0.99 | 4.38 | 4.63 |
11 Explain key components of SBE, framed by available evidence | 249 | 4.37 | 4.00 | 1.04 | 4.24 | 4.50 |
C2: Training simulation educators | ||||||
12 Demonstrate implementation of educational theory in teaching practice | 247 | 4.51 | 5.00 | 1.01 | 4.38 | 4.63 |
13 Compare and contrast SBE methods used in health professional education | 245 | 4.68 | 5.00 | 0.92 | 4.56 | 4.79 |
14 Critically evaluate the educational efficacy of simulation | 246 | 4.36 | 4.00 | 0.97 | 4.23 | 4.48 |
15 Demonstrate and analyse debriefing and feedback methods used in SBE | 246 | 4.64 | 5.00 | 1.03 | 4.51 | 4.77 |
16 Design and facilitate a simulated learning event | 245 | 4.30 | 4.00 | 1.12 | 4.16 | 4.44 |
17 Develop an evaluation plan of a simulated learning event | 245 | 4.15 | 4.00 | 1.12 | 4.01 | 4.29 |
E1: Manikin-based simulation | ||||||
18 Describe the range of manikins available for healthcare simulations | 130 | 4.92 | 5.00 | 0.89 | 4.76 | 5.08 |
19 Identify features of manikins (e.g. mobility, immediacy of feedback etc.) | 130 | 4.78 | 5.00 | 0.93 | 4.62 | 4.95 |
20 Demonstrate how to decides which manikin to use | 132 | 4.57 | 5.00 | 1.03 | 4.38 | 4.75 |
21 Describe the benefits and difficulties of programming a manikin | 130 | 4.43 | 5.00 | 1.19 | 4.22 | 4.64 |
22 Plan and demonstrate a manikin-based simulation activity | 125 | 4.63 | 5.00 | 1.08 | 4.44 | 4.82 |
E2: Simulated patient (SP) methodology | ||||||
23 Describe contemporary applications of SP methodology | 69 | 4.94 | 5.00 | 0.79 | 4.75 | 5.13 |
24 Discuss responsibilities of SPs in teaching sessions | 69 | 5.00 | 5.00 | 0.91 | 4.78 | 5.22 |
25 Create SP roles for teaching and assessment | 69 | 4.78 | 5.00 | 1.13 | 4.51 | 5.05 |
26 Describe a systematic approach to SP training for role portrayal | 69 | 5.06 | 5.00 | 0.84 | 4.85 | 5.26 |
27 Consider the content of feedback from SPs to trainees | 69 | 4.97 | 5.00 | 0.95 | 4.74 | 5.20 |
28 Describe principles of feedback as they apply to SP-based education | 69 | 4.93 | 5.00 | 0.94 | 4.70 | 5.15 |
29 Describe the practicalities of training SPs and SP educators (including assessment) | 68 | 4.82 | 5.00 | 1.04 | 4.57 | 5.07 |
E3: Virtual environments | ||||||
30 Describe the advantages and disadvantages of virtual environments and their applications | 45 | 4.43 | 4.00 | 1.00 | 4.13 | 4.74 |
31 Describe a range of virtual environments | 45 | 4.75 | 5.00 | 1.06 | 4.43 | 5.07 |
32 Articulate the role of educational design in effective virtual environments | 44 | 4.39 | 4.00 | 1.02 | 4.08 | 4.70 |
33 Describe issues in developing virtual environments | 45 | 4.41 | 4.00 | 1.06 | 4.09 | 4.73 |
34 Teach simulation educators to review virtual environments for SBE | 45 | 4.25 | 4.00 | 1.14 | 3.90 | 4.60 |
E4: Hybrid simulation (Patient focused simulation) | ||||||
35 Outline key elements of SP-based methodology | 144 | 4.71 | 5.00 | 0.95 | 4.55 | 4.86 |
36 Describe the rationale for hybrid simulations | 144 | 4.90 | 5.00 | 0.92 | 4.74 | 5.05 |
37 Define patient-focused simulations | 144 | 4.84 | 5.00 | 0.95 | 4.68 | 5.00 |
38 Outline the scope practice of hybrid simulations | 144 | 4.76 | 5.00 | 0.92 | 4.60 | 4.91 |
39 Describe relevant theory underpinning patient-focused simulations | 144 | 4.67 | 5.00 | 1.02 | 4.50 | 4.84 |
40 Review written scenarios for patient-focused simulation | 144 | 4.50 | 5.00 | 1.07 | 4.32 | 4.67 |
41 Describe a systematic approach to SP training for role portrayal | 144 | 4.87 | 5.00 | 0.99 | 4.70 | 5.03 |
42 Consider the role of feedback in patient-focused simulations | 143 | 5.00 | 5.00 | 0.87 | 4.86 | 5.14 |
43 Practice training SPs for feedback in patient-focused simulations | 143 | 4.62 | 5.00 | 1.20 | 4.42 | 4.82 |
44 Describe the practicalities of training SPs and SP educators in patient-focused simulations (including assessment) | 143 | 4.62 | 5.00 | 1.05 | 4.44 | 4.79 |
T1: Equipment sustainability and task trainers | ||||||
45 Describe the roles of the simulation specialist | 59 | 4.83 | 5.00 | 0.86 | 4.58 | 5.08 |
46 Demonstrate ability to perform hazard analysis and risk assessment for simulation activities | 59 | 4.81 | 5.00 | 0.89 | 4.55 | 5.07 |
47 Describe the value of preventative maintenance and how it can be accomplished | 59 | 4.90 | 5.00 | 0.72 | 4.69 | 5.11 |
48 Describe the concept of fidelity in relation to this role | 59 | 5.13 | 5.00 | 0.67 | 4.93 | 5.32 |
49 Discuss the needs of actors involved in simulation | 59 | 4.88 | 5.00 | 0.87 | 4.62 | 5.13 |
T2: Basic manikins and moulage | ||||||
50 Articulate the capabilities and limitations of various manikins | 55 | 4.59 | 5.00 | 1.02 | 4.31 | 4.87 |
51 Identify various cables and connections and what they do | 55 | 4.41 | 5.00 | 1.35 | 4.04 | 4.78 |
52 Describe the basics of programming manikins | 55 | 4.17 | 5.00 | 1.34 | 3.80 | 4.53 |
53 Familiarization with manikin parts and systems | 54 | 4.57 | 5.00 | 1.14 | 4.26 | 4.89 |
T3: Advanced manikins and audio visual (AV) systems | ||||||
54 Review the different types of AV systems | 51 | 4.33 | 4.00 | 1.07 | 4.02 | 4.63 |
55 Review the different AV connectors | 50 | 4.37 | 5.00 | 1.24 | 4.01 | 4.72 |
T4: Delivery of scenarios | ||||||
56 Articulate the process required to the design and delivery of a purposeful scenario | 54 | 4.31 | 5.00 | 1.24 | 3.98 | 4.65 |
57 Demonstrate an ability to develop, prepare and deliver an effective scenario | 54 | 4.22 | 5.00 | 1.28 | 3.87 | 4.57 |
58 Use reflection and constructive feedback to critique | 54 | 4.33 | 5.00 | 1.13 | 4.02 | 4.64 |
HWA’s original proposal was that AusSETT alumni would support the training of 6000 further educators and technicians through a separately funded program. National train-the-trainer programs are rare in this field and we have not identified any national programs for simulation educators and simulator technicians on a global scale. This paper describes the AusSETT Program and the associated objectives-oriented program evaluation.
The institutional members, responsible for developing the AusSETT Program, came from four states and included leading simulation education groups. The development team was comprised of ten authors with clinical (e.g. medicine, nursing, paramedicine, operating theatre technician), content (e.g. human factors, communication, assessment) and process (e.g. education, simulation modality) backgrounds. Authors have experience of several simulation modalities including manikins, task trainers, hybrid simulation, simulated patients, confederates, virtual patients and virtual environments and in different settings such as purpose-built centres, non-specialist settings and in situ. They have between six and 35 years experience of using simulation as an educational method to support learning in healthcare. All authors lived and worked in Australia at the time of development. The ten reviewers had a similar profile although three were from Canada and the United Kingdom.
Brief description of AusSETT Program modules
Module number | Module name | Module description | Format |
---|---|---|---|
C1 | Simulation-based education: Contemporary issues for healthcare professions | This module covers the rationale for simulation-based education (SBE) drawing on practices from high-reliability industries. Theories are explored as they relate to different simulation modalities (e.g. manikins, task trainers, simulated patients, virtual patients, gaming etc.) and domains (e.g. knowledge, attitudes and skills). The relevance of simulation to safety and quality in healthcare are highlighted. Participants consider historical and contemporary approaches to SBE with attention to the benefits and challenges in health professions education. Evidence for SBE is presented. The module offers key content to ensure that all participants have a common understanding of basic principles for SBE. The module is essential prior to attending any workshops. | e-learning |
C2 | Training simulation educators | This module covers basic principles of “train-the-trainer” programs, as well as the opportunity to practice and/or learn new simulation skills. One of the educator’s roles is to provide a safe learning environment, especially during feedback and debriefing. The module also explores how to teach and evaluate debriefing skills. Reference is made to the range of debriefing approaches used in different centres across Australia. Because there is no high level evidence for the use of a single debriefing approach, the Program is distinguished by its flexibility and willingness to support multiple approaches. This is a core module for all simulation educators and technicians. | e-learning; workshop |
E1 | Manikin-based simulation | This module covers the core aspects of manikin and task trainer simulations. This includes an overview of different manikin and task trainer capabilities and the impact these have on SBE. Scenario development, basic programming and scenario delivery are explored together with the concept of fit for purpose manikins. | e-learning; workshop |
E2 | Simulated patient (SP) methodology | In this module, participants explore fundamentals of SP practices including recruitment, selection, training for role portrayal and the role of SPs in feedback. The breadth of SP practices is explored and potential opportunities for applications across different contexts. | e-learning; workshop |
E3 | Virtual environments | This module enables participants to explore virtual environments as an educational method. Formal models and different design modalities provide participants with frameworks for practice. This module also provides an opportunity for participants to debate key issues in virtual patient methodologies. | e-learning |
E4 | Hybrid simulation (Patient focused simulation) | In this module, participants focus on blending simulation modalities. Examples include the linking of SPs with task trainers (patient focused simulation) to support the integration of all skills necessary for safe and effective clinical performance. Participants explore scenario development, briefing and feedback/debriefing. | e-learning; workshop |
T1 | Equipment sustainability and task trainers | This module focuses on the principles of equipment use, repair and maintenance and the role of technicians in supporting learning. | e-learning; workshop |
T2 | Basic manikins and moulage | This module focuses on the technical elements of the operation, troubleshooting and maintenance requirements of “low technology” manikins (e.g. Megacode Kelly and Megacode Kid). Participants develop skills in the operations and functionality of the manikins, including their limitations. Moulage techniques are explored including the use of realistic cuts, grazes and bruises and full body multi-trauma and burns. | e-learning; workshop |
T3 | Advanced manikins and audio visual (AV) systems | This module is designed for technicians who work with advanced manikins (e.g. SimMan, SimBaby, iStan, Meti Man) and AV systems. The module focuses on the set-up, operation, troubleshooting and maintenance requirements of manikins and AV systems, including their functionality and limitations. | e-learning; workshop |
T4 | Delivery of scenarios | This module T1, T2 and T3 enabling participants to develop, set-up and deliver a scenario. Participants are expected to set up the AV system, manikins and environment and deliver the scenario, with peers. The module emphasizes the importance of cooperative efforts of educators and technicians in all phases of SBE. | e-learning; workshop |
-
Apply knowledge and skills in the design and delivery of SBE; and
-
Practice contemporary training approaches that influence, motivate and inspire learners.
The Program combined e-learning and workshops and was free to participants. HWA networks nominated participants. The learning management system was custom designed while the e-portfolio was a proprietary product (PebblePad) [9]. Workshops were conducted across the country with six to 28 participants. Over ten months, 230 educators completed the Program.
-
To what extent can a national program for healthcare simulation educators and technicians prepare participants to support others in using SBE?
-
What are the strengths and limitations of the Program as identified in structure, process and outcome factors?
Methods
We adopted an objectives-oriented program evaluation approach [10] exploring the structure, process and outcome [11]. In the context of the AusSETT Program examples of structure include the range of skills and abilities of the team to develop and deliver the Program, the sequence and validity of the content (including the learning objectives), the scope of the Program within the time frame and the physical infrastructure for delivery. Examples of process include the educational methods such as the e-learning, workshop format, the e-portfolio and feedback mechanisms etc. Outcome refers to the extent to which the program goals were met. These were mainly focused on outcomes for participants and these were made clear in the learning objectives (Table 1). However, there were likely to be unexpected outcomes too. It was beyond the scope of our evaluation to monitor these changes but we anticipated identifying some through qualitative interviews with participants.
Summary of instruments used in the evaluation of the AusSETT Program
Instrument | Respondent | Evaluation component | Example | |
---|---|---|---|---|
1 | Module review | Consortium experts External experts | Structure Process | Not applicable |
2 | Baseline questionnaires | Participants Faculty | Structure | Additional file 1 |
3 | Observation of workshops | External experts | Structure Process Outcomes | Additional file 1 |
4 | End of module evaluations | Participants | Process Outcomes | Additional file 1 |
5 | Individual interviews | Participants | Structure ProcessOutcomes | Additional file 1 |
Module review
Program partners and external subject matter experts reviewed each module prior to delivery to ensure content was contemporary, accurate and contextualised. External reviewers were identified through collegial networks and published literature. Written feedback was sought and used to adjust content and methods. Internal review was undertaken throughout the development and implementation process – at meetings, in response to specific requests for feedback, and at workshops. Although all modules were sent externally for review with the same instruction, not all received formal written responses. Authors who nominated subject matter experts and wrote personally to the reviewers received detailed written feedback. Feedback was iteratively incorporated into modules and text of feedback was thematically analysed.
Baseline questionnaires
Online questionnaires were completed by all participants and faculty before commencing the Program and sought demographic and professional characteristics, and current practices of SBE (Additional file 1). Data were analysed using descriptive statistics in SPSS and content analysis.
Observation of workshops
In each state and territory, an external observer was invited to observe workshops (Additional file 1). The eighteen observers were selected based on their expertise in health professional education, simulation education and higher education. Observers used a template for each workshop enabling consistency. Free text comments were collated and content analysis was used to identify commonly recurring themes. They were not trained to use the semi-structured form relying on their expert judgement.
End of module evaluations
On completion of each e-learning module, participants were asked to rate the degree to which they met learning objectives and the value of the educational methods in attaining them. Prior to the program, we aspired for mean scores to equal or exceed 4.5 on the 6-point rating scale (1 = not at all met to 6 = completely met). That is, participants have more than ‘moderate’ achievement of learning objectives. Participants were also asked to identify five things they learned during the module that would inform their ability to train others, what worked well in the module and what needed to be improved (Additional file 1). Data was analysed using descriptive statistics. Conventional content analysis was used to collate free text comments [12].
Individual interviews
The goal of the individual telephone interviews was to gain deeper insight into participants’ experiences. Interviews were also used to interrogate other evaluation data (Additional file 1). Purposeful sampling identified participants across jurisdictions, health professional group and experience in simulation-based education. A topic guide was developed with overall Program goals in mind and to explore ‘unexpected’ outcomes. Telephone interviews were conducted by an independent researcher employed for the task and were scheduled for up to one hour. They were recorded and transcribed prior to thematic analysis [13]. JH and DN undertook the thematic analysis using an inductive approach. Key themes were identified independently and then negotiated. The data was then checked again for confirming and disconfirming evidence. An audit trail of analysis was maintained. The interviews were carried out two months after the completion of the Program.
Numerical data was analysed by JH while both JH and DN undertook the analysis of textual data. The Monash University Human Research Ethics Committee approved the study (#CF12/0244-2012000099). Written consent for participation was obtained from the participants to include their data in this study.
Results
Module review
Recommendations included raising the profile of interprofessional SBE, distinguishing essential from recommended or optional readings, indicating areas of overlap between modules, providing a glossary of terms and checking that content addressed general rather than specific principles of SBE. These structural concerns were undertaken and addressed.
Baseline questionnaire
Participants demographic information
Number of participants | Percentage | |
---|---|---|
Gender | ||
Female | 172 | 63 |
Male | 103 | 37 |
Age group | ||
40 years of age or less | 100 | 37 |
41 years of age or older | 168 | 63 |
Choice of stream | ||
Educator | 235 | 78 |
Technician | 65 | 22 |
Principle employer | ||
Public health care service | 153 | 55 |
University | 80 | 29 |
Private health care service | 23 | 8 |
Vocational education sector | 10 | 4 |
Other | 11 | 4 |
Target group for teaching | ||
Undergraduate & postgraduate | 133 | 52 |
Postgraduate only | 73 | 28 |
Undergraduate only | 51 | 20 |
Current role | ||
Nursing/midwifery | 161 | 57 |
Medical | 49 | 17 |
Allied health | 34 | 12 |
Educational/administrative or other non-clinical | 37 | 13 |
Simulation experience | ||
Spent 10 h of less per month | 135 | 53 |
Spent over 80 h per month | 18 | 7 |
Had previous simulation training | 102 | 40 |
The majority of participants had a nursing or midwifery background (57 %, n = 161), 17 % (n = 49) had a medical background and 12 % (n = 34) were from allied health with fourteen professions noted, including physiotherapy, paramedicine and occupational therapy. Thirteen percent (n = 37) described educational, administrative or other non-clinical backgrounds.
On simulation experience, the majority of participants (53 %, n = 135) spent ten hours or less per month on simulation activities, including 25 participants with no simulation experience. Seven percent (n = 18) indicated they spent over 80 h per month on simulation activities, which would indicate a full-time simulation education position. Forty percent (n = 102) of participants had previous simulation training, such as a debriefing course, an Advance Life Support instructor course, train-the-trainer courses or fellowships. Of these, nineteen had undertaken or were undertaking post-graduate studies in simulation.
Content analysis of participants’ views on simulation and the AusSETT Program
Number | |
---|---|
Top five themes: Most challenging aspect of simulation-based education | 259 |
1 Debriefing or aspects of debriefing such as feedback | 44 |
2 Issuesof authenticity, such as ‘making it real’ | 43 |
3 Engagement of staff or students as a challenge | 40 |
4 Finding the ‘time’ to conduct simulation-based education | 27 |
5 Resources, such as money or equipment | 21 |
Top five themes: Areas of interest in simulation practice | 262 |
1 Debriefing or aspects of debriefing such as feedback | 43 |
2 Manikin modality or simulation centre type simulation | 28 |
3 Professional skills, with 15 specifically mentioning teamwork | 28 |
4 Scenario development | 25 |
5 All aspects of simulation | 21 |
Top five themes: Reasons for enrolling in the AusSETT Program | 270 |
1 To develop, expand or consolidate knowledge and skills | 164 |
2 To train/support others | 52 |
3 A passion, interest or belief in simulation-based education | 40 |
4 Invited, nominated or part of employment | 40 |
5 Directly relevant to employment role | 25 |
Observation of workshops
The key outcomes of observer reports suggested that educational methods supported participants in meeting learning objectives. That is, there was alignment between content and process and perceived participants’ needs. Observers noted effective management of group dynamics, the creation of safety for participants and the benefits of learning across consecutive days with relationships forming between participants and faculty. Recommendations included making better use of e-learning in workshop activities, a similar template for all slides, extended time for discussion of local issues, and longer review time for some experiential activities.
End of module evaluations
Table 1 summarises the participants’ evaluation of the attainment of learning objectives. There were 58 learning objectives in total with 17 in the core modules, 27 in the educator modules and 14 in the technician modules. Overall, 37 learning objectives exceeded 4.5 while the remaining 21 learning objectives were in the range of 4.15 to 4.43. That is, overall participants reported at least partially meeting most objectives. For educator modules, E2 (7 objectives) and E4 (10 objectives) all learning objectives exceeded 4.5. For technician modules, all objectives in T1 exceeded 4.5 while in T2, two of four objectives exceeded 4.5 while neither T3 nor T4 reached this standard. Within the core modules (C1 and C2), one of the highest rated learning objectives was to describe various types of simulation activities with a mean of 4.72 (Median = 5.00, SD = 0.95, 95 % confidence interval = 4.60 to 4.84). The lowest rated learning objective was to develop an evaluation plan of a simulated learning event with a mean of 4.15 (Median = 4.00, SD = 1.12, 95 % confidence interval = 4.01 to 4.29).
A mean of 5.06 (Median = 5.00, SD = 0.84, 95 % confidence interval = 4.85 to 5.26) was reported for describe a systematic approach to simulated patient (SP) training for role portrayal. This is the highest reported learning objective within the educator stream. The lowest reported learning objective within the educator stream was for teach simulation educators to review virtual environments for SBE (Mean = 4.25, Median = 4.00, SD = 1.14, 95 % confidence interval = 3.90 to 4.60). In the technician stream, the highest rated learning objective was for describe the concept of fidelity in relation to [specialist/technician] role with a mean of 5.13 (Median = 5.00, SD = 0.67, 95 % confidence interval = 4.93 to 5.32). The lowest rated learning objective was for describe the basics of programming manikins (Mean = 4.17, Median = 5.00, SD = 1.14, 95 % confidence interval = 4.26 to 4.89).
Content analysis
Qualitative responses for module evaluation covered three areas: i) what worked well; ii) what needs improvement; and iii) things that participants had learned. As the data collected provided a large body of information, only selected responses are included.
What worked well
There were many successes with the modules such as the “open” small and large group discussions, high quality facilitation, the opportunity to learn from each other, debriefing exercises especially with observing different styles of debriefing, the use of non-clinical scenarios for learning about SBE principles, e-learning exercises and debriefing resources, the reference to educational theory relevant to SBE and the breadth of simulation modalities.
“It was a great discussion-based learning experience, drawing on others’ knowledge and experiences to create themes around simulation and debriefing; The faculty were extremely supportive”
“Different simulation modalities and the understanding that you don’t need the best simulator to achieve the outcome”
“I enjoyed the hands on part of the module where we designed a simulation exercise and needed to give feedback; Also the chance to meet other people trying to do the same thing or with other ideas/experience was great; As I am very new to this area it was all a bit daunting, so hearing some of their tips/techniques was helpful”
What can be improved
“The response boxes need to allow for editing. After brainstorming some ideas and submitting I realised that I could not re-enter to finalise my ideas; Scrolling between 2 pages to re-read quotes then meant that what I had entered into the text box had been lost; Quite frustrating…” (C1)
“The computer interface; Many of my responses were partially or fully deleted on submission; I was unable to then edit the response after submission and 90 % of my scenario was not loaded; I found this extremely frustrating and do not feel the saved answers reflect well either what I actually wrote or my effort in this module.” (E1)
“Couldn’t access the pebblepad” (C2)
“I have had difficulties accessing many of the links and the audio component, so that made it difficult; There was also a huge amount of readings required making it very lengthy to work through” (E2)
“I think AusSETT needs to have a clearer idea of where this project is going, as the presenters just didn’t have the answers about the next few steps; I worry that by the time the project is developed fully, I will have forgotten what I am supposed to train others in.” (C2)
Content analysis of participants’ reported learning
Content analysis of top five things learned from Module C2: Training simulation educators (n = 102)
Number | ||
---|---|---|
1 | Practical skills (e.g. using workshop slides, giving feedback, debriefing etc.) | 55 |
2 | Reinforced important processes of SBE (e.g. learning objectives, testing scenarios, debriefing practices etc.) | 25 |
3 | Obtained resources/tools/template (e.g. access to resources) | 18 |
4 | Theoretical knowledge (e.g. theories that inform simulation practice) | 14 |
5 | Processes that worked well (e.g. managing group dynamics, interactive workshops, different levels of expertise, co-facilitation etc.) | 6 |
Individual interviews
Verbatim statements from interviews with participants with key themes
Personal impact | |
1. Builds self-confidence | |
I went in as a person involved in simulation for about four years and stepping out of it I felt that, even with my meagre period of experiences in this field, I stepped up feeling more confident, I stepped out knowing that I had specific things that I could contribute to the running of a simulation centre and the use of simulation in education. (Participant 1) | |
I do the tech programme, the university simulation sessions, which are repeated every year. That sort of the length and breadth of what I do, but I’m stopping in the next six months looking perhaps doing some things with local GPs which would be taking simulation to them in their rooms to do ALS training. And that’s probably grown out of some confidence that I got from the AusSETT Program, and the concept of trying to spread simulation. (Participant 8) | |
2. Reinforce knowledge on simulation | |
I think it’s reinforced the usefulness of simulation (Participant 2) | |
I think it’s reinforced a few things, it’s reinforced the merits of simulation in particular areas where there has been some research, it’s reinforced my view that we still need to do a fair bit of work around researching what constitutes effective simulation training and it has reinforced something for me…. (Participant 6) | |
3. Gained new knowledge on simulation | |
… see how other people do things, and then you learn that there are different ways of doing things, and I think that probably enriches you as an individual by having a look at a little bit of everything. (Participant 4) | |
it’s certainly influenced my thinking and certainly influenced my knowledge in seeing what’s out there and what other people are doing and how it’s being used in other settings. (Participant 9) | |
4. Interest in other simulation program for professional development | |
I’m actually looking at doing a formal course now, as like the follow-on, perhaps a [removed] course or something like that. It’s certainly got my interest and I’m really keen to do more. So it’s been successful in that regard. (Participant 5) | |
I think my interest from here on is obviously to get a sense of the next steps and getting a clearer picture on who have the intention of doing further training and then starting to bring that together, because I think we may get some useful feedback with collaboration with others who wish to be facilitators. (Participant 6) | |
Organisational impact | |
1. Championing the use of simulation within organisation | |
… its allowed a broader range of people to start thinking about it and getting more knowledge about it, and then hopefully discussing it more in their workplace. (Participant 2) | |
I guess just opportunities to look at sharing some of what I’ve got from the program with others in our discipline as well who I know are really keen or get them along to the program to hear it as well. (Participant 5) | |
2. Use of simulation | |
We very much do a lot of mannequin training, and she very much opened up the doors to this hybrid system and using real people…I think that’s very interesting, and we actually have done a little bit now and we’ve had good feedback. (Participant 4) | |
3. Encouraging inter-professional learning | |
I’m moving simulations so one of the people in my group for this semester for example is from dentistry and veterinary science so we’re helping her set up some simulation in dentistry and also we’ve spoken about it with veterinary science as well. So we’re sort of looking at expanding it… (Participant 3) | |
Professional community impact | |
I think it has the potential to open people’s eyes to, that there is community out there and you can get support, you can get training, and you can go somewhere and find information or ask someone for information. So I guess that’s going back to that community sort of aspect there. (Participant 7) | |
I think we developed a quite cohesive group for the three days we were there and I know that we have, some of us have met up again at conferences and we’re happy, because we exchange phone numbers and emails etc.…. So I think that, yes, it was very good at developing, networking and collegial relationships. (Participant 3) | |
I do the tech programme, the university simulation sessions, which are repeated every year. That sort of the length and breadth of what I do, but I’m stopping in the next six months looking perhaps doing some things with local GPs which would be taking simulation to them in their rooms to do ALS training. And that’s probably grown out of some confidence that I got from the AusSETT Program, and the concept of trying to spread simulation. (Participant 8) |
Discussion
The results support the notion that the AusSETT Program was an important national initiative. The interviews and observations indicate the value that participants place upon the program and how access to the Program has resulted in significant goodwill in the healthcare simulation professional community. The overwhelming sense was that participants had high standards, were aspirational for their practices and wanted to make a difference using simulation as an educational method. However, the impact of the Program cannot be fully measured until the participants themselves offer simulation educator and technician training.
Structural factors
Many strengths of the Program were primarily structural. They included the building of a significant and enduring repository of contemporary resources (e-learning and workshops), and its offering in urban and rural regions, across jurisdictions, simulation modalities and professions. The Program also had a national profile with the representativeness of the development team. The module review process enhanced the strength of the structural factors.
Enabling structural factors from participants’ perspectives included being identified to attend the Program, some dedicated time off work to focus on professional development, networking associated with spending time immersed in SBE and offering the Program free of charge to participants. Other enablers included the access to e-learning. In fact, the e-learning also formed a significant structural constraint, particularly with respect to the functionality of features. Additionally, participants largely resisted the use of PebblePad, which did not seem related to the tool itself but to its use in this particular context. At a higher level, a constraining factor was that, at the time of the Program delivery, the next steps were not known to the Program faculty. Although HWA had longer-term goals, the mechanisms of government, especially in relation to budgets, did not permit sharing of plans in a more timely fashion. This created a barrier for engagement by individuals and institutions with the Program.
Process factors
Participants and workshop observers identified the types of active learning opportunities as a strength of the Program. Indeed, participants identified a key point of learning as including a series of ‘processes that worked well’, which were modelled during the process factors. The need to have skilled facilitators was reinforced by workshop observers; this may be a key enabling process factor for any future Program.
Outcome factors
Participants’ reported that they largely met learning objectives. Where objectives were not met as well as our pre-set standard, this may have related to the relative inexperience of some participants. Although observers reported alignment of content with perceived needs of participants, this may not have been achieved in every workshop as not all were observed.
It is worth noting is that there was no assessment associated with the AusSETT Program so we have no way of knowing if participants did achieve competence relevant to the learning objectives. This may be seen as both an enabling factor – in terms of access - and a constraining factor – in terms of accreditation – depending on perspective. Our philosophical view was that we wanted to provide continuous professional development and that at this nascent stage of development of a national community of practice of healthcare simulation educators and technicians, assessment may not have been well received. Also, within the funding model it was not possible to offer robust assessment, but this may be a future step.
Overall, the evaluation results give some indication that a national program for simulation educators and technicians is valuable for supporting others in using simulation as an educational method, at least from the perspectives of those who participated as future ‘trainers’. There is one major caveat. Many participants had no or limited experience with simulation and very few had formal qualifications. Indeed the first reason given for enrolling with the program was to “develop, expand or consolidate” skills. The AusSETT program was intended for experienced simulation experts, who could support others, not for those who were novices. In many ways this suggests that a ‘training’ program rather than a ‘train-the-trainer’ (‘educate-the-educator’) program may be more aligned with participants’ needs. However, the question of training others will only be fully answered once the AusSETT alumni have had an opportunity to offer professional development to other cohorts. However, there is no question that the Program promoted individual development and ignited interest in SBE in local facilities.
The contractual agreement strongly influenced some elements of the Program’s development and implementation and some of the language used to describe simulation practices – such as those of educator and technician. Although the opportunity to pilot the Program is likely to have reduced the scale of iterative development, the actual adjustments to the Program were minimal. The funding body requested a train-the-trainer model but in many respects we adopted a perspective of ‘educate-the-educator’, especially with respect to sharing theoretical underpinnings of many facets of simulation-based education.
Strengths and limitations of the study
The strengths of the study were the objectives-oriented program evaluation design, mixed methods, multiple sources of data (participants and subject matter experts) and the multiple data points collecting baseline, immediate responses after e-learning and workshops and then those in the longer term (two-months), at least for some participants.
Limitations of the study are that not all participants gave permission to publish their data which accounts for some of the discrepancies in reporting. The data is self-report which may not reflect the actual practices of participants. As internal evaluators, our own biases and assumptions may have influenced the data analysis and that which we have chosen to share from the very large data set. However, we employed an independent researcher (JH) to play a leading role in data analysis. Finally, in the absence of data-driven accounts of similar programs, it is not possible to discuss our findings in that context.
Conclusions
In summary, the evaluation of a national ‘train-the-trainer’ program for simulation educators and technicians suggested the value and need for a nationwide approach to training educators in simulation methodologies. The multi-layered objectives-oriented evaluation identified strengths and weaknesses of the Program with specific recommendations for improving its’ content and processes. We hope that sharing the details of learning objectives and our broader experiences may inform others embarking on simulation educator and technician training course development. The ‘national’ identity was achieved through the scale and diversity of the program in terms of simulation modalities, the geographical spread, breadth of professional disciplines, range of educational methods and the multi-institutional development.
Declarations
Acknowledgments
Health Workforce Australia funding of the program through the Department of Health. Our colleagues in several jurisdictions including Harry Owen, Joanna Tai, Matt Shuker, Lisa McCoy, Stephanie O’Regan and Nigel Choong for valuable contributions.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Authors’ Affiliations
References
- McGaghie WC, Issenberg SB, Petrusa ER, Scalese RJ. A critical review of simulation-based medical education research: 2003–2009. Med Educ. 2010;44(1):50–63. doi:10.1111/j.1365-2923.2009.03547.x. PubMed PMID:20078756. Epub 2010/01/19.
- Navedo D, Simon R. Specialized courses in simulation. In: Levine A, DeMaria S, Schwartz A, Sim A, editors. The Comprehensive Textbook of Healthcare Simulation. New York: Springer; 2013. p. 593–7.View ArticleGoogle Scholar
- Zigmont JJ, Oocuma N, Szyld D, Maestre J. Educator training and simulation methodology courses. In: Palaganos J, Maxworthy J, Epps C, Mancini M, editors. Defining Excellence in Simulation Programs. Philadelphia: Wolters Kluwer; 2015. p. 546–57.Google Scholar
- Kern D, Thomas P, Howard D, Bass E. Curriculum development for medical education: A six-step approach. Baltimore: Johns Hopkins University Press; 1998.Google Scholar
- Nestel D, Watson M, Bearman M, Morrison T, Pritchard S, Andreatta P. Strategic approaches to simulation-based education: A case study from Australia. J Health Spec. 2013;1(1):4–12.View ArticleGoogle Scholar
- Issenberg SB, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features and uses of high-fidelity medical simulations that lead to effective learning: a BEME systematic review. Med Teach. 2005;27(1):10–28. doi:10.1080/01421590500046924.PubMed. Epub 2005/09/09.View ArticleGoogle Scholar
- Health Workforce Australia. Health Workforce Australia 2012 [cited 2012 October 29]. Available from: https://www.hwa.gov.au/sites/uploads/hwa-work-plan-2012-13-approved-SCoH-20120810.pdf.
- Health Workforce Australia (HWA). Use of Simulated Learning Environments (SLE) in Professional Entry Level Curricula of selected professions in Australia: HWA; 2010. Available from: http://www.hwa.gov.au/sites/uploads/simulated-learning-environments-2010-12.pdf.
- Pebble Learning Ltd. Pebble Learning Home Telford, UK: Pebble Learning Ltd,; [updated 2015; cited 2014 23/12/2014]. Available from: http://www.pebblelearning.co.uk/.
- Fitzpatrick J, Sanders J, Worthen B. Program-Oriented Evaluation Approaches. In: Fitzpatrick J, Sanders J, Worthen B, editors. Program Evaluation: Alternative Approaches and Practical Guidelines. 4th ed. Upper Saffle River, NJ: Pearson Education; 2011. p. 153–71.Google Scholar
- Weiss C. Evaluation Research: Methods of Assessing Program Effectiveness. Prentice Hall: Englewood Cliffs, New Jersey; 1972.Google Scholar
- Hseih H, Shannon S. Three approaches to qualitative content analysis. Qual Health Res. 2005;15(9):1277–88.View ArticleGoogle Scholar
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3:77–101.View ArticleGoogle Scholar