A national training program for simulation educators and technicians: evaluation strategy and outcomes

Module review

Program partners and external subject matter experts reviewed each module prior to delivery to ensure content was contemporary, accurate and contextualised. External reviewers were identified through collegial networks and published literature. Written feedback was sought and used to adjust content and methods. Internal review was undertaken throughout the development and implementation process – at meetings, in response to specific requests for feedback, and at workshops. Although all modules were sent externally for review with the same instruction, not all received formal written responses. Authors who nominated subject matter experts and wrote personally to the reviewers received detailed written feedback. Feedback was iteratively incorporated into modules and text of feedback was thematically analysed.

Baseline questionnaires

Online questionnaires were completed by all participants and faculty before commencing the Program and sought demographic and professional characteristics, and current practices of SBE (Additional file 1). Data were analysed using descriptive statistics in SPSS and content analysis.

Observation of workshops

In each state and territory, an external observer was invited to observe workshops (Additional file 1). The eighteen observers were selected based on their expertise in health professional education, simulation education and higher education. Observers used a template for each workshop enabling consistency. Free text comments were collated and content analysis was used to identify commonly recurring themes. They were not trained to use the semi-structured form relying on their expert judgement.

End of module evaluations

On completion of each e-learning module, participants were asked to rate the degree to which they met learning objectives and the value of the educational methods in attaining them. Prior to the program, we aspired for mean scores to equal or exceed 4.5 on the 6-point rating scale (1 = not at all met to 6 = completely met). That is, participants have more than ‘moderate’ achievement of learning objectives. Participants were also asked to identify five things they learned during the module that would inform their ability to train others, what worked well in the module and what needed to be improved (Additional file 1). Data was analysed using descriptive statistics. Conventional content analysis was used to collate free text comments [12].

Individual interviews

The goal of the individual telephone interviews was to gain deeper insight into participants’ experiences. Interviews were also used to interrogate other evaluation data (Additional file 1). Purposeful sampling identified participants across jurisdictions, health professional group and experience in simulation-based education. A topic guide was developed with overall Program goals in mind and to explore ‘unexpected’ outcomes. Telephone interviews were conducted by an independent researcher employed for the task and were scheduled for up to one hour. They were recorded and transcribed prior to thematic analysis [13]. JH and DN undertook the thematic analysis using an inductive approach. Key themes were identified independently and then negotiated. The data was then checked again for confirming and disconfirming evidence. An audit trail of analysis was maintained. The interviews were carried out two months after the completion of the Program.

Numerical data was analysed by JH while both JH and DN undertook the analysis of textual data. The Monash University Human Research Ethics Committee approved the study (#CF12/0244-2012000099). Written consent for participation was obtained from the participants to include their data in this study.

Module review

Recommendations included raising the profile of interprofessional SBE, distinguishing essential from recommended or optional readings, indicating areas of overlap between modules, providing a glossary of terms and checking that content addressed general rather than specific principles of SBE. These structural concerns were undertaken and addressed.

Baseline questionnaire

The majority of participants had a nursing or midwifery background (57 %, n = 161), 17 % (n = 49) had a medical background and 12 % (n = 34) were from allied health with fourteen professions noted, including physiotherapy, paramedicine and occupational therapy. Thirteen percent (n = 37) described educational, administrative or other non-clinical backgrounds.

On simulation experience, the majority of participants (53 %, n = 135) spent ten hours or less per month on simulation activities, including 25 participants with no simulation experience. Seven percent (n = 18) indicated they spent over 80 h per month on simulation activities, which would indicate a full-time simulation education position. Forty percent (n = 102) of participants had previous simulation training, such as a debriefing course, an Advance Life Support instructor course, train-the-trainer courses or fellowships. Of these, nineteen had undertaken or were undertaking post-graduate studies in simulation.

Observation of workshops

The key outcomes of observer reports suggested that educational methods supported participants in meeting learning objectives. That is, there was alignment between content and process and perceived participants’ needs. Observers noted effective management of group dynamics, the creation of safety for participants and the benefits of learning across consecutive days with relationships forming between participants and faculty. Recommendations included making better use of e-learning in workshop activities, a similar template for all slides, extended time for discussion of local issues, and longer review time for some experiential activities.

End of module evaluations

Table 1 summarises the participants’ evaluation of the attainment of learning objectives. There were 58 learning objectives in total with 17 in the core modules, 27 in the educator modules and 14 in the technician modules. Overall, 37 learning objectives exceeded 4.5 while the remaining 21 learning objectives were in the range of 4.15 to 4.43. That is, overall participants reported at least partially meeting most objectives. For educator modules, E2 (7 objectives) and E4 (10 objectives) all learning objectives exceeded 4.5. For technician modules, all objectives in T1 exceeded 4.5 while in T2, two of four objectives exceeded 4.5 while neither T3 nor T4 reached this standard. Within the core modules (C1 and C2), one of the highest rated learning objectives was to describe various types of simulation activities with a mean of 4.72 (Median = 5.00, SD = 0.95, 95 % confidence interval = 4.60 to 4.84). The lowest rated learning objective was to develop an evaluation plan of a simulated learning event with a mean of 4.15 (Median = 4.00, SD = 1.12, 95 % confidence interval = 4.01 to 4.29).

A mean of 5.06 (Median = 5.00, SD = 0.84, 95 % confidence interval = 4.85 to 5.26) was reported for describe a systematic approach to simulated patient (SP) training for role portrayal. This is the highest reported learning objective within the educator stream. The lowest reported learning objective within the educator stream was for teach simulation educators to review virtual environments for SBE (Mean = 4.25, Median = 4.00, SD = 1.14, 95 % confidence interval = 3.90 to 4.60). In the technician stream, the highest rated learning objective was for describe the concept of fidelity in relation to [specialist/technician] role with a mean of 5.13 (Median = 5.00, SD = 0.67, 95 % confidence interval = 4.93 to 5.32). The lowest rated learning objective was for describe the basics of programming manikins (Mean = 4.17, Median = 5.00, SD = 1.14, 95 % confidence interval = 4.26 to 4.89).

Content analysis

Qualitative responses for module evaluation covered three areas: i) what worked well; ii) what needs improvement; and iii) things that participants had learned. As the data collected provided a large body of information, only selected responses are included.

What worked well

There were many successes with the modules such as the “open” small and large group discussions, high quality facilitation, the opportunity to learn from each other, debriefing exercises especially with observing different styles of debriefing, the use of non-clinical scenarios for learning about SBE principles, e-learning exercises and debriefing resources, the reference to educational theory relevant to SBE and the breadth of simulation modalities.

What can be improved

Content analysis of participants’ reported learning

Individual interviews

Structural factors

Many strengths of the Program were primarily structural. They included the building of a significant and enduring repository of contemporary resources (e-learning and workshops), and its offering in urban and rural regions, across jurisdictions, simulation modalities and professions. The Program also had a national profile with the representativeness of the development team. The module review process enhanced the strength of the structural factors.

Enabling structural factors from participants’ perspectives included being identified to attend the Program, some dedicated time off work to focus on professional development, networking associated with spending time immersed in SBE and offering the Program free of charge to participants. Other enablers included the access to e-learning. In fact, the e-learning also formed a significant structural constraint, particularly with respect to the functionality of features. Additionally, participants largely resisted the use of PebblePad, which did not seem related to the tool itself but to its use in this particular context. At a higher level, a constraining factor was that, at the time of the Program delivery, the next steps were not known to the Program faculty. Although HWA had longer-term goals, the mechanisms of government, especially in relation to budgets, did not permit sharing of plans in a more timely fashion. This created a barrier for engagement by individuals and institutions with the Program.

Process factors

Participants and workshop observers identified the types of active learning opportunities as a strength of the Program. Indeed, participants identified a key point of learning as including a series of ‘processes that worked well’, which were modelled during the process factors. The need to have skilled facilitators was reinforced by workshop observers; this may be a key enabling process factor for any future Program.

Outcome factors

Participants’ reported that they largely met learning objectives. Where objectives were not met as well as our pre-set standard, this may have related to the relative inexperience of some participants. Although observers reported alignment of content with perceived needs of participants, this may not have been achieved in every workshop as not all were observed.

It is worth noting is that there was no assessment associated with the AusSETT Program so we have no way of knowing if participants did achieve competence relevant to the learning objectives. This may be seen as both an enabling factor – in terms of access - and a constraining factor – in terms of accreditation – depending on perspective. Our philosophical view was that we wanted to provide continuous professional development and that at this nascent stage of development of a national community of practice of healthcare simulation educators and technicians, assessment may not have been well received. Also, within the funding model it was not possible to offer robust assessment, but this may be a future step.

Overall, the evaluation results give some indication that a national program for simulation educators and technicians is valuable for supporting others in using simulation as an educational method, at least from the perspectives of those who participated as future ‘trainers’. There is one major caveat. Many participants had no or limited experience with simulation and very few had formal qualifications. Indeed the first reason given for enrolling with the program was to “develop, expand or consolidate” skills. The AusSETT program was intended for experienced simulation experts, who could support others, not for those who were novices. In many ways this suggests that a ‘training’ program rather than a ‘train-the-trainer’ (‘educate-the-educator’) program may be more aligned with participants’ needs. However, the question of training others will only be fully answered once the AusSETT alumni have had an opportunity to offer professional development to other cohorts. However, there is no question that the Program promoted individual development and ignited interest in SBE in local facilities.

The contractual agreement strongly influenced some elements of the Program’s development and implementation and some of the language used to describe simulation practices – such as those of educator and technician. Although the opportunity to pilot the Program is likely to have reduced the scale of iterative development, the actual adjustments to the Program were minimal. The funding body requested a train-the-trainer model but in many respects we adopted a perspective of ‘educate-the-educator’, especially with respect to sharing theoretical underpinnings of many facets of simulation-based education.

Strengths and limitations of the study

The strengths of the study were the objectives-oriented program evaluation design, mixed methods, multiple sources of data (participants and subject matter experts) and the multiple data points collecting baseline, immediate responses after e-learning and workshops and then those in the longer term (two-months), at least for some participants.

Limitations of the study are that not all participants gave permission to publish their data which accounts for some of the discrepancies in reporting. The data is self-report which may not reflect the actual practices of participants. As internal evaluators, our own biases and assumptions may have influenced the data analysis and that which we have chosen to share from the very large data set. However, we employed an independent researcher (JH) to play a leading role in data analysis. Finally, in the absence of data-driven accounts of similar programs, it is not possible to discuss our findings in that context.