Designing simulation-based curriculum content for emergency medicine residents in France: a Delphi method

Abstract

Background

The value of simulation in emergency medicine is indisputable because it allows training and acquisition of many technical and non-technical skills (TS and NTS). In France, there are no curriculum regarding the use of simulation as a teaching tool during emergency medicine residency. The aim of this study was to design the content of a national simulation-based curriculum for emergency medicine residency programs.

Methods

The Delphi method was conducted between March and June 2022. The questionnaire was divided into three sections: TS, NTS and clinical situations as starting points (SSPs). A panel of emergency physicians’ experts on simulation education was established. An online survey was conducted in which they were asked to score, on a four-point Likert scale, the suitability of skills and SSPs to be taught through simulation courses during the emergency medicine residency. The questionnaire was revised between each round following comments or suggestions for additional items from the experts.

Results

Sixty-six experts completed the Delphi process. The initial questionnaire included 64 TS, 37 NTS and 103 SSPs. The experts’ comments led to the addition of 12 TS, 24 NTS and 6 SSPs. Consensus was obtained after three rounds. The experts selected 24 TS and 20 NTS to be taught as a priority through simulation during the emergency medicine residency, and 15 SSPs to be used in priority.

Conclusion

With a Delphi method, French experts in simulation-based emergency medicine education have selected 24 technical and 20 non-technical skills to be taught as a priority with simulation-based training to emergency medicine residents.

Background

The use of simulation-based education (SBE) as a teaching tool in medicine has significantly increased in the past decades and is associated with an improvement of knowledge, skills, and behaviors [1, 2]. Simulation in emergency medicine (EM) is now routinely used both for graduate and undergraduate education, as it allows the acquisition of many technical and non-technical skills (TS and NTS) [3,4,5,6]. Simulation also leads to enhanced and increased retention of learners’ knowledge. Different methods exist, each serving a specific purpose aligned with corresponding learning objectives [7]: procedural simulation to learn TS, such as upper airway management which is frequently studied [8]; mannequins or standardized patients to assess knowledge and teach NTS such as crisis resource management, communication, teamwork, multiple patients management, cognitive errors avoidance and ethical considerations. These training can be implemented directly in the emergency departments (EDs) through in situ simulations [9]. Other tools are being developed or evaluated: serious games, virtual reality or escape games [10]. In addition to the frequent clinical situations experienced in EDs, simulation also offers the opportunity to train in specific domains such as medical regulation [11] or mass casualties’ incidents [4].

To ensure that EM residents acquire the TS and NTS associated with the required knowledge to optimally manage patients, simulation should be integrated into their educational curriculum, in association with other existing educational tools [4, 7, 12]. Its use has become quite ubiquitous [13, 14] and is highly appreciated by EM residents [15, 16].

In France, EM has been recognized as a medical specialty since 2017 and the educational objectives of EM residents have been specified [17]. The ministerial decree includes simulation-based learning in the training of residents in almost all specialties. This is in addition to the recent reform of medical studies which promotes the use of a competence-based approach in health training programs. However, there are no guidelines or curriculum regarding the use and content of simulation-based training during the EM residency. Even though this educational method is well recognized, a great heterogeneity in its use is observed between the universities [13]. The development of a unique, national curriculum, in agreement with the specificities of EM practice in France, would facilitate the conception of simulation programs and scenarios by teachers. It would also allow students to benefit from an optimal, standardized simulation training throughout the country.

The aim of the work was to design the content of a national simulation-based curriculum for EM residency programs.

Methods

A Delphi method was performed by a working group of the National College of Emergency Medicine Academics (Collège National des Universitaires de Médecine d’Urgence, CNUMU), that included 12 academic emergency physicians, and staff emergency physicians, all involved in SBE in this specialty. The Delphi method is a process that involves consulting a panel of experts with the aim of achieving a consensus on a specific topic, in the absence of established scientific evidence. It can be used to validate the content of a curriculum [18, 19].

Development of the questionnaire

The initial questionnaire was designed by the CNUMU working group and divided into three sections: TS, NTS and situations as starting points (SSPs).

Following the main different steps when using a Delphi method, the first work consisted in a scoping review focused on simulation training for EM residents, with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology [20] in order to select TS and NTS. The inclusion criteria were broad to provide a comprehensive view of simulation curricula for EM residents around the world, without restriction of language, date, study design or publication type. The search was conducted in MEDLINE (Medical Literature Analysis and Retrieval System Online), Embase, Web of Science and Cochrane Database, using keywords specific to each database, from their inception to 31 December 2021 (Additional File 1). Of 1300 citations identified, after eliminating duplicates and articles not related to EM, simulation and/or residency education, 58 publications were finally included and analyzed by two independent authors (JT, PCT) to extract the TS and NTS taught by simulation. Redundant items have been merged and some skills have been rephrased.

SSPs are generic situations which cover the common circumstances, symptoms, complaints and findings that the physician should be able to manage on day one of residency [21]. SSPs are an important element of simulation scenarios’ design. For this reason, it was important to identify which SSPs to address with simulation techniques. They were obtained from the list published by the French experts in medical education of National Coordination of Medicine Teachers’ Colleges (Coordination Nationale des Collèges d’Enseignants en Médecine, CNCEM) during the reform of French medical studies. Two authors (JT, PCT) independently selected situations consistent with EM. All items selected by at least one of the two authors were retained. It was previously decided not to include pediatric-related content as there is an optional specialized curriculum for pediatric emergencies. It will probably be appropriate to conduct an independent study, in collaboration with pediatricians. This was done in Canada for the design of the pediatric EM curriculum [22]. Ultrasound items were also excluded as there is also a specific curriculum and educators had opted for training in ultrasound techniques without simulation [23].

The Expert Panel

The second step was to establish a panel of experts, according to the following criteria: being an EM physician practicing in France, with significant experience in SBE or being a regional director of an EM teaching department. The non-director experts were identified either by their participation in previous work on simulation-based training [24], or by their scientific involvement in the subject (conferences, publications), or on the proposal of a regional director or a previously identified expert (using snowball method that is recommended in a Delphi method). Ninety experts were identified. After presenting the methodology and verifying whether their profile matched the inclusion criteria, 81 were selected to participate. The identity of the experts was confidential (only known to the members of the working group).

The Delphi Process

The Delphi method was set to be completed in a minimum of three rounds, starting in March 2022. For each round, the experts were given three weeks to answer an online questionnaire (GoogleForm®). A maximum of three email reminders were sent out. The questions were “should these skills be taught through simulation during the EM residency?” or “should these SSPs be used for teaching through simulation during the EM residency?”. Simulation was defined as the use of all fidelity mannequins, procedural simulation, animal or cadaveric models, standardized patients, objective structured clinical examinations, serious games or virtual reality. The experts were asked to answer on a Likert scale from 1 (“no, not at all”) to 4 (“yes, definitely”). The questions were randomly ordered for each expert.

In the first two rounds, the experts were invited to comment on the submitted skills and SSPs, or to suggest additional items. Thus, the questionnaire was revised between each round according to the experts’ suggestions, by a consensus reached within the working group.

After the first two rounds, the participants were informed of the mean score for each item and the methodology used to select the items retained in the future curriculum (Table 1). This decision rule did set a mean score of 3 as the minimum threshold for determining which skills (and SSPs) should or should not be taught (used) with simulation in EM. This methodology had previously been employed in a similar Canadian Delphi study [18].

Table 1 Decision rule for simulation-based education during the EM residency as determined by the mean score given by the experts

Statistical Analysis

All statistical analyses were performed using Excel®. In each round, the mean score and its standard deviation were calculated. Items immediately selected (score ≥ 3.5) or excluded (score < 2.5) were removed from the questionnaire. The remaining items were resubmitted for voting in the next round. Two authors (JT, PCT) reviewed the qualitative comments of the experts, which led to modifications of the questionnaire (addition or modification of items). The Delphi process continued until a consensus was reached, defined as no item with a score ≥ 3.5 or < 2.5. The correlation between the marks of the first and third rounds were assessed with Spearman coefficient (r_s), for unchanged proposals.

Results

Eighty-one experts agreed to participate, including 59 men (73%), 28 academics (35%) and 11 regional directors (14%), with a median age of 43 years (Q1Q3 36–52, min–max 30–67). Among them, 77 (95%) completed the first round, 73 (90%) the second round and 66 (80%) the third round of the Delphi method. The initial questionnaire included 64 TS, 37 NTS and 103 SSPs. The experts’ comments resulted in additional 12 TS, 24 NTS and 6 SSPs (Fig. 1). Consensus was obtained after 3 rounds, in June 2022. The experts finally selected 24 TS and 20 NTS to be taught as a priority through simulation during EM residency (Tables 2 and 3), and 15 SSPs to be used in priority (Table 4). The skills to be eventually taught, the SSPs to be possibly used and the excluded items are presented in the Supplementary Tables 1–6, Additional File 2.

Fig. 1

Flow of the Delphi method. TS: technical skills; NTS: non-technical skills, SSP: situations as starting points

Table 2 Technical skills to be taught in simulation during the EM residency

Table 3 Non-technical skills to be taught in simulation during the EM residency

Table 4 Situations as starting points to be used in simulation during the EM residency

The correlation between the marks of the first and third rounds was very strong, with r_s = 0.81 (p < 0.001), 0.77 (p = 0.001) and 0.74 (p < 0.001) for the TS, NTS and SSPs, respectively.

Discussion

Sixty-six experts participated in the complete Delphi process, resulting in a list of 44 technical and non-technical skills that should be taught in priority through simulation to EM residents. Forty-six additional skills could potentially be taught, depending on the availability of local teaching resources.

Some experts suggested that the skills to be taught may differ depending on the year of residency. Indeed, the learning objectives of EM residents evolve as they advance in their training [25]. Some teaching curricula include different objectives for different years of experience within the specialty training, focusing for example on medical history taking, differential diagnosis formulating and patient stabilization for junior residents, and on advanced resuscitation, leadership, communication and task discontinuation for senior residents [26]. In France, the EM residency is a 4-year program in which the residents work full-time in hospitals and have monthly seminars. The first year is called the “basics” year (it includes a 6 months rotation in an ED), second and third years are the “intensification” period (including 6 months in an intensive care unit, 6 months in an emergency medical service (EMS), 6 months in pediatrics and a free 6 months rotation). The last year is a consolidation period (including 6 months in an ED and 6 months in an EMS). However, the competency-based approach requires an early overview of the skill set, with the goal of progressing step-by-step through each skill, using milestones to identify level of progression for each skill or knowledge. The alignment of teaching objectives with the students’ level of expertise is a major challenge in simulation-based training, in accordance with the flow theory [27]. In that sense, it seems wise to leave a certain degree of freedom to educators to align teaching goals to the respective level of expertise of students.

We could also have questioned the experts about the optimal simulation tool for each skill. However, literature regarding the impact of fidelity on learning states that the technology itself is not that relevant to the quality of learning [28]. The selected items could therefore serve as guidelines to help educators deploying their teaching programs, while leaving the required flexibility imposed by human and material resources [13]. Further work is required to assess the impact of the implementation of this curriculum content on teachers and students, and make further improvements.

Other authors have published their simulation-based training curriculum for EM residents. The methodology is not always specified [29, 30] or often the result of a “panel of experts” [16, 31]. Kern’s 6-step method has been proposed [26]. We chose to use a Delphi method, as Kester-Greene et al. previously in Canada [18]. The Delphi method is a validated process to obtain expert consensus on complex issues, in the absence of scientific evidence. Regarding content and teaching goals, the majority of published curricula included TS and NTS.

Most of the selected TS are similar to those found in other curricula, mainly related to cardiopulmonary resuscitation, upper airway management and vascular access [16, 18, 29, 31]. The procedures related to the occurrence of an unplanned delivery are more unusual and could be related to the specificity of the EMS pre-hospital organization in France. Some important TS have not been selected because they were taught earlier in medical school and were considered already acquired (e.g. insertion of a peripheral venous line), others because there is no appropriate simulation tool (e.g. reduction of dislocated joints). The large number of selected NTS reflects the importance of their acquisition for future emergency physicians. Crisis resource management, leadership, teamwork and communication are also present in previously published curricula [18, 26, 29, 31]. These similarities confirm that the priority topics selected by our experts are essential to any EM simulation training. The identification of the TS and NTS to be acquired as a priority by the EM residents can be useful to the teachers, for the construction of their simulation scenarios, but also to the students. By having access to clear, predefined learning goals, students may appreciate furthermore the simulation curriculum and be more rigorous to reach these learning objectives.

Some authors suggested detailed clinical cases [29, 30] or specific diseases (e.g. acute coronary syndrome) as a starting situation [16, 18]. We decided to validate various SSPs because it is aligned with the actual educational reform and also to help teachers to design clinical scenarios. Indeed, skills should always be contextualized. The selected SSPs are suggested for preferential use in setting up simulation scenarios in EM residency programs, but their use is at the discretion of the teachers. The SSPs can be declined in an unlimited number of scenarios leading to various diagnoses, thus allowing the learning of the targeted TS or NTS to be adapted to the progression of the residents. No matter which teaching tool is used, each scenario must be designed to meet one or more specified learning objectives, in compliance with a formalized model. The French-speaking Society of Health Simulation (Société Francophone de Simulation en Santé, SoFraSimS) proposed a homogeneous and reproducible framework for the construction of a simulation scenario, to facilitate the writing and reading of the scenarios by various stakeholders, but also to allow their open access for teachers of different simulation centers [32]. For example, they recommended 3 to 5 main learning objectives (TS or NTS) per scenario.

A good curriculum must be usable. Even if this educational tool is increasingly used, particularly in EM, there are certain barriers to the implementation of a SBE program. This results in a very heterogeneous density of teaching programs in French universities [13]. In Canada, a study showed a frequency of simulation courses ranging from weekly to twice a year depending on the university [33]. The most frequently cited barriers to the use of simulation are lack of time (66–75%), lack of teacher training (54–56%) and the cost of equipment and/or space (44–47%) [14, 33]. Even if solutions exist, such as shared simulation platforms, it will be necessary to adjust the curriculum to the local needs. This requires an assessment of the feasibility of implementing this curriculum content in the teaching programs for EM residents in all French universities, its usability and its effectiveness for teaching the skills of future emergency physicians. This process will have to be regularly updated by consulting the various stakeholders (teachers and students). A simulation assessment guide could also be helpful. In situ simulation is also a solution because it allows regular, interprofessional, realistic simulation training without having to attend the simulation center and therefore its associated cost [34].

The skills selected by the experts to be taught to EM residents through simulation have been approved by the CNUMU and a French document will be distributed to the regional directors of the EM teaching departments in all French universities. The methodology for evaluating the effective implementation of this curriculum content is currently under discussion and is scheduled to occur two years after implementation.

Limitations

This work has some limitations, mainly inherent to the Delphi method. First, the authors decided the selection and the number of experts. The panelists may not be representative of all EM teachers using simulation. However, we had defined a priori inclusion criteria that allowed us to precisely select experts with significant experience of teaching through simulation in EM. An important number of experts was reached, with varied teaching profiles (academic and non-academic) and good national representativeness. Second, the number of experts dropped from 77 to 66 between the first and third rounds, for reasons we have not investigated. Third, the high number of questions could have made the experts weary. The random order of questions limited this evaluation bias and the halo effect. Fourth, the experts may not have given the same scores in each round. Whether they were consistent throughout the entire Delphi process is unknown. However, the correlation between the scores given in rounds 1 and 3 was good. This validates our method of immediately selecting or excluding items with an excellent or poor score, respectively. Finally, this curriculum was developed by French experts for French EM residents and may not be applicable in some countries with different healthcare systems. However, the only French specificity is the pre-hospital aspect of EM, and other countries have the same pre-hospital organization of care.

Conclusions

Using a Delphi method, French experts in simulation-based EM education have selected 24 TS and 20 NTS to be taught as a priority to EM residents, using simulation. The aim of this curriculum is to tend towards uniformity and standardization of training, while allowing teachers to keep some level of flexibility to set up courses according to their local human and material constraints.