Adult essential extracorporeal membrane oxygenation (ECMO) skills for use in an e-learning program for ICU physicians, nurses and perfusionists: a consensus by a modified Delphi questionnaire

Abstract

Background

Education in ECMO starts with basic theory and physiology. For this type of training, self-assessment e-learning modules may be beneficial. The aim of this study was to generate consensus on essential ECMO skills involving various professional groups involved in caring for ECMO patients. These skills can be used for educational purposes: development of an e-learning program and fine-tuning of ECMO-simulation programs.

Methods

Experts worldwide received an e-mail inviting them to participate in the modified Delphi questionnaire. A mixture of ECMO experts was contacted. The expert list was formed based on their scientific track record mainly in adult ECMO (research, publications, and invited presentations). This survey consisted of carefully designed questionnaires, organized into three categories, namely knowledge skills, technical skills, and attitudes. Each statement considered a skill and was rated on a 5-point Likert-scale and qualitative comments were made if needed. Based on the summarized information and feedback, the next round Delphi questionnaire was developed. A statement was considered as a key competency when at least 80% of the experts agreed or strongly agreed (rating 4/5 and 5/5) with the statement. Cronbach’s Alpha score tested internal consistency. Intraclass correlation coefficient was used as reliability index for interrater consistency and agreement.

Results

Consensus was achieved in two rounds. Response rate in the first round was 45.3% (48/106) and 60.4% (29/48) completed the second round. Experts had respectively for the first and second round: a mean age of 43.7 years (8.2) and 43.4 (8.8), a median level of experience of 11.0 years [7.0-15.0] and 12.0 years [8.3-14.8]. Consensus was achieved with 29 experts from Australia (2), Belgium (16), France (1), Germany (1), Italy (1), Russia (2), Spain (1), Sweden, (1), The Netherlands (4). The consensus achieved in the first round was 90.9% for the statements about knowledge, 54.5% about technical skills and 75.0% about attitudes. Consensus increased in the second round: 94.6% about knowledge skills, 90.9% about technical skills and 75.0% about attitudes.

Conclusion

An expert consensus was accomplished about the content of “adult essential ECMO skills”. This consensus was mainly created with participation of physicians, as the response rate for nurses and perfusion decreased in the second round.

Background

Extra Corporeal Membrane Oxygenation (ECMO) is increasingly used in modern Intensive Care Units (ICU’s), especially since the publication of the ECMO to rescue Lung injury in severe ARDS (EOLIA) trial, and H4N1 Influenza A and Severe Acute Respiratory Syndrome Coronavirus 2 (SARS-CoV-2) pandemics.[1,2,3,4,5] As this support becomes more widespread, there is also a growing need for proper training of physicians, perfusionists and ECMO nurses.[6,7,8] Taking care of ECMO-patients is not only technically demanding, but requires knowledge of certain pathophysiology. To master adequate problem solving in case of emergencies, different educational training programs with certification are organized worldwide. [9,10,11,12]

ELSO lists in its guideline all topics which an “ECMO-specialist”-program should cover, including theoretical concepts and hands-on water-drills. To start with, each participant must master the physiology and circuit components. The program needs to provide technical aspects like cannulation guidance, circuit changes and decannulation strategies. Cooperation from the different stake holders is trained with special attention to daily management, human factors and ethical considerations [8] These training sessions may result in a team whose ultimate goal is to discharge the patient with optimal quality of life. As indicated by the ELSO guideline, each ECMO center should develop center specific guidelines and policies for training ECMO specialists. Since these guidelines were published in February 2010, it is reasonable to suppose that they need updating. Teaching ECMO nurses, specialty registrars, clinical fellows and perfusionists and updating members of staff is time consuming not only for the tutors but also for the learners. In order to improve the accessibility and flexibility, developing an E-learning module can be the strategy to opt for.[11, 13] The aim of this study was to generate consensus on essential ECMO skills for developing an e-learning program and fine-tuning of the existing ECMO-simulation program at Ghent University Hospital, Belgium. A modified Delphi questionnaire approach was used.

Methods

Study design

This is a prospective international study using the modified Delphi methodology. The Delphi method was developed in the 1950s as a part of a US military defense project.[14] The modified Delphi technique is a structured consensus method, used in medical literature to accomplish an overall agreement or expert opinion about definitions, problems or other ideas. This method ensures that each participant can make judgments in complete anonymity, on an equal footing with the other participants, and can change his or her mind during the process. Experts are asked to give their opinion on statements in successive rounds. The evaluation is done by using an ordinal scale (e.g., Likert scale) for each statement, next to qualitative commentary in a text field. Until a certain level of consensus is reached, the procedure can be repeated. [14,15,16] In this study we used a five-point Likert scale. Consensus about a key competency in this study was defined by 80% of experts who rated a skill with a score of 4 or more and when the internal consistency shows a Cronbach Alpha score > 80%. This study aimed to accomplish consensus in two or three rounds.

Experts

An expert was defined as an experienced health care professional (HCP), working in a high volume ECMO center. The high volume was estimated to be at least 20 cases a year. The physicians were active in intensive care unit (ICU), anesthesia, cardiac surgery, cardiology, and internal medicine. ICU nurses and perfusionists have also been interrogated. At the start of this survey 106 experts were contacted by e-mail. A mixture of ECMO experts was contacted. The expert list was formed based on their scientific track record mainly in adult ECMO (research, publications, and invited presentations).

Questionnaires

Panel members were contacted by e-mail with information about the study and the purpose of the questionnaire. They were invited by an auto-generated e-mail to participate in the survey, which was created in the web-based software platform Research Electronic Data Capture (RedCap®). The research group consisted of two physicians (HP, IVH) and one perfusionist (FDS), experienced in ECMO and/or educational research from our center. This research group selected a list of statements based on several sources: “Extracorporeal Life Support: The ELSO Red Book (5th Edition), the “ECMO specialist training manual (4th Edition), the handbook of the course “ECMO-course for physicians and nurses” of the Leiden University Medical Center and the educational program from the Ghent University Hospital. The competency areas were grouped into three categories, namely knowledge skills, technical skills, and attitude. In the first round, experts were asked to score statements covering knowledge skills, technical skills, and attitude. Each skill was rated on a 5-point Likert-scale from ‘1, Strongly disagree’ to ‘5, Strongly agree’. For each topic, the panelist could make a proposal to change or nuance the statement. Pilot testing was done by a small group of individuals, who did not participate in the survey. After the first round, results were anonymously analyzed, and the suggested adjustments were applied. Based on the information obtained from round one, new statements and modifications proposed by panelists were collected and discussed in the research group to rephrase for clarity. The distributions of scores (median and interquartile range) for each statement from the first round were included within the second-round questionnaire. The experts were instructed in the second round to re-consider the statements presented in the first round, or slightly different statements and the newly added topics. In addition, a mock question was included. A mock question is inserted in a survey to check whether the participant is still focused, the answer is obvious, it ensures that the survey was not randomly completed.

Data collection and data analysis

The completed surveys were collected by the software platform RedCap®. Information on demographics and expertise was collected through a questionnaire in the first round of the survey. Panelist were given six weeks to complete the study, with reminders sent out every week to non-responders. The second-round questionnaire was only sent out to experts who had completed the first round. Normal distribution of the responses was checked with the Shapiro Wilk test. Normal distributed variables are reported as mean (standard deviation), and non-normal distributed variables as median [interquartile range]. To determine the ranking of topics and skills from the questionnaire, median values of the panelists’ scores were used. The Cronbach’s Alpha score, for which an alpha value of 0.80 was chosen as an indicator of consensus, was calculated to test internal consistency.[17] The intraclass correlation coefficient (ICC) was used as the reliability index for interrater consistency and agreement. [18] To compare differences between educational backgrounds, the Kruskal-Wallis test was used, given the multiple testing, correction was made using the Bonferroni method. Wilcoxon-signed ranks test was used to compare ratings of the elements between the first and the second round. A p-value of 0.05 is determined as statistically significant. Statistical analysis was performed using SPSS 26.0 (Statistical Package for the Social Sciences, IBM Company, US).

Ethics

This is a prospective, observational study in which the International Council for Harmonization (ICH) Good Clinical Practice (GCP) guideline is followed. Approval by the local research medical Ethics Committee of Ghent University Hospital was obtained on 30/07/2020 (reference number BC-07929). All participants had to give written consent prior to the start of the study and anonymity has been guaranteed.

Results

Demography

Of the 106 invited panelists, 54 started the survey, and 48 (45,3%) completed the first round. Fourteen of them were female (29.2%). The participants had a mean age of 43.7 (8.2) years with median experience of 11.0 [7.0-15.0] years. They worked in a unit with a median number of 45.0 [25.0-64.5] ICU beds and 94% of them worked in academic setting. The panel consisted of four (8.3%) nurses, 12 (25.0%) perfusionists and 32 (66.7%) physicians. 50% of them practiced in Belgium. In the second round the response rate was 29/48 (60.4%); 21 (72%) physicians, three (10%) nurses and five (17%) perfusionists completed the second round. Consensus was achieved with 29 experts from Australia (2), Belgium (16), France (1), Germany (1), Italy (1), Russia (2), Spain (1), Sweden, (1), The Netherlands (4).” (See Tables 1 and 2).

Table 1 Demographics and working experience

Table 2 Countries where panelist were practicing

Delphi results

In the first round, experts were asked to score 56 statements covering 33 knowledge skills (including one mock question), 11 technical skills, and 12 attitudes. After analysis of this first round, four knowledge statements were added, and 46 statements were reformulated. Of the knowledge skills 28 (84.8%) skills were rephrased, all (100%) statements about technical skills were adapted, and seven (58.3%) attitude statements were changed. The changes ranged from minimal additions such as specifying who within the team should master the skill and clarifications of the statement. Consensus was accomplished in the first round in 45/56 (80,4%) of the statements, in the second round in 54/60 (90%) of the statements. There was excellent reliability when calculating Cronbach’s alfa, for the first round it was 0.921 and the second had 0.907. Regarding the first round, there was a significant interrater agreement between all 48 experts (ICC 0.861 with p < 0.001). The same was found in the second round with a significant interrater agreement between all 29 experts (ICC 0.942 with p < 0.001). For eight statements we found a significant difference in ratings between the first and second Delphi questionnaire, they were adjusted in between rounds. (Supplementary material)

Knowledge

For the knowledge topics, in the first-round consensus was achieved in 90.9% of the topics and for 94.4% of the topics in the second round. Cronbach’s alfa was 0.909 for the first round and 0.864 for the second round. Two of the 36 knowledge topics could not be retained as key competencies and were removed from the list: “Pre-ECMO evaluation including RESP-score and SAVE-score (RESP = Respiratory Extracorporeal membrane Oxygenation Survival Prediction, SAVE = Survival After VA ECMO).” and “Knowledge of correct ECMO nomenclature.”

Also, the mock question was deleted. The three most important skills for knowledge were ranked as follows: (1) “Knowledge of ECMO physiology: in VA-ECMO: Optimization of hemodynamic support, including blood flow, native cardiac function.”, (2) “Knowledge of ECMO physiology: Optimization of pCO2.” and (3) “Knowledge of symptoms and clinical signs of limb ischemia and prevention of it.”. See Table 3.

Table 3 Top 3 rated skills for each of three categories and the skills without consensus

Technical skills

Cronbach’s alfa was calculated 0.809 and 0.707, respectively for the first and the second round. In the first round there was only agreement for 54.5% of the skills, all statements have been reformulated. In the second-round consensus was achieved in 90.9% of the skills. This resulted in the removal of one technical skill: “For the physician: be able to measure the vessel diameter and perform ultrasound guided puncture in peripheral ECMO.” from the list, see also Table 3. The top three technical skills are: (1) “For the perfusionist: being able to change the circuit/oxygenator.”, (2) “For the perfusionist: priming of the circuit.”, (3) “For the physician and perfusionist: correct insertion/connection of the cannulas.”

Attitude

In the category attitude, experts agreed in 75% of the statements in the first round and second round. Three of the 12 statements (25%) were not considered to be key competencies and were discarded, see Table 3. Calculated Cronbach’s alfa for attitude topics. was 0.714 and 0.778, respectively for the first and second round. The following statements made the top three: (1) “Know his/her limits and call for help if needed.”, (2) “There should be an experienced team available 24/24 7/7 for trouble-shooting.” and (3) “Be able to consider the risks/ benefits for every ECMO run.”.

Differences between professional groups for ranking a skill as an “essential skill”, were noted in 12 statements, and only in round one (Table 4).

Table 4 Skills rated differently between the three participating professional groups

A list of all the statements of the consensus can be found in Table 5.

Table 5 Statements of the first round (R1) and second round (R2) distribution parameters and consensus percentage

Discussion

In this International Delphi consensus on key elements for an e-learning program for ICU personnel taking care of patients treated with ECMO we identified 34 knowledge items, 10 technical skills, and 9 attitudes.

Experts, who participated in this modified Delphi questionnaire, came from Europe and Australia, with more than 10 years’ experience in a dominantly academic setting. The panel consisted of nurses, perfusionists and physicians. Our center will certainly not be the first to implement E-learning in ECMO education. In 2017, already 36% of ECMO simulation sites in the USA reported computer-based self-assessment learning modules for ECMO-practitioners.[11] The benefits could be that this is a learner centered way of teaching, resulting in an active learner in a psychological safe environment, working at his own pace, where knowledge can be tested with immediately given feedback.[19,20,21].

The modified Delphi study

The modified Delphi technique has already proven its use in health care disciplines for example in developing fundamental skills, safety behavior or key interventions and quality indicators. [22,23,24]

This study’s response rate of 45.3% in the first round and 60.4% in the second round is comparable to Maertens et al. with a response rate of 43%. But in contrast, Hoste et al. had a response rate of 90% in a study in which only 20 experts where invited. Mostly a mix of backgrounds of professionals is used. In line with Hoste e.a., this questionnaire invited nurses and physicians.[22,23,24].

Internal consistency was “excellent”, as shown by the calculated Cronbach’s alfa scores. A possible explanation for this can be motivational influence, as the panelist participated voluntary and may have felt responsible to achieve important conclusions. The scores for the different competency areas (knowledge, technical skills, and attitude) decreased from the first to the second round, because of a lower number of panelists.[25] For knowledge only, the internal consistency was “good”, for technical skills it decreased from “good” to “acceptable”. For attitudes it increased, but stayed in the range of “acceptable”.[17] We also showed a good intraclass correlation with a significant interrater agreement, meaning that the experts had the same expertise or gave homogeneous answers.[14, 26].

Fundamental knowledge skills

The top three most important knowledge skills indicate that the experts value the knowledge of ECMO physiology which is in line with existing ECMO training programs. [20, 27, 28] Recognition of limb ischemia due to cannula placement has been included in the top three of knowledge skills. This can indicate that prevention, early recognition and treatment stays important because it is peripheral placed veno-arterial ECMO care specific, more than e.g. the recognition of neurological problems or infection. Focus on recognition of limb-ischemia is, as far as we know, always included in ECMO-training programs. [28, 29]

Final consensus on 34 of 36 knowledge topics was achieved. These are topics that have been clearly described in the standard ECMO manuals registered by ELSO guidelines and in “ECMO-course for physicians and nurses” of the Leiden University Medical Center, comprising basic pathophysiology, mechanical-human interaction and problem-solving for life-threatening situations.[8, 19] In contrast, the following topics were not included in our knowledge topics: “the history of ECMO”, neither specific neonatal or pediatric topics, nor congenital heart pathologies. In contrast, knowledge of the different prediction scores was not retained as core competency. This may be explained by the fact that scores seldom directly affect daily practice of ECMO-patients. Although these topics were not withheld in another paramedic and registrar training, they are mentioned as a possible predictive tool in “the ELSO Red Book”. Moreover, correct indications may optimize the outcome scores and will increase the value of ECMO as a recommendable salvage therapy; correct tariffication and related data registries should evaluate quality-based programs and justify proper financing.[30, 31].

Fundamental technical skills

The consensus in the first round regarding the proficiency of technical skills was low 54% but rose to 90% in the second round. The adaptation of the statements has led to a significantly higher consensus among the experts, 10/11 were in fact retained as essential. The panelists gave the following statements the highest scores: “For perfusionist; being able to change the circuit/oxygenator.“, “For the perfusionist: being able to prime the circuit.“, “For the physician and perfusionist; being able to insert/connect the cannulas correctly.“ Apparently, all respondents value the dedicated service of the perfusion department in supporting device maintenance and set-up, managing technical problems, implementing safety checks, and optimizing best practice. Statements on technical aspects were thoroughly commented and were modified and shifted in emphasis consequently. The selected technical skills are in line with technical trainings described by other authors and so they are brought together and confirmed by our modified Delphi.[11, 28, 32, 33] This study adds that the different responsibilities and tasks within these technical operations become more explicit in contrast to the position paper of the ECMO-net, that compiles different technical skills and puts emphasis on working in multidisciplinary teams. [4] Raffelli e.a. follow a “dual provider” model, wherein specific tasks are dedicated to specific professional groups, but these specific technical skills are not listed for each profession. [20]

Fundamental behavioral skills

Consensus was reached in 75% of the statements for both rounds: nine attitudes were scored essential to team members in daily ECMOcare. Emphasis was put on emergency situations, safety principles and weighing the risk-benefit balance, with top ranking for: “learn to assess your own knowledge and skills correctly” and “call for help when necessary”. The consensus also highlights the need to have a well-trained ECMO team available 24/7. The fact that the expertise can only increase when there is sufficient exposure, emphasizes that this supportive treatment should be offered in specialized centers.[1, 34] Although e-learning may be more learner-oriented, the e-learning should emphasize the importance of the team approach, in which each individual specializes in his professional actions and behaviors. The behavioral skill “to ask for an informed consent”, which also is proposed by the ELSO guidelines for training ECMO-specialists, did not reach consensus of the experts, but in contrast all other attitudes defined as essential are not part of the ELSO guidelines and were proposed by the research group based on the experience of the Ghent University Hospital ECMO simulation team.

The results of this Delphi Consensus are in parallel with key aspects of good practice presented by the International ECMO network and the ELSO.[4, 19, 20] The ELSO guidelines grew upon expertise in teaching HCP trough the last decades and educational programs were developed on expert opinion. This guidelines date from 2010 and served well in developing this Delphi questionnaire.

Differences in scoring depending on educational background

In the first round, twelve statements were retained with a significantly different grading between the experts with different background, but in the second round, these differences were no longer significantly different. This may be explained by rephrasing the statements and sharing the scores obtained in the first round allowing experts to reach consensus Additionally, the lower response rate in the second round with fewer perfusionists may have influenced this result. There were 32 vs. 21 physicians participating, 4 vs. 3 nurses and 12 vs. 5 perfusionists participating in the first and second rounds, respectively.

Limitations

We want to discuss the following limitations to this study. Only 60.4% (29 of 48) of the initial participants, responded in the second round, possibly leading to selection bias. The dropouts possibly can be explained by a global surge in the COVID 19-pandemia, which occurred when the experts were asked to fill out the second round.[3, 35] At the start of this survey 106 experts were contacted by an automatic generated e-mail by the software RedCap®. Given the initial low response rate in the first round, we contacted some of the invited colleagues, who indicated not have been contacted by e-mail from RedCap®. RedCap® technical support clarified that the generated e-mails may have been blocked by hospital firewalls. Another limitation of the study is that no experts from the USA, Latin America or Asia participated.

Experts in the US were contacted for the first round, none participated in the Delphi questionnaire, one did send an email that he liked the design but was too busy with a considerable amount of clinical work.

Also important to mention is the low rate in nurses and perfusionists that collaborated in the Delphi questionnaire. Seven out of ten experts were physicians in the second round.

The number of questions may also have deterred experts from participating.

Conclusion

Although ECMO is frequently used to support patients in ICU, the process needed to determine essential educational topics has not been published. By establishing this modified Delphi consensus, an expert opinion was achieved about the content of “ECMO essential skills” among physicians, ICU nurses and perfusionists anno 2021. This consensus was mainly created with participation of physicians, as the response rate for nurses and perfusion decreased in the second round. In the topics knowledge, technical skills, and attitudes essential skills were identified by experts and are now used to guide the development of an e-learning module. These essential skills will also enhance simulation scenarios in the hands-on training sessions. Whether this e-learning and simulation-based training will provide better care, is the subject for a subsequent study.