Validation of an instrument for the evaluation of exchange transfusion (INEXTUS) via an OSCE

Abstract

Introduction

Exchange transfusion is the treatment of choice for patients with severe hyperbilirubinemia who do not respond to phototherapy. This procedure is highly complex and requires substantial expertise to perform, however it´s not done frequently enough to guarantee adequate training. Traditional learning scenarios do not have a space reserved for teaching this procedure or an instrument that fully and objectively evaluates the skills that a professional must acquire. 

Objectives

The purpose was to construct and evaluate the INEXTUS instrument´s validity evidence relevant to internal structure, in a simulated scenario through the performance of an objective structured clinical exam (OSCE). 

Materials and methods

The Delphi consensus methodology was utilized to design the instrument; six experts participated through three rounds using the Google Forms platform. The categories and items previously obtained were subjected to validation by nine experts through a dichotomous survey. Prior to data collection, the evaluators were trained through a pilot test with 10 medical students. Subsequently, all residents of a paediatric programme were evaluated through the OSCE methodology in a simulated scenario, with 6 stations, of a clinical case of a new-born with an explicit need for exchange transfusion. During their participation in the scenario, the residents were first evaluated with the instrument developed. Additionally, audio and video filming of all students who participated was performed with the aim of conducting a second evaluation two weeks after the first four evaluators participated.

Results

The final INEXTUS instrument consists of 46 subitems grouped into 23 items divided into 6 categories, demonstrating an inter-rater intraclass correlation coefficient of 0.96 (95% CI 0.94, 0.98 p-value < 0.001). For the Fleiss Kappa of the 23 items evaluated, concordance was evaluated for 14 items but could not be determined for the 9 remaining items because all the ratings were equal, either because the items were not performed or they were all performed adequately. Of the 14 items, 9 good scores were obtained (95% CI 0.61 to 0.8; p value < 0.001), and 5 very good scores were obtained (95% CI 0.81 to 1; p value < 0.001).

Conclusions

The INEXTUS instrument evaluates exchange transfusion skills in medical personnel in training in simulated scenarios using the OSCE methodology; it has high validity and reliability and is a high-impact educational tool.

Introduction

Jaundice is the leading cause of hospitalization in neonatal intensive care units (NICU) [1]. Patients who present severe hyperbilirubinemia and do not respond to initial management with phototherapy have a high risk of bilirubin-induced neurotoxicity, requiring immediate management by exchange transfusion, a highly complex procedure that leads to adverse events in 67–87% of cases, depending on the expertise of the professional and the gestational age of the patient [2].

Exchange transfusion consists of replacing the neonatal blood volume (recommended that it be a double volume). The goal is to lower serum bilirubin levels and remove circulating maternal antibodies from circulation. The choice of donor blood type depends on the hemoclassification of the mother and the neonate. There are different routes for carrying out the procedure. To carry out the procedure, it must be considered to connect sequentially to the line: catheter for drainage of extracted blood, that of donor blood and a syringe pre-filled with saline solution. The exchange volume of each aliquot should be 5–10% of the infant's blood volume [3, 4].

The advent of new methods to prevent jaundice from Rh incompatibility, such as the administration of anti-D immunoglobulin and the development of advanced noninvasive technology for the treatment of jaundice such as phototherapy, has led to a decrease in the need for exchange transfusion, especially in developed countries, where exchange transfusion rates are close to 1.9 per 100,000 live births [5, 6]. Second, there has been a lack of competence in the skills of professionals to perform this procedure, given the limited opportunities during postgraduate education and later in paediatric practice, which leads to an increase in adverse effects such as infection, cardiac arrest and death [6].

Clinical learning scenarios do not have a space reserved for teaching this procedure or a method that fully and objectively evaluates the skills that the professional must acquire. The Objective Structured Clinical Examination (OSCE) arises as a methodology aimed at evaluating in an objective, planned and structured way the components that make up clinical skill in a simulated scenario [7]. The OSCE is presented as a valid and reliable instrument for the evaluation of clinical components of the medical profession, for which it is necessary to appropriately design both the test and the instrument that evaluates the acquired skills [8,9,10,11].

The instrument ensures that the acquisition of skills is evaluated objectively, avoiding biases or impartial and unfounded judgements [12, 13]. An evaluative instrument can be useful to score clinicians frequently and guarantee the quality of the procedure, compare the skills between doctors, ascertain retentive skills and provide feedback during training [12].

Both a simulated scenario and an instrument that evaluates the performance of the student in itself is especially relevant in clinical practice scenarios that occur sporadically and in those of high complexity [14]. The exchange transfusion procedure is high-complexity procedure, and despite being performed infrequently, if indicated, it changes the neurological outcome and even prevents the death of patients severely compromised by hyperbilirubinemia [15, 16]. Despite the existence of multiple management guidelines in which the exchange transfusion technique is systematically explained in detail, after conducting a literature search on the subject, an instrument validated to measure paediatric residents´neonatal transfusion skills is not available.

The objective of the present study is to construct and evaluate validity evidence relevant to internal structure an instrument that evaluates skills in exchange transfusion and to evaluate its validity and reliability in a simulated scenario through an OSCE. The rubric designed will support the development of the research protocol published in Trials Maldonado et al. Trials (2020) 21:387. https://doi.org/10.1186/s13063-020-04312-3 and registered in clinical trials 18/08/2019 (Trial registration: NCT04070066).

Methodology

Development of INEXTUS

Phase 1: Bibliographic search and definition of need for designing an instrument

The study began with an extensive literature review on the evaluation instruments of the exchange transfusion procedure in Medline, Scopus, Cochrane, LILACS and SciELO, determining that to date, there is no related instrument. The need for designing an exchange transfusion evaluation instrument (INEXTUS) is defined using the Delphi-type consensus methodology of multiple, individual rounds and without contact between the experts consulted. Figure 1 shows the Flowchart Intexus development Phases.

Fig. 1

Flowchart Inextus development phases 

Phase 2: Selection of the panel of experts

A first preselection of 30 expert neonatologists or paediatricians came from different hospitals, with care experience in medium- and high-complexity neonatal units was proposed, selecting 12 who practiced in university hospitals with teaching experience and who had performed the exchange transfusion procedure during their professional practice. The pre-selected experts were invited to participate, and 6 of them agreed to participate.

Phase 3: Development of the Delphi consensus

Three rounds were conducted. First, a survey was sent to each expert via Google Forms®. The authors used an open-ended question to instruct the experts to list the skills for exchange transfusion. Once the results of the first round were in, the authors compiled and categorized the skills, constructing a general matrix from which the second round was carried out. In this questionnaire, all items were listed with questions on a 5-point Likert scale and to include an item definitively in the instrument, a cut-off point of 80% or more was established for totally agree and agree; and those items with 80% or more for totally disagree or disagree were excluded. After this round, a total of 25 items were excluded, obtaining a preliminary instrument with 3 categories, 18 items and 20 subitems. In the third round, the research group submitted the questionnaire again to the experts for their evaluation, asking their agreement on including each item through a dichotomous survey (yes or no). In this round, only one item was excluded.

Phase 4: Validation of experts

A new group of 9 expert neonatologists or paediatricians was contacted to send them the instrument for their consideration of whether they agreed with each of the items included. A survey was sent via Google Forms® with questions on a dichotomous scale of Yes or No responses. In this survey, they could make additional comments where they could suggest including an item or reorganizing the categories and related items. After this expert validation, an instrument of 6 categories, 23 items and 46 subitems was obtained.

Phase 5: Pilot test

Prior to data collection, the evaluators were trained in the operation of the instrument through a pilot test with 10 final year medical students so the evaluator could assess their comprehension of the instrument and the feasibility of applying it.

Population, procedures and scenario

The study population corresponds to residents enrolled in the paediatrics programme of the Universidad de La Sabana. It was developed in the Simulated Hospital of the Universidad de La Sabana during the month of January 2020. The Simulated Hospital is an academic and research centre that teaches clinical simulation with high-fidelity simulators in simulated clinical scenarios. The pediatric residents included in the study received training prior to the start of data collection. For this, an educational strategy was developed that consisted of a video and guide of the procedure and recommended references for individual reading. Students who participated in the study signed an informed consent, and it was clarified that their participation in the study would not affect their academic situation and that the confidentiality of the data would be guaranteed.

All 24 residents underwent the evaluation and application of the checklist for a standardized clinical case according to the OSCE methodology.

The clinical case scenario was a term neonate with normal weight for gestational age of 13 h of life, blood group incompatibility, clinical jaundice, haemolysis and signs of hyperbilirubinemic encephalopathy who was with the mother.

This evaluation scenario consisted of six stations, (Fig. 2: Flow chart OSCE stations) that evaluated each subtopic of the evaluation instrument. Patients (simulated mother) participated in stations one and two, head nurses in station five, and high-fidelity neonatal simulators with monitors and equipment for exchange transfusion. Figure 3 shows the simulated scenario.

Fig. 2

Flowchart OSCE stations

Fig. 3

simulated scenario

The evaluators responsible for applying the instrument were professors from the School of Medicine and the Universidad de La Sabana Clinic, two neonatologists and two paediatricians, with experience in the procedure and in medical education in simulated scenarios.

During the participation of the paediatric residents in the scenario, the first assessment was carried out by the evaluators using the instrument developed (INEXTUS—Exchange transfusion evaluation instrument, Universidad de La Sabana). In order to perform the proof reproof intra-raters reliability, audio and video filming of the entire participation of the student in the scenario was made with the objective of performing the second assessment two weeks after the four evaluators first participated.

Statistical analysis

A sample size was calculated for inter-rater and intra-rater reliability, taking into account an expected intraclass correlation coefficient (ICC) of 0.8, with a 95% confidence interval (CI) of 0.2 and alpha error of 0.05, power of 80% and with four observers a total of 24 participants [17].

Each item of the instrument was scored dichotomously: 0 (did not meet or partially met) and 1 (all met). The score of each subtopic was obtained by dividing the student’s score by the maximum possible and multiplying it by 100. Similarly, the total score was obtained from the score of each subtopic divided by the possible total and multiplied by 100. For inter-observer and intra-observer reliability, the ICC was calculated with two mixed factors and single measures with the 95% CI for the percentage of the total score of the instrument. Likewise, to evaluate whether agreement of the classification was partially met and/or not met (0) and all met (1) for the inter- and intra-evaluator test–retest, the Fleiss Kappa coefficient with 95% CI and the Kappa coefficient 95% CI were calculated, respectively.

The recording and analysis of the information was performed in Microsoft Excel for Windows 10 and IBM SPSS Statistics 26 software. A value of p < 0.05 was established as a value of significance.

Results

Twenty-four residents of the paediatrics programme participated, 6 of whom had completed their studies and not yet graduated. The sociodemographic characteristics are summarized in Table 1. The four evaluators responsible for applying the instrument were professors from the School of Medicine and the Universidad de La Sabana Clinic (two neonatologists and two paediatricians), with experience in the procedure and in medical education in simulated scenarios (Table 2).

Table 1 Characteristics of the participants 

Table 2 Residents rating (%) for each station

Assessment instrument

A total of 9 neonatologists with academic and healthcare experience participated in the consensus rounds. One was head of department in their School of Medicine, and four were coordinators or heads of neonatal units. All exercised their clinical practice in university hospitals, responsible for training paediatric residents.

The final instrument incorporated the different aspects of the exchange transfusion procedure, from evaluating and identifying patients who require the procedure, to patient preparation and stabilization, requesting tests and necessary equipment, to managing the parents and the technique of the procedure.

The final instrument to assess skills in exchange transfusion (INEXTUS) consists of 46 subitems grouped into 23 items, divided into 6 categories: Clinical history and risk approach of patients with neonatal jaundice, Comprehensive assessment of the patient with jaundice, Definition of clinical behaviour, Preparing equipment for the procedure, Perform the exchange transfusion, Inform and manage the parents. In the pilot test, the feasibility of the application of the instrument and its easy and rapid comprehension by the evaluator was observed (see Table 3 INEXTUS Instrument (Instrument of exchange transfusion of La Sabana)).

Table 3 Final INEXTUS instrument

Reliability

The inter-rater intraclass correlation coefficient was 0.96 (95% CI 0.94, 0.98 p value < 0.001). The test–retest intra-rater reliability is shown in Table 4, obtaining excellent reliability values.

Table 4 Intra-rater test–retest reliability

Table 5 shows the Fleiss kappa coefficient for the agreement between evaluators of each item at each evaluated station. Of the 23 items, agreement for 14 items was possible to calculate since in the remaining 9, the assessments by the four evaluators was equal (evaluators assigned an equal rating to all the residents,either complied or not complied). Table 6 shows the intra-rater agreement (evaluator 1) test–retest for each item at each station, finding moderate to high agreement for 14 of the 23 items evaluated.

Table 5 Fleiss Kappa coefficient for each station item (inter-rater reliability)

Table 6 Kappa coefficient for each station item (intra-rater reliability for evaluator 1)

Discussion

Through this work, we constructed and reviewed targeted validity evidence of the INEXTUS instrument intended to evaluate the skills of residents in training in exchange transfusion under simulated scenarios using an OSCE for its high validity and reliability, thus promoting learning through the realistic recreation of scenarios [18, 19]. The reliability estimated by the intraclass correlation coefficient and by the Fleiss Kappa is found reasonably reliable and reproducible ranges for use in the training of paediatric graduate studies. Validating the rubric allowed us to refine the content and make decisions about the items to be included. These data are important as input for generating a controlled educational environment based on clinical simulation in the exchange transfusion procedure [20].

Opportunities in daily practice to develop skills in exchange transfusion are scarce, and paediatric professionals do not reach a sufficient level of competence to master this procedure in the clinical setting [21]. Clinical simulation scenarios are proposed as a valid, reliable and effective teaching option for skills acquisition in graduate-level medical personnel in training [22, 23].

Exchange transfusion is a highly complex procedure that is performed in neonatal intensive care units and under specific medical indications, so it requires a high level of expertise [24]. To date, there are no reports in the literature of the development of a rubric to assess the exchange transfusion procedure; therefore, the instrument developed (INEXTUS) will allow a solution to a known problem in the training of future paediatricians, that together with the simulated scenario is an appropriate strategy for the development and evaluation of these skills [25, 26].

The assessment of clinical skills is a key objective in the training of human talent in health; however, it is a challenge since knowledge, skills and attitudes must be evaluated simultaneously. A continuous and comprehensive assessment drives learning, recognizing that it is a complex process that requires a consistent methodology through valid and reliable instruments [27]. The rubric designed for exchange transfusion allows assessing theoretical skills, communication, procedural skills, among others, allowing teachers and students to identify the skills and abilities that should be reinforced in a more granular way, similar to what has been described by Smith and collaborators during the development of a rubric to evaluate students during assessing virtual patients [28].

The present study has some limitations. The research was limited to 2 sources of validity evidence- test content and internal structure. Although a sample size was calculated for the main outcome, the number of students included may be low, and only pediatric residents from one university were included. This limits the generalizability of the results to other settings. Similarly, all residents were evaluated without stratifying the level of training, which could impact the results. However, the objective was to evaluate the reliability of the instrument, observing the reproducibility of the results when applied by the same evaluator, by different evaluators and at different times.

In this context, the future use of INEXTUS is recommended at stratified training levels, with a greater number of participants and from different universities and even scenarios (it may be already trained personnel). As strengths of the present study, the development of an instrument with content constructed through an in-depth search of the literature and the contribution of experts in the exchange transfusion procedure is identified, which supports its valid content. There were no problems during the evaluation in the simulated scenario or during the data collection. Within the limitations, it was not possible to directly evaluate evidence relevant to relationships to other measures since there is no instrument with which to compare it.

Conclusions

The exchange transfusion instrument is validated based on reliability and its construction to be used to evaluate students during the development of an of exchange transfusion OSCE. The results obtained show that the validity of the instrument and the intra- and inter-rater reliability were achieved.

This instrument will allow the evaluation of paediatric students when performing the exchange transfusion procedure, providing teachers with objective information about their performance. Likewise, when the evaluation is developed in an OSCE, students can be trained, and objective feedback can be given to the participants.