Trainees aiming to specialize in Neonatal Perinatal Medicine (NPM), must be competent in a wide range of procedural skills as per the Royal College of Canada. While common neonatal procedures are frequent in daily clinical practice with opportunity to acquire competence, there are substantial gaps in the acquisition of advanced neonatal procedural skills. With the advent of competency by design into NPM training, simulation offers a unique opportunity to acquire, practice and teach potentially life-saving procedural skills. Little is known on the effect of simulation training on different areas of competence, and on skill decay.
We designed a unique simulation-based 4-h workshop covering 6 advanced procedures chosen because of their rarity yet life-saving effect: chest tube insertion, defibrillation, exchange transfusion, intra-osseus (IO) access, ultrasound-guided paracentesis and pericardiocentesis. Direct observation of procedural skills (DOPS), self-perceived competence, comfort level and cognitive knowledge were measured before (1), directly after (2), for the same participants after 9–12 months (skill decay, 3), and directly after a second workshop (4) in a group of NPM and senior general pediatric volunteers.
The DOPS for all six procedures combined for 23 participants increased from 3.83 to 4.59. Steepest DOPS increase pre versus post first workshop were seen for Defibrillation and chest tube insertion. Skill decay was evident for all procedures with largest decrease for Exchange Transfusion, followed by Pericardiocentesis, Defibrillation and Chest Tube. Self-perceived competence, comfort and cognitive knowledge increased for all six procedures over the four time points. Exchange Transfusion stood out without DOPS increase, largest skill decay and minimal impact on self-assessed competence and comfort. All skills were judged as better by the preceptor, compared to self-assessments.
The simulation-based intervention advanced procedural skills day increased preceptor-assessed directly observed procedural skills for all skills examined, except exchange transfusion. Skill decay affected these skills after 9–12 months. Chest tube insertions and Defibrillations may benefit from reminder sessions, Pericardiocentesis may suffice by teaching once. Trainees’ observed skills were better than their own assessment. The effect of a booster session was less than the first intervention, but the final scores were higher than pre-intervention.
Pediatric residents lack key resuscitative and critical life-saving skills at the end of their training [1, 2]. Only 17% were confident enough to independently conduct neonatal resuscitations, despite the Royal College of Canada (RCPSC) mandated pediatric competencies at the end of training [3,4,5]. Neonatal Perinatal Medicine (NPM) subspecialty trainees (pediatricians seeking specialized training) must be competent in an even wider range of procedural skills . While NPM trainees perform better than junior residents in common neonatal procedures, there are still substantial gaps in the performance of more advanced procedural skills [7, 8]. With a documented reduction in clinical training opportunities, [9, 10] and with the transition to competency-by-design (CBD) in Canadian pediatric training, many objectives of training cannot be met by opportunistic clinical experience alone [11, 12].
Performance of infrequent procedural skills are addressable via simulation practice . By utilizing adult learning theory principles such as self-direction, frequent feedback, and Ericsson’s principle of deliberate practice , simulation is the promising method for procedural learning [15, 16]. Simulation-based procedural training has been shown to be effective at achieving positive learner outcomes with regards to behavior and skill training , help reduce procedure complications,  uncover more critical errors than written examination , and has a role for life-long skill maintenance .
There is limited data on the effectiveness of simulation-based training for critical neonatal procedural skills. Addressing this gap will help inform training methodologies for neonatal trainees, with the ultimate goal of improving procedural competence and patient outcomes. In this study, we aim to assess the impact of an educational intervention targeting six rarely performed procedures in a cohort of NPM and senior general pediatrics trainees on observed skill level in simulation, self-perceived competence, comfort, and cognitive knowledge. We will describe skill acquisition and retention by measuring outcomes immediately before and after training, and at long-term follow-up.
Study design and participants
We conducted a prospective observational cohort study with a pre-post design. NPM subspecialty trainees and senior pediatric residents enrolled at Cumming School of Medicine, Calgary, Canada, were approached to participate. Background training information was collected, International Medical Graduate (IMG), or Canadian Medical Graduate (CMG), and year of training. This constitutes an opportunistic convenience sample.
Inclusion criteria were NPM trainees, or senior pediatric residents (PGY-3 or -4), and willingness to participate in 2 required sessions, 9–12 months apart. There were no specific criteria for exclusion. Written informed consent was obtained from all participants. Conjoint Health Research Ethics Board (CHREB) approval was granted (REB18-1079).
Calgary pediatric residency and NPM subspecialty training Program Directors approved this study. An email invitation to all eligible trainees was sent and all interested individuals were accommodated. Participation in this research study was voluntary, and no feedback occurred between preceptors and Program Directors as to individual performances.
Six critical procedures were identified as skills requiring further training (low self-assessed competence scores) based on a prior cross-sectional survey of 47 NPM trainees from 13 Canadian programs (unpublished data). “Advanced Neonatal Procedural Skills Day” workshops of 4 h duration each were offered during 2018–2020, as in-person simulation-based practice of the six selected skills. The workshop was developed by AS, following Kern’s approach for curriculum development . Input was sought from neonatal nursing educators, pre-existing educational materials and local practice guidelines. Further video instructions and review articles were added to Course Materials. For both the Paracentesis and Pericardiocentesis stations, specific task trainers were developed locally (Appendix A). Course Materials were sent 2 weeks in advance in a flipped classroom approach, after obtaining consent for the research study [22, 23].
- Welcome letter (Appendix B) with course overview, agenda, timeline (Appendix C), and instructions on how to do the surveys before pre-course videos and materials are viewed.
- SurveyMonkey™ links with questionnaires regarding self-perceived competence, comfort level, and cognitive knowledge regarding each procedure.
- Pre-Course video links, to be viewed after initial survey, and before attending course.
“Advanced Neonatal Procedural Skills Day”: Six stations were run, each 40 min long with 5 min intermission for 1–2 participants each (Table 1). Each station was taught by 1–2 preceptors who are experts in their fields and who had reviewed the pertinent course materials (Table 1). Consistency in preceptors ensured consistent delivery of content between workshops.
Pre-assessment: Each participant was assessed individually before teaching and together as a group to learn from each other. Each trainee was asked to briefly review indications, contraindications, and potential complications for each of the individual procedures, and perform the procedure under observation, if the participant felt competent.
Individual teaching: Depending on learners’ needs, not following a specific script, coaching included discussion and correction of any mistakes. The procedure was either shown or coached through, with deliberate practice. Repeated attempts within the time frame were encouraged until both participant and trainer were satisfied. The final procedure without further help was scored as post-assessment. A summary of learning points for most of the procedural skills was sent out to participants after the workshop for easy reference (Appendix D).
Outcomes and Assessment Tools: Primary outcome was: (a) Preceptor assessment of competence for each of the skills before (Time point 1) and after (Time point 2) simulation training “Advanced Neonatal Procedural Skills Day” via Direct Observation of Procedural Skill (DOPS, Appendix E) tool adopted from Barton .
Secondary outcomes: a) Self-perceived competence, comfort level, and cognitive knowledge for each of the advanced neonatal procedures before (Time point 1) and after (Time point 2) simulation training “Advanced Neonatal Procedural Skills Day”; and (b) Assessment of DOPS, self-perceived competence, comfort level, and cognitive knowledge for each of the skills 9–12 months after (Time point 3) simulation training which represents skill decay, and after a second intervention (Time point 4). c) Any DOPS difference between CMGs and IMGs.
The Direct Observation of Procedural Skills (DOPS) tool was first described by Wilkinson,  and adapted by Barton for the UK Foundations Doctors’ Program (Appendix E). Scored items are based on features of procedural skills such as consent, preparation, technical ability and awareness of limitations and complications. Barton showed its validity and reliability of 0.81 for trainees learning colonoscopies, and that scores were highly correlated with global assessments . The entrustability scale used for each of the items on the DOPS was adapted from Gofton  and since also used by the RCPSC adaptation of CBD [4, 6]. Preceptors were given their station’s DOPS to familiarize themselves, but no specific training or calibration was conducted. Rating occurred in real time and was not blinded.
Self-perceived competence and comfort level to perform each procedure if clinically indicated was assessed on a Likert scale from 1–5 with 1 (uncomfortable) and 5 (comfortable) . Similar tools have been used to correlate self-efficacy of NICU health care providers with performance of chest compressions and ventilation on a manikin . Self-perceived competence is relevant to correlate with objective performance data, and comfort level was used to assess whether participants would be willing to attempt the procedure faced in a real patient.
Cognitive knowledge was assessed via survey questions for each procedure, developed by AS. These were chosen based on RCPSC procedural knowledge of indications, contraindications, and possible complications. An answer key was provided a priori based on literature review . Scores were given by a single evaluator (EF) with 0–1 points for each indication, contraindication and potential complication for a max score of 3 for each procedure.
Descriptive statistical analysis was used. Mean, median and standard deviations were used for the continuous variables and evaluations. We used R  and the package “lme4”  to perform a repeated measures analysis of the relationship between the score from each assessment tool and the four study time points. We used time point as out single into the model, and as random effects we included intercepts for participants and procedural skill. Using participant and procedure as random effects accounts for two sources of non-independence. Any observable effect over time was quantified through the slope term for each skill.
There were six “Advanced Neonatal Procedural Skills Days” conducted between August 2018 and September 2020 (Aug 22, 2018; July 3, July 17, and Sept 4, 2019; July 15 and Sept 9, 2020) with 23 trainee participants (16 NPM trainees and 7 pediatric residents), 16 (70%) of which completed two workshops (11 NPM traineess and 5 residents). Reasons not to attend the second workshop for the 7 dropouts were: finished the program and moved away (× 3), maternity leave (× 2), dates could not be accommodated (× 1), and dropping out of the program (× 1). There were 15 IMGs and 8 CMGs, one CMG was an NPM trainee. Out of the 16 NPM trainees, 9 were 1st year, 5 were 2nd year, and 2 were in their 3rd year.
The DOPS for all six procedures combined increased from 3.83 (95% CL 3.72–3.94) to 4.59 (95% CL 4.37–4.82) post first intervention and to 4.74 (95% CL 4.5–4.98) after second intervention. Figure 1 shows the changes for each of the 6 procedures over the four time points: Steepest slopes with most skill increases were seen for Defibrillation and Chest Tube insertion. The only skill with no significant change was Exchange Transfusion (Fig. 1). When comparing CMGs vs IMGs for each of the skills, CMGs were consistently better in Defibrillation only, while IMGs exceeded, or were equal in all the other skills (data not shown).
Skill Decay: There was evidence of skill decay in the DOPS comparisons for all procedures: The largest decrease was shown for Exchange Transfusion, followed by Pericardiocentesis, Defibrillation and Chest Tube (Table 2). There was little drop in IO and Paracentesis. Self-perceived competence and comfort in contrast increased most for Chest Tube, followed by Defibrillation (Table 2). Worsening self-perceived competence were noted for Exchange Transfusion while comfort increased for all (Table 2).
Self-perceived competence for all six procedures increased over the four time points by 0.51 (95% CI 0.43 – 0.59) combined, with the steepest curve for Pericardiocentesis and the flattest for Exchange Transfusion (Fig. 2a).
Self-perceived comfort for all six procedures increased over the four time points by 0.44 (95%CI 0.36 – 0.53) combined, with again Pericardiocentesis being the steepest, the other skills were variable (Fig. 2b). When comparing DOPS vs self-perceived competence, all skills were judged as better by the preceptor, compared to self-assessment, with some discrepancy, most notably for Pericardiocentesis and Exchange Transfusion (Fig. 3).
Cognitive Knowledge across all six procedures increased over the four time points by 0.12 (95% CI 0.06 – 0.18) combined, with the steepest linear trend for Defibrillation and Paracentesis and the flattest for Chest tube insertion and IO access.
Our simulation training “Advanced Neonatal Procedural Skills Day” improved trainee performance in six rarely performed but potentially life-saving clinical skills (Fig. 1). Cognitive knowledge, self-perceived competence and comfort all increased, with most improvement seen in the DOPS scores which increased after first intervention. In the order of skill increase most improvement was seen for Defibrillation, Chest Tube insertion, Paracentesis, Pericardiocentesis, and IO access, while Exchange transfusion skills remained stagnant. Most at-risk for skill decay was Exchange Transfusion, followed by IO access, Pericardiocentesis, and Defibrillation. This may directly inform curricula development aiming to teach these advanced neonatal skills, especially moving towards CBD where simulation is the most viable route for such teaching.
Simulation has been used widely to improve neonatal skill acquisition in boot-camp-like workshops such as this one, [31, 32] as overall the clinical experiences cannot consistently meet training demands . The most common skill targeted by neonatal care providers in the literature has been airway management and intubation which has been shown improvements following simulation training . In a workshop similar to ours for 16 pediatric residents, chest tube insertion similarly was found to be the most improved skill, among lumbar puncture, intubation, bag-valve ventilation, pleural tap and central line insertion . They did not include Defibrillation and the follow-up was restricted to 1 week later. Our study is the first to assess systematically not only skill acquisition with a workshop, but also skill retention and decay over a longer time frame of 9–12 months. As we transition to CBD in pediatric and neonatal education, regular formative and summative assessment of learners will become commonplace and necessary. Rare procedural skills are nearly impossible to assess in the workplace without compromising patient safety. Simulation can help address learning needs on-demand and in a supervised environment; it can serve to expand gaps imposed by the infrequent occurrence of certain conditions and/or complications.
Especially Chest Tube insertion and Defibrillation, followed by the 2 ultrasound-guided skills Pericardiocentesis and Paracentesis and IO showed steep increases with instruction. Self-perceived competence and comfort increased congruently with DOPS assessments suggesting a “true” increase in competence and appropriate self-gauge. Defibrillation stood out due to the highest risk of skill decay which may be explained by the intricate nature of decision-making and perceived high stakes under pressure. CMGs seemed to perform better in this particular skill, compared to IMGs, which may be explained by the frequent practice of Pediatric Advanced Life Support in the CMG group. Particularly for an IMG trainee this skill may be worth consolidating with several instructional sessions and built-in repetition.
Pericardiocentesis stood out as the having the most discrepancy between what the trainees thought they could do versus the better preceptor assessment. It also had the most increase in self-assessed competence and comfort with instruction which may speak to the fact that it is an extremely rarely encountered skill, and intervention is simple enough if practiced before. Instruction here may be sufficient annually, or even biannually. Generally, trainees underestimated their preceptor-assessed skill which may be reflecting professional modesty, or simply discomfort.
Exchange Transfusion stood out as a skill that showed the least impact of the intervention, and also no impact on either self-perceived competence, nor comfort, during the observed time frame, but conversely showed the most skill decay. This skill also showed the most discrepancy between DOPS vs self-assessments (Fig. 3), indicating that participants performed better than they thought. The reason behind this may only be speculated on: This is a generally nursing-driven skill, and few exam questions have been dedicated to it. Motivation may have been reduced, which is a known important factor required for skill acquisition . Simulation may also not be the best way to teach this particular skill, as there are fewer manual skills involved, after the tubing has been assembled, compared to the other skills.
With the inauguration of this annual educational activity, educators and trainees acknowledged the impact, particularly for CBD. We chose to continue the annual skills day beyond this research study. With our results we can give valuable recommendation as to frequency of educational intervention and specific things to keep in mind when dealing with IMGs vs CMGs. Our results may be generalizable to similar trainees but need to be ascertained in other cohorts.
Strengths and limitations
Our cohort study assessed a group of learners over a 12-month time period with 70% completing both workshops. Assessment was consistent, with valid pre-identified tools with the same 1–2 instructors for each intervention. We could not standardize to pre-intervention skill level or exposure to these skills in clinical environment, due to small sample size. There might have been some skills encountered outside of the intervention. Also there was no prior training for preceptors before the course, nor were we able to calibrate the preceptor assessments. Cognitive skill assessment may have been limited by lack of effort by the trainees due to low stakes and no specific review.
The simulation-based intervention “Advanced Neonatal Procedural Skills Day” increased preceptor-assessed directly observed procedural skills for all skills examined, except exchange transfusion. Skill decay affected all skills after 9–12 months except Intraosseous and Paracentesis with most decrease in Exchange Transfusion. Exchange Transfusion seemed less amenable to teaching via simulation. Chest Tube insertion and Defibrillation may be best suited for deliberate reminder sessions, especially for IMG trainees. Pericardiocentesis showed the most increase in self-assessed competence and comfort levels and may be suited well by teaching once during training. The effect of a booster session was less than the first intervention, but the final scores were higher than pre-intervention. Trainees underestimated their own skills consistently compared to preceptors’ assessment.
Availability of data and materials
Raw data and excel sheets are available from the corresponding author upon request.
Competency by design
Canadian medical graduate
Direct observation of procedural skills
International medical graduate
Maintenance of certification
Neonatal intensive care unit
Neonatal perinatal medicine
Neonatal resuscitation program
Office of the health medical education scholarship
Royal college of physicians and surgeons of Canada
Amin H, Singhal N, Cole G. Validating objectives and training in Canadian paediatrics residency training programmes. Med Teach. 2011;33(3):131–44.
Bismilla ZFE, McNamara PJ, LeBlanc V, Jefferies A, Whyte H. Failure of pediatric and neonatal trainees to meet Canadian Neonatal Resuscitation Program standards for neonatal intubation. J Perinatol. 2010;30(3):182–7.
Frank J, Snell L, Cate O, Holmboe E, Carraccio C, al e. Competency-based medical education: theory to practice. Med Teach. 2010;32(8):638-645.
AlShammari A, Inayah A, Afsar N, Nurhussen A, Siddiqui A, Anwer M, et al. Evaluation of effectiveness of a paediatric simulation course in procedural skills for paediatric residents - A pilot study. J Pak Med Assoc. 2018;68(2):240–6.
Ericsson K, Harwell K. Deliberate practice and proposed limits on the effect of practice on the acquisition of expert performance: why the original definition matters and recommendations for future research. Front Psychol. 2019;10:2396.
Bosse H, Mohr J, Buss B, Krautter M, Weyrich P, Herzog W, et al. The benefit of repetitive skills training and frequency of expert feedback in the early acquisition of procedural skills. BMC Med Educ. 2015;15:22.
Taylor D, Hamdy H. Adult learning theories: implications for learning and teaching in medical education: AMEE Guide No. 83. Med Teach. 2013;35(11E1561–72):180–4.
McGaghie W, Issenberg S, Cohen E, Barsuk J, Wayne D. Does simulation-based medical education with deliberate practice yield better results than traditional clinical education? A meta-analytic comparative review of the evidence. Acad Med. 2011;86(6):706–11.
Barsuk J, McGaghie W, Cohen E, O’Leary K, Wayne D. Simulation-based mastery learning reduces complications during central venous catheter insertion in a medical intensive care unit. Crit Care Med. 2009;37(10):2697–701.
Nunnink L, Venkatesh B, Krishnan A, Vidhani K, Udy A. A prospective comparison between written examination and either simulation-based or oral viva examination of intensive care trainees’ procedural skills. Anaesth Intensive Care. 2010;38(5):876–82.
Barton J, Corbett S, van der Vleuten C. The validity and reliability of a Direct Observation of Procedural Skills assessment tool: assessing colonoscopic skills of senior endoscopists. Gastrointest Endosc. 2012;75(3):591–7.
Sawyer T, White M, Zaveri P, Chang T, Ades A, French H, et al. Learn, see, practice, prove, do, maintain: an evidence-based pedagogical framework for procedural skill training in medicine. Acad Med. 2015;90(8):1025–33.
Finan E, Bismilla Z, Campbell C, Lebalanc V, Jefferies A, Whyte H. Improved procedural performance following a simulation training session may not be transferable to the clinical environment. J Perinatol. 2012;32(7):539–44.
We would like to extend gratitude to Norma Oliver for her invaluable help and input in designing the Paracentesis and Pericardiocentesis task trainers and to Leah Donnelly, administrative support for extensive help in logistics and analysis.
Leah Donnelly, administrative support for extensive help in logistics and analysis; Norma Oliver for her invaluable support with task trainers.
Funding was obtained from the Office of the Health Medical Education Scholarship (OHMES) Calgary, Canada, and used to supplement nursing preceptors’ time teaching, materials and obtain RCPSC maintenance of certification Sect. 3 credits for participants and preceptors.
Local University of Calgary OHMES (Office of Health & Medical Education Scholarship) supported this research with $5000 in 2018.
Authors and Affiliations
Section of Neonatology, Department of Pediatrics, Cumming School of Medicine, University of Calgary, Libin Cardiovascular Institute of Alberta, Alberta Children’s Hospital Research Institute, Foothills Medical Centre, 780-1403 29Th St NW, Calgary, AB, T2N 2T9, Canada
Dr. Stritzke conceived the idea, developed the course content, precepted the courses, obtained MOC accreditation, interpreted the data, wrote the first draft, reviewed edits, and approved the final version. Dr. Murthy contributed to the protocol, precepted the course, interpreted the data, critically reviewed and edited the manuscript, and approved the final version. Dr. Fiedrich supplemented background information from previous research (needs assessment), precepted the course, reviewed and analyzed the short answers part, critically reviewed and edited the manuscript, and approved the final version. Dr. Assaad contributed to the protocol, critically reviewed, and edited the manuscript, and approved the final version. Dr. Howlett contributed to the protocol, precepted the course, critically reviewed and edited the manuscript, and approved the final version. Dr. Cheng contributed to the protocol, critically reviewed and edited the manuscript, and approved the final version. David Vickers conducted the analysis. Dr. Amin contributed to the protocol, precepted the course, interpreted the data, critically reviewed and edited the manuscript, and approved the final version. The author(s) read and approved the final manuscript.
All experimental protocols were approved by the Calgary Conjoined Health Research Ethics Board: REB18-1079 and carried out in accordance with relevant guidelines and regulations. Written informed consent was obtained from all subjects.
Consent for publication
None of the authors have any conflicts of interest or competing interests to disclose.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
Stritzke, A., Murthy, P., Fiedrich, E. et al. Advanced neonatal procedural skills: a simulation-based workshop: impact and skill decay.
BMC Med Educ23, 26 (2023). https://doi.org/10.1186/s12909-023-04000-1