Learning curves of novice residents on cataract surgery simulator: the E3CAPS pedagogic study

Abstract

Introduction

Training novice ophthalmology residents on the EyeSi^® simulator increases cataract surgery safety. However, there is no consensus regarding how much training residents should perform before their first time on patients. We evaluated the French national training program through the analysis of the learning curves of novice residents.

Methods

This prospective multicentric pedagogic study was conducted with French novice residents. Each resident completed the recommended four two-hour training sessions and performed a standardized assessment simulating standard cataract surgery before the first session (A0), at the end of the first (A1), second (A2), third (A3) and fourth (A4) sessions. For each surgical step of each attempt, the following data were collected: score, odometer, completion time, posterior capsular rupture and cumulative energy delivered (ultrasounds) during phacoemulsification. A performance threshold was set at a score of 80/100 for each surgical step, 400/500 for the overall procedure. Only descriptive statistics were employed.

Results

Sixteen newly nominated ophthalmology residents were included. Median score progressively increased from 95 [IQR 53; 147]) at A0 to 425 [IQR 411; 451] at A4. Despite a significant progression, the “emulsification” step had the lowest A4 scores 86 [IQR 60; 94] without reduction in completion time, odometer or ultrasounds delivered. The rate of posterior capsular rupture decreased linearly from 75% at A0 to 13% at A4 during “emulsification” and from 69 to 0% during “irrigation and aspiration”. At A4, only 25% [8; 53] of residents had > 80 at each step and only 75% [47; 92] had > 400/500 overall.

Conclusion

A training program consisting of four two-hour sessions on the EyeSi simulator over four consecutive days effectively enhances the surgical skills of novice ophthalmology residents. Undergoing more training sessions may improve scores and decrease the incidence of surgical complications, particularly at the emulsification step of cataract surgery. The learning curves presented here can reassure residents who are progressing normally and help identify those who need a further personalized training program. 

Trial registration

ClinicalTrials registration number: NCT05722080 (first submitted 22/12/2022, first posted 10/02/2023).

Introduction

Cataract surgery is the most frequent surgical procedure worldwide: approximately 1 million procedures are performed in France each year [1] and about 80 million people have moderate or severe distance vision impairment or blindness due to cataract [2]. This procedure under operating microscope demands a high level of expertise due to the risk of surgical complications, notably rupture of the posterior capsule, which is the main vision-threatening intraoperative complication [3]. In fact, the incidence of posterior capsular rupture is higher in patients operated on by ophthalmology residents (5 to 19%) [4,5,6,7,8,9,10] than by experienced surgeons (0.5 to 3.5%) [11]. While these complication rates seem relatively minor, they finally impact many patients. This partly explains the difficulties in accessing real-life surgical training. In two surveys involving hundreds of residents from more than 30 European countries, 42% were completely dissatisfied with the surgical skills achieved and more than 25% had not undergone any live cataract surgery training by the end of residency [12, 13]. Many residents must therefore continue their training after the end of their residency. Alternative teaching methods for cataract surgery are: theoretical learning, observation of surgeons in real life and manipulation of porcine eyes or artificial eyes. Rehearsing surgical steps on porcine eyes or artificial eyes is useful but insufficient because of significant differences in tissue thickness and texture and resistance compared to human eye. In France, cataract surgery is taught throughout the 4-year residency, in successive stages: theoretical training, practical work on pig’s eyes, artificial eyes and virtual simulator, then gradual training in real life one surgical step at a time, under the supervision of a senior surgeon. Full surgeries are then performed under supervision once each step has been mastered individually, and unsupervised surgeries are possible after the end of the 4-year residency.

Using cataract surgery simulators is a valuable approach to moderate the risks associated with training and support the “never the first time on a patient” principle set by the French National Health Agency (ANSM) [14]. The EyeSi^® surgical simulator (VRMagic, Mannheim, Germany) holds the highest usage rate in France: it enables to virtually execute the majority of the steps involved in cataract surgery, adjusting the complexity level and providing the student with both qualitative and quantitative evaluations for each surgical step. The benefits of using the EyeSi^® in resident training have been shown: the score achieved on the simulator correlates with various real-life surgical skill scores; the simulator distinguishes between skilled and novice operators, enhances surgical skill scores of both novice and experienced surgeons and decreases the occurrence of surgical complications [9, 15,16,17,18,19,20,21,22].

However, access to the simulator is hampered by many factors: expensive and limited equipment, travel time and cost, limited availability of the residents and teachers also taken up by other essential learning activities [23,24,25]. Organizing rigorous simulation training in all residency training centers before the first surgery on a patient is needed but challenging: determining the minimum number of training sessions is needed to lay the foundations of an effective and realistic resident training program. In France, the “Collège des Ophtalmologistes Universitaires de France” (COUF) recommends undergoing 4 simulation sessions, each lasting 2 h, during the initial year of residency.

The aim of the present study is to evaluate the relevance of the French training program through analysis of the learning curve of novice ophthalmology residents undergoing 4 simulation sessions. We hypothesize that it will improve the training program by determining whether additional training sessions are needed and which surgical steps are the most difficult for students to master.

Methods

Design and intervention

This prospective interventional study corresponds to Axis 1 of the E3CAPS pedagogic study (ClinicalTrials registration number: NCT05722080 (first submitted 22/12/2022, first posted 10/02/2023). The methodology of the study and the technical specifications of the EyeSi simulator have been previously described in detail [26]. Briefly, The EyeSi surgical simulator is a high-fidelity virtual reality simulator that includes a platform on which a patient’s head dummy is placed. The resident inserts two handpieces into a digital box positioned in place of the eye and uses a realistic two-axis phaco foot pedal to interact with the virtual operating field created by stereoscopy via operating microscope eyepieces placed above the platform. This installation provides a highly immersive reproduction of real-life conditions. Training modules include navigation training exercises and most steps of cataract surgery: capsulorhexis, hydromaneuver, phacoemulsification, irrigation and aspiration, intraocular lens (IOL) insertion and posterior capsular rupture management. Throughout the simulation, a digital interface provides real-time feedback and assigns positive points for completed tasks and negative points for errors, inappropriate or dangerous gestures, eye tissue injury or eye shift. Finally, the simulator provides a score for each attempt.

Each resident completed four two-hour training sessions over a period of 4 consecutive days under the supervision of an experienced cataract surgery educator. To increase the session’s relevance and manage resident fatigue, each two-hour session was divided into four 30-minute training periods, followed by four 30-minute recovery or observation periods, alternating with another resident. To measure the learning curve, each resident performed a standardized assessment at the beginning of the first session (A0), then repeated at the end of the first (A1), second (A2), third (A3) and fourth (A4) sessions. Additional assessments of the capsulorhexis step alone were performed at the beginning of the second (pre-A2), third (pre-A3) and fourth (pre-A4) sessions: this served to assess any loss since the end of the previous session. To avoid inducing fatigue or reducing the length of the training session, the capsulorhexis step was chosen because it is short, difficult to master and responsible for critical surgical complications.

The following data were collected for each resident: age, sex and dominant hand. To account for any population heterogeneity, real-life surgical practice experience was assessed using the number of attempts of any task performed in the operating room (incision, capsulorhexis, hydromaneuver, phaco-emulsification, irrigation and aspiration, IOL insertion, viscous removal, stromal hydratation for incision closure).

This clinical study was approved by the Ethics Committee of Nantes University on October 17th, 2022. The clinical study was conducted in accordance with the French Public Health Code, national and international Good Clinical Practice (GCP) guidelines, and the Declaration of Helsinki, each in the applicable version.

Participant and sample size

Given the exploratory nature of this study, the sample selected was a convenience sample, represented by the 16 newly nominated ophthalmology residents in the participating University Hospitals of Nantes, Tours, Angers and Rennes. All study participants provided written informed consent to participate. The only exclusion criterion was the absence of written consent.

Outcomes

A standardized assessment took approximately 20–30 min and consisted of the following exercises: capsulorhexis level 1, hydromaneuver level 2, phacoemulsification divide and conquer level 6, irrigation and aspiration level 3 and IOL insertion level 3. Exercises were chosen and placed in the correct order to simulate a complete standard cataract surgery. For each attempt, the following data were collected: EyeSi score (ranging from 0 to 100), distance traveled in the eye by the instruments (odometer in millimeters), completion time, posterior capsular rupture (during phacoemulsification and irrigation and aspiration) and amount of energy delivered by ultrasounds (during phacoemulsification). To ensure that the same instructions were given to all residents, standardized instructions were given orally by a Senior Surgeon Investigator (JBD) based on a written document. Assessments were performed without any student present in the room to avoid competition bias. The results of each attempt remained anonymous and confidential. They were given privately to each resident after the completion of the Axis 1 study.

Data analysis

Quantitative variables were described by their median and interquartile range (IQR). For categorical variables, the frequencies and percentages for each modality were displayed. A learning curve was derived from the plot of the score against the number of sessions. The percentage of residents reaching the performance threshold (success defined as: 80/100 for each surgical step, 400/500 for the overall procedure) at the end of each attempt and its 95% confidence interval were estimated. To assess the impact of previous real-life surgical practice experience, overall scores of residents with more than 10 real-life attempts of any surgical step were compared to the scores of residents with strictly less than 10 at A0 and A4 (Wilcoxon rank-sum test for independent data). An alpha risk of 5% was used. Statistical analyses were performed using R (version 4.0.2) [27]. Figures were drawn using the ggplot2 package [28].

Results

Study population

All sixteen newly appointed ophthalmology residents in November 2022 were included during their third month of residency. The first resident was enrolled on January 9th, 2023, and the last resident ended their training program on January 30th, 2023. Residents’ median age was 25.5 [25.0; 26.0] and the proportion of women was 44% (Table 1). Surgical practice experience was low: fifteen (94%) had never performed real-life corneal incisions, capsulorhexis, hydromaneuver and phacoemulsification. Some had made a few attempts at irrigation and aspiration (4/16, 25%) and IOL insertion (6/16, 37%).

Table 1 Baseline characteristics of ophthalmology residents included

EyeSi scores

The multidimensional curves of overall progression show a linear increase in score and decrease in completion time and odometer, associated with a decrease in variance without a plateau (Fig. 1). Throughout the training sessions, the median score increased progressively from 95 [IQR 53; 147] at A0 to 425 [411; 451] at A4 (E-Table 1). No significant difference in overall score was observed between residents who had more than 10 real-life attempts of any surgical step and residents with fewer than 10 at A0 (100 [IQR 86.5; 152] versus 69 [IQR 39; 141] respectively, p = 0.60) and A4 (445 [IQR 400; 454] versus 422 [IQR 418; 444] respectively, p = 0.63).

Fig. 1

Multidimensional learning curves showing changes in score, completion time and odometer for capsulorhexis, phacoemulsification, and irrigation and aspiration steps, along with the overall procedure. In the box plots (columns 1, 3 and 4), the edge of the box indicates the 25th and 75th percentiles, the black line within the box indicates the median. The upper whiskers extend from the edges of the box to the highest and lowest values up to 1.5 times the IQR. Data beyond the end of the whiskers are plotted individually as grey circles. The blue lines connect the medians across sessions. In column 2, the individual overlaid curves show each ophthalmology resident’s score changes across sessions (grey lines). The bold black line shows the evolution of the median score across sessions. For both graphs representing the changes in score, the dotted horizontal line indicates the corresponding performance threshold (80/100 for each step, 400/500 for the overall procedure)

The “capsulorhexis” step showed a linear increase in score with reduction in variance without reaching a plateau. Completion time and odometer decreased linearly without a plateau or reduction in variance. Additional “capsulorhexis” assessments at the beginning of each training session revealed high inter-individual and intra-individual variability during the first 3 sessions with a tendency to decrease compared with the previous session (E-Figure 1).

The “hydromaneuver” step had the least valid learning curve (E-Figure 2). The score at the first assessment before any training (A0) was unusually high and represented the only step for which the median A0 score was higher than 0 (36 [0; 75]).

The scores for the “emulsification” step were the lowest with a lower interquartile < 80 (86 [60; 94] at A4. While the increase in score for the “emulsification” step was linear, it did not reach a plateau and was not associated with a reduction in completion time and odometer of ultrasounds delivered.

The learning curves of the “irrigation and aspiration” step had a negative exponential shape and reached a plateau with the highest decrease in variance. This was the step for which the residents had the highest final scores (median 98 [96; 100]).

The “IOL insertion” step showed a linear increase in score with reduction in variance without reaching a plateau. Completion time and odometer did not decrease throughout the training program (E-Figure 2).

Success rates

The overall average success rate progressively increased throughout the training sessions from 0.0% [0.0; 24.1] at A0 to 75.0% [47.4; 91.7] at A4 (Fig. 2 and E-Table 2). Although the residents’ scores progressed for the 5 steps, progression was found to be heterogeneous: slow for the “emulsification” step for which the success rate at A4 was 56.2% [30.6; 79.2] and rapid for “irrigation and aspiration”, for which the success rate was 87.5% [60.4; 97.8] at A3 and 93.8% [67.7–99.7] at A4. Moreover, the percentage of residents who consecutively succeeded each of the 5 steps at A4 was 25.0% [8.3; 52.6].

Fig. 2

Success rate for each step throughout the training program. The success rate corresponds to the percentage of residents reaching the performance threshold of 80/100 for the corresponding surgical step. Error bars represent the 95% confidence interval

Posterior capsular rupture

The average rates of posterior capsular rupture decreased from 75% (12/16) at A0 to 12% (2/16) at A4 during the “emulsification” step and from 69% (11/16) at A0 to 0% at A4 during the “irrigation and aspiration” step (E-Table 1). The decrease was linear and continuous throughout the training program (Fig. 3).

Fig. 3

Posterior capsular rupture rate throughout the training program. The rupture rate corresponds to the percentage of residents experiencing a posterior capsular rupture during the corresponding surgical step. Error bars represent the 95% confidence interval

Discussion

In the present study, we provide valid and valuable learning curves of residents during a training program including four two-hour EyeSi simulator sessions, conducted over four consecutive days. First, “irrigation and aspiration” was the only step mastered by residents at the end of the training program. Given the risk of posterior capsular rupture during this surgical step, the learning curve for “irrigation and aspiration” is particularly reassuring for skill transfer in the operating room. Second, the “emulsification” step was the most difficult to perform for students who had a large number of insufficient scores at the end of the training program (i.e. 56.2% success rate, for more information see E-Table 3) with no reduction in completion time and odometer. Third, because the “hydromaneuver” step has the least valid learning curve, it cannot be used to assess a student’s progression. Fourth, the rate of posterior capsular rupture decreased linearly throughout the training program, but remained abnormally high at the end when compared to the incidence reported in the literature [10]. The dynamics of the curves and capsular rupture rate suggest that more than four EyeSi training sessions would be beneficial for obtaining better scores and fewer surgical complications.

Aditya et al. analyzed heart rate variations in residents during real-life cataract surgery and discovered that emulsification was the most stressful step for residents and therefore should benefit from special preparation [29], as we also observed. Belyea et al. found that resident training on the Eyesi Surgical Simulator resulted in a reduction in time to perform phacoemulsification with a reduction in the use of ultrasound power. As we did not observe reduction in emulsification completion time or in ultrasound power used, this result supports our conclusion that more than four training sessions are required to observe further progression [30]. Al-Jindan et al. assessed 200 real-life surgeries performed by 22 residents: the steps that most frequently required assistance, in descending order, were emulsification, capsulorhexis, and irrigation and aspiration. Conversely, the steps where surgical complications most commonly occurred were, in descending order, capsulorhexis, irrigation and aspiration, and emulsification [8]. Taken together, these findings support the strong need for preparation with respect to the following steps: capsulorhexis, irrigation and aspiration, and emulsification. Therefore, the current training program could be of particular benefit to the “capsulorhexis” and “irrigation and aspiration” steps for which the final scores were high, with a decrease in the occurrence of posterior capsular rupture.

A growing effort is being made to define the best way to integrate the EyeSi simulator into surgical training programs. Thomsen et al. aimed to define the proficiency level required before surgery by evaluating the ability of all EyeSi modules to discriminate between novice and expert surgeons: capsulorhexis and emulsification discriminated the best, whereas hydromaneuver did not discriminate at all, with similar mean scores between novices and experts [31]. In total, they selected the 7 most discriminating training exercises (capsulorhexis, emulsification and five navigation training exercises) to be part of an evaluation for which the pass/fail threshold was the discriminating score between novice and experts (422/700). This pass/fail score was probably underestimated because expert surgeons only had 10 min to familiarize themselves with the EyeSi simulator, so we can assume that their score would have been higher with more training. The number of training sessions required to reach this threshold was not determined, but Bergkvist et al. demonstrated that repeated training with EyeSi simulator improved scores: ten novice residents with four training sessions outperformed ten other residents with only two training sessions [32]. They were unable to determine how many sessions there should be in a training program and advised further research into learning curves, as we did in our study.

The EyeSi simulator comprises a standard training program divided into four modules, which include navigation training exercises and surgical steps (A, B, C for fundamental skills and standard cataract surgery and module D for complex cases and complication management): before moving to the next task the resident must achieve the minimum score three times in a row at each step (minimum score of 50 for category A, 70 for category B and 85 for category C). Montrisuksirikun et al. found that completing modules A, B and C prior to the first surgery decreases the total rate of complications of resident-performed phacoemulsification and shortens the learning curve for cataract surgery training. However no data on the time required to complete the simulation program was available [18]. As recommended, modules A and B must be completed before the first surgery in the United Kingdom [33, 34]. McClay and Lockington analyzed 103 resident accounts on the EyeSi simulator in Scotland: the average time to complete each module was 2h18m ± 1h08 for module A, 3h40m ± 1h37m for module B and 6h03m ± 3h48m for module C [34]. The number of training sessions required to complete modules A, B and C was not available: it was certainly spread over more than four sessions while these times are only related to “time in the eye” and do not include set-up times and time spent reading instructions. Even though these modules and minimum scores are not comparable to our standardized assessments, these data corroborate our results, which suggest that more than 4 training sessions are necessary to obtain sufficiently high scores on the simulator.

The main limitations of our study were related to external constraints and the need to produce a realistic study: the relatively small sample size was due to the need for a homogenous population of novice residents without surgical experience. Including additional undergraduate students or residents from other specialties would have induced a significant bias due to differences in age and motivation. Extending the study over several years would have delayed dissemination of the results and monopolized major research efforts on the part of students and teachers. Despite the precaution of including novices during their third month of residency, some had already performed several surgical steps in the operating room: in the hypothesis that this bias artificially improved their scores and progression, it would reinforce our findings that more than 4 sessions would be beneficial for novice residents.

The rigor of our methodology, combined with the properties of the EyeSi simulator, resulted in valid learning curves that are in line with the state of the art [35]. The validity of the y-axis (score, completion duration and odometer) is supported by the extensive literature available relating to the benefits for residents and patients of using the EyeSi simulator, the reproducibility and the correlation of score with real-life skills [9, 15,16,17,18,19,20,21]. The high consistency of the methodology supports the validity of the x-axis, which measures the learning effort of residents: each resident was given strictly equal units of time spent practicing on the same simulator with a homogeneous group of instructors and the assessments were standardized and consistent over time. The curves had the expected shape with a negative exponential relationship (progressively decreasing slope), a decrease in variance and no ceiling or floor effect which would have indicated overly easy or hard assessments [35]. We conducted the study under reproducible and immersive conditions during a single month with a homogenous population. Fatigue bias was avoided by multiple breaks during the half-day training session and a half-day’s rest before the next session. Competition bias was prevented by conducting confidential assessments, with no other resident present, and by keeping the results anonymous, even after the study was completed.

In conclusion, our study finds that a training program, consisting of four two-hour sessions on the EyeSi simulator over four consecutive days, effectively enhances the surgical skills of ophthalmology residents. However, the high incidence of complications and low performance scores, particularly at the emulsification step, indicate that the skills acquired may not yet be at an acceptable level for performing surgery in the operating room. This suggests a need for further research to optimize the training program. The learning curves that we provided can be a valuable tool in daily practice for a personalized training perspective. They can reassure ophthalmology residents who are progressing normally and help identify those who require additional, tailored training. These findings could serve as a basis for the development of a national certification or ‘license to operate’ in the field.