Eye-tracking during simulation-based echocardiography: a feasibility study
BMC Medical Education volume 23, Article number: 490 (2023)
Due to the technical progress point-of-care ultrasound (POCUS) is increasingly used in critical care medicine. However, optimal training strategies and support for novices have not been thoroughly researched so far. Eye-tracking, which offers insights into the gaze behavior of experts may be a useful tool for better understanding. The aim of this study was to investigate the technical feasibility and usability of eye-tracking during echocardiography as well as to analyze differences of gaze patterns between experts and non-experts.
Nine experts in echocardiography and six non-experts were equipped with eye-tracking glasses (Tobii, Stockholm, Sweden), while performing six medical cases on a simulator. For each view case specific areas of interests (AOI) were defined by the first three experts depending on the underlying pathology. Technical feasibility, participants’ subjective experience on the usability of the eye-tracking glasses as well as the differences of relative dwell time (focus) inside the areas of interest (AOI) between six experts and six non-experts were evaluated.
Technical feasibility of eye-tracking during echocardiography was achieved with an accordance of 96% between the visual area orally described by participants and the area marked by the glasses. Experts had longer relative dwell time in the case specific AOI (50.6% versus 38.4%, p = 0.072) and performed ultrasound examinations faster (138 s versus 227 s, p = 0.068). Furthermore, experts fixated earlier in the AOI (5 s versus 10 s, p = 0.033).
This feasibility study demonstrates that eye-tracking can be used to analyze experts and non-experts gaze patterns during POCUS. Although, in this study the experts had a longer fixation time in the defined AOIs compared to non-experts, further studies are needed to investigate if eye-tracking could improve teaching of POCUS.
Eye-Tracking is an objective tool, which delivers insights into gaze behavior of the operator during medical procedures [1,2,3]. With this innovative technology the visual focus of a healthcare provider during performance of medical tasks can be determined [4,5,6]. Although gaze behavior cannot be equated with cognition, it enables a better understanding of the thought processes of operators with various levels of expertise. Thus, differences in gaze patterns between experts and novices have been reported in several studies. Experts mainly remain longer in predefined areas of interest and have a clear focus to their gaze [7,8,9]. Moreover, studying an expert’s gaze behavior may even play a role in medical education for better understanding an expert’s approach and method, especially for challenging tasks (e.g., ultrasonography in critical care medicine).
Due to the technical progress of portable ultrasound devices and appropriate education of nonradiologists and noncardiologists in sonography, point-of-care ultrasound (POCUS) is increasingly used in critical care medicine . POCUS provides real-time diagnosis and supports physicians in decision making to develop treatment strategies . As clinicians with insufficient knowledge may harm patients with inaccurate diagnosis, training to acquire adequate competence is essential. However, the optimal training (e.g., course design, practical training) for becoming competent and the best way in which experts can deliver support for novices during POCUS is unknown.
A handful studies investigated the technology of eye-tracking in medical education for training, assessment as well as a feedback tool [4, 8, 12, 13]. Several studies reported that gaze trained novices demonstrated patterns closer to experts [12, 14,15,16]. Furthermore, in the educational setting eye-tracking can even be used as a feedback tool. This allows trainees to focus on the area, where training and improvement is required . Due to these findings, insights into gaze behavior between experts and novices may be a useful step for teaching POCUS. Nevertheless, eye-tracking has not been studied during echocardiography.
Therefore, the aims of this study were to investigate technical feasibility as well as usability of eye-tracking during simulation-based echocardiography and to evaluate differences of gaze patterns between experts and non-experts.
This feasibility study was approved by the data protection committee of the Medical University of Vienna, Austria (05/2021) and was performed according to the Declaration of Helsinki guidelines regarding research on human subjects. Due to the manikin-based simulation character of this study the ethics board gave this study an exempt status. This study was performed between September 2021 and March 2022 at the Medical University of Vienna.
In this study, nine clinicians with expertise on critical care echocardiography and six residents with limited expertise were recruited to voluntarily participate in the study. Informed written consent was obtained from all participants before beginning. Experts were specialists from the Department of Internal Medicine as well as from the Department of Anaesthesia, Intensive Care Medicine and Pain Medicine with diploma and regular use of focus assessed transthoracic echocardiography (FATE) for more than two years. As non-experts, six residents of the Department of Anaesthesia, Intensive Care and Pain Medicine with theoretical courses, but low practical expertise on echocardiography, were included. All residents participated in an 1 h online repetition on the basics of echocardiography prior to the study on the basics of echocardiography.
Prior to the study initiation, six pre-installed medical cases of the simulator U/S MENTOR™ (3D Systems, Airport City, Israel) were selected by the study team. For each, scenario participants were given details about history and clinical symptoms of the patient (supplementary data 1). To ensure reproducibility and to offer optimal conditions for the interpretation of ultrasound images, a standardized set-up for simulator, ultrasound monitor, chair and ambient lighting was provided and checked before each scenario. Participants were equipped with mobile Tobii Pro 2.0 eye-tracking glasses (Tobii Pro, Stockholm, Sweden). Calibration of the eye-tracking glasses was performed according to the manufacturer’s instruction before each scenario. During echocardiography, the participants described orally at which anatomical structure they were looking at. Audio and eye movements at a rate of 50 Hz were recorded with the eye-tracking glasses. For analysis, the Tobii Pro Glasses Analyzer Software (Tobii Pro, Stockholm, Sweden) was used. The areas of interests (AOIs) were determined by the study team for each of the standard FATE views (parasternal long axis, parasternal short axis, apical and subxiphoidal) for every scenario in accordance with the fixation patterns observed from the first three experts. The sequence of the scenarios was randomized using latin squares to account for habituation and learning effects.
Feasibility was defined as the accordance of the orally described area of visual attention with the area of attention marked by the eye-tracking glasses (n = 12). For differences between experts (n = 6) and non-experts (n = 6), the relative dwell time (focus on a specific AOI) as well as the absolute time to first fixation in the case specific AOI, and total duration of echocardiography were assessed. The usability of the eye-tracking glasses was evaluated by all participants (n = 15) after the last scenario using a questionnaire with 11 items about comfort and distraction (5-point Likert scale for each item) which was also used in a previous feasibility study . Baseline characteristics included age, gender, medical background, years of professional experience as well as years of experience in echocardiography.
Sample size calculation
For this feasibility study, sample size was based on previous work by Borg et al., who analyzed the feasibility of eye-tracking glasses during ultrasound-guided regional anaesthesia with 5 experts and 5 non-experts . For determination of the case specific AOIs it was suggested that in total 3 experts were sufficient.
Descriptive statistical analysis was performed using SPSS Version 27 (IBM, Armonk, NY, USA), while feasibility, usability, and the comparison between experts and non-experts was analyzed using SAS 9.4 (SAS Institute, Cary, NC, USA, 2016). Demographic data are presented as median (minimum and maximum) due to an asymmetric distribution of some variables. The accordance of the verbally stated area of visual attention with the marked area are reported as percentage of the correctly stated areas of the total number of verbalized areas of visual attention. Relative dwell time, as well as first time of fixation and duration of echocardiography, are described as median (minimum and maximum) for each group (experts; non-experts). Differences between these groups were estimated using a linear mixed model with fixed effects group, case, and period, and a random proband effect to adjust for period effects and dependence within probands. The conditional model residuals were successfully checked for approximate normal distribution and for outliers and influential observations. The conditional R-squared measure is calculated using the function r.squaredGLMM in the R package ‘MuMln’ Version 4.3.0 . Time to first fixation was log-transformed to stabilize the residual distribution (group differences are back-transformed to result in a relative group difference); an additional random effect for view within proband was added. Usability of eye-tracking glasses are reported as mean for each group.
Demographic data of the participants are depicted in Table 1.
During the entire duration of all 72 recordings, the area of attention was marked by the eye-tracking glasses. In total 1366 areas were orally explained. In 96% of these, the orally described area was in accordance with the marked area of attention.
Differences between experts and non-experts
Differences between experts and non-experts are depicted in Table 2. Although not statistically significant, experts had a longer relative dwell time in the defined AOI than non-experts (raw data across all cases: 50.6% versus 38.4%; adjusted difference: 9.4%-points, 95% confidence interval (CI): -0.9 to + 19.6, p = 0.072, R2 0.51). Furthermore, experts performed a faster ultrasound examination (median raw data across all cases:138 s versus 227 s; adjusted difference: 68 s, 95% CI: -5 to + 142, p = 0.068, R2 0.72). The median time to first fixation in the AOI (supplementary data 2) was significantly lower in the expert group compared to the non-expert group (median raw data across all cases and views: 5 s versus 10 s; adjusted relative difference: 45% reduced fixation time, 95% CI: 68% to 5%, p = 0.033, R2 0.18). Differences of gaze patterns between experts and non-experts are demonstrated in Fig. 1.
Usability of eye-tracking glasses
At the end of all 6 scenarios each participant evaluated the subjective experience with the eye-tracking glasses (Table 3). 13 of 15 participants denied that eye-tracking glasses distracted them from performing echocardiography. Most participants disagreed with discomfort due to eye-tracking glasses. Whereas all non-experts would wear eye-tracking glasses during a real scenario, only 2 of 9 experts considered this option.
To our knowledge, this is the first study evaluating eye-tracking during simulation-based echocardiography. The findings of this pilot study demonstrated that eye-tracking during transthoracic echocardiography was technically feasible, relatively comfortable for the performer, and suggested that non-experts and experts have different gaze patterns.
Eye-tracking and assessment of gaze behavior holds a great potential for educational purposes, as well as for understanding and quantifying the development of expertise in POCUS [6, 19]. In previous studies, eye-tracking was investigated while participants were looking at prepared sonography images (e.g., trauma sonography, regional anaesthesia, coronary angiogram) and demonstrated that analysis of gaze patterns is even feasible in case of moving POCUS examinations [6, 19,20,21]. Due to the development of portable ultrasound machines, POCUS is not only reserved to specialists (e.g. cardiologists) anymore. POCUS gains in importance in emergency medicine, even in out-of-hospital scenarios as it may be useful for urgent triage, guidance of procedural interventions, evaluation of clinical symptoms, therapeutic monitoring as well as during cardiopulmonary resuscitation [22, 23]. Although, training in POCUS is recommended for medical students as well as for residents and specialists, the most effective training method for POCUS is unknown. Interpretation of gaze behavior during POCUS examinations, which was demonstrated in this trial to be technically feasible, opens new horizons for educators and trainees as eye-tracking can be used for training, assessment as well as a feedback tool to improve individual’s learning curve.
In this pilot study experts and non-experts had different gaze patterns, although only the duration to first fixation in the AOI was statistically significant (5 s versus 10 s, p = 0.033). These findings are comparable with previous studies. Differences in gaze fixation behavior between experts and non-experts were detectable for image interpretation of focused assessment with sonography for trauma, ultrasound-guided regional anaesthesia as well as for coronary angiogram [6, 19,20,21]. Several studies demonstrated that experts spend less time in identifying anatomy and thus pathologic findings were focused more quickly [6, 24]. In the field of surgery, novices were trained with gaze patterns of experts leading to faster and more accurate performance . Analyzing gaze paths recordings of experts could even be useful for novices trained in POCUS to visualize regions of interests in cases with different pathologies.
This pilot study carries some limitations. First, due to the small sample size interpretation and generalization of the findings of this study should be done with caution. The 95% confidence interval for the percentage of correctly identified AOIs (92.3% to 98.9%) is fully above 90%, which was the acceptable lower bound defined in the study protocol for establishing feasibility. A formal sample size calculation has not been performed for this study, which could have led to insufficient data. However, the low width of the confidence interval demonstrates sufficient power for this feasibility study, though only in a post-hoc manner. Moreover, the non-experts in this study had a median experience of echocardiography of 5 (0.5–7) years and thus cannot be classified as complete novices.
In conclusion, the findings of this study support the feasibility of eye-tracking during POCUS examinations and suggest that interpretation of gaze patterns has the potential to become a tool for clinical education. For trainees, analysis of gaze behavior may facilitate the evaluation of learning and may provide valuable feedback about ultrasound performance. Future studies are recommended to explore this technology and its relevance for educational purposes of POCUS.
Availability of data and materials
The datasets used and analyzed during the current study are available from the corresponding author on reasonable request. Exemplary recordings are available from the corresponding author on request.
Katz TA, Weinberg DD, Fishman CE, Nadkarni V, Tremoulet P, Te Pas AB, et al. Visual attention on a respiratory function monitor during simulated neonatal resuscitation: an eye-tracking study. Arch Dis Child Fetal Neonatal Ed. 2019;104:F259–64.
Law BHY, Cheung P-Y, Wagner M, van Os S, Zheng B, Schmölzer G. Analysis of neonatal resuscitation using eye tracking: a pilot study. Arch Dis Child Fetal Neonatal Ed. 2018;103:F82–4.
Wagner M, Gröpel P, Bibl K, Olischar M, Auerbach MA, Gross IT. Eye-tracking during simulation-based neonatal airway management. Pediatr Res United States. 2020;87:518–22.
Tien T, Pucher PH, Sodergren MH, Sriskandarajah K, Yang G-Z, Darzi A. Eye tracking for skills assessment and training: a systematic review. J Surg Res United States. 2014;191:169–78.
Brunyé TT, Carney PA, Allison KH, Shapiro LG, Weaver DL, Elmore JG. Eye movements as an index of pathologist visual expertise: a pilot study. PLoS ONE. 2014;9: e103447.
Borg LK, Harrison TK, Kou A, Mariano ER, Udani AD, Kim TE, et al. Preliminary Experience Using Eye-Tracking Technology to Differentiate Novice and Expert Image Interpretation for Ultrasound-Guided Regional Anesthesia. J Ultrasound Med. 2018.
Wilson M, McGrath J, Vine S, Brewer J, Defriend D, Masters R. Psychomotor control in a virtual laparoscopic surgery training environment: gaze control parameters differentiate novices from experts. Surg Endosc. 2010;24:2458–64.
Wilson MR, Vine SJ, Bright E, Masters RSW, Defriend D, McGrath JS. Gaze training enhances laparoscopic technical skill acquisition and multi-tasking performance: a randomized, controlled study. Surg Endosc. 2011;25:3731–9.
Kelm DJ, Morrow MM, Kennedy CC, Beckman TJ. Feasibility and Utility of an Eye-Tracking Device for Assessing Teachers of Invasive Bedside Procedures. Mayo Clin proceedings Innov Qual outcomes. 2020;4:339–44.
Díaz-Gómez JL, Mayo PH, Koenig SJ. Point-of-Care Ultrasonography. N Engl J Med United States. 2021;385:1593–602.
Moore CL, Copel JA. Point-of-care ultrasonography. N Engl J Med United States. 2011;364:749–57.
Ashraf H, Sodergren MH, Merali N, Mylonas G, Singh H, Darzi A. Eye-tracking technology in medical education: A systematic review. Med Teach England. 2018;40:62–9.
Harvey A, Vickers JN, Snelgrove R, Scott MF, Morrison S. Expert surgeon’s quiet eye and slowing down: expertise differences in performance and quiet eye duration during identification and dissection of the recurrent laryngeal nerve. Am J Surg. 2014;207:187–93.
Vine SJ, Masters RSW, McGrath JS, Bright E, Wilson MR. Cheating experience: Guiding novices to adopt the gaze strategies of experts expedites the learning of technical laparoscopic skills. Surgery. 2012;152:32–40.
Soh BP, Reed WM, Poulos A, Brennan PC. E-tutorial improves students’ ability to detect lesions. Radiol Technol. 2013;85:17–26.
Sodergren MH, Orihuela-Espina F, Froghi F, Clark J, Teare J, Yang GZ, et al. Value of orientation training in laparoscopic cholecystectomy. Br J Surg. 2011;98:1437–45.
Wagner M, den Boer MC, Jansen S, Groepel P, Visser R, Witlox RSGM, et al. Video-based reflection on neonatal interventions during COVID-19 using eye-tracking glasses: an observational study. Arch Dis Child Fetal Neonatal Ed. 2022;107:156–60.
Johnson PC. Extension of Nakagawa & Schielzeth’s R(2)(GLMM) to random slopes models. Methods Ecol Evol. 2014;5:944–6.
Bell CR, Szulewski A, Walker M, McKaigney C, Ross G, Rang L, et al. Differences in Gaze Fixation Location and Duration Between Resident and Fellowship Sonographers Interpreting a Focused Assessment With Sonography in Trauma. AEM Educ Train. 2021;5:28–36.
Brunyé TT, Nallamothu BK, Elmore JG. Eye-tracking for assessing medical image interpretation: A pilot feasibility study comparing novice vs expert cardiologists. Perspect Med Educ. 2019;8:65–73.
Lee WF, Chenkin J. Exploring Eye-tracking Technology as an Assessment Tool for Point-of-care Ultrasound Training. AEM Educ Train. 2021;5: e10508.
Lee L, DeCara JM. Point-of-Care Ultrasound. Curr Cardiol Rep. 2020;22:149.
Ultrasound Guidelines: Emergency, Point-of-Care and Clinical Ultrasound Guidelines in Medicine. Ann Emerg Med. 2017;69(5):e27–e54. https://doi.org/10.1016/j.annemergmed.2016.08.457.
Harrison TK, Kim TE, Kou A, Shum C, Mariano ER, Howard SK. Feasibility of eye-tracking technology to quantify expertise in ultrasound-guided regional anesthesia. J Anesth. 2016.
We gratefully thank Lisa Berger, Lisa Kumer and Lukas Zwingl for technical support with the eye-tracking glasses.
This study was not funded.
Ethics approval and consent to participate
This feasibility study was approved by the data protection committee of the Medical University of Vienna, Austria (05/2021) and was performed according to the Declaration of Helsinki guidelines regarding research on human subjects. Due to the manikin-based simulation character of this study the ethics board gave this study an exempt status. Informed written consent was obtained from all participants before beginning.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Hafner, C., Scharner, V., Hermann, M. et al. Eye-tracking during simulation-based echocardiography: a feasibility study. BMC Med Educ 23, 490 (2023). https://doi.org/10.1186/s12909-023-04458-z