Skip to main content
Download PDF

Eye-tracking during simulation-based echocardiography: a feasibility study

BMC Medical Education volume 23, Article number: 490 (2023) Cite this article

Abstract

Introduction

Due to the technical progress point-of-care ultrasound (POCUS) is increasingly used in critical care medicine. However, optimal training strategies and support for novices have not been thoroughly researched so far. Eye-tracking, which offers insights into the gaze behavior of experts may be a useful tool for better understanding. The aim of this study was to investigate the technical feasibility and usability of eye-tracking during echocardiography as well as to analyze differences of gaze patterns between experts and non-experts.

Methods

Nine experts in echocardiography and six non-experts were equipped with eye-tracking glasses (Tobii, Stockholm, Sweden), while performing six medical cases on a simulator. For each view case specific areas of interests (AOI) were defined by the first three experts depending on the underlying pathology. Technical feasibility, participants’ subjective experience on the usability of the eye-tracking glasses as well as the differences of relative dwell time (focus) inside the areas of interest (AOI) between six experts and six non-experts were evaluated.

Results

Technical feasibility of eye-tracking during echocardiography was achieved with an accordance of 96% between the visual area orally described by participants and the area marked by the glasses. Experts had longer relative dwell time in the case specific AOI (50.6% versus 38.4%, p = 0.072) and performed ultrasound examinations faster (138 s versus 227 s, p = 0.068). Furthermore, experts fixated earlier in the AOI (5 s versus 10 s, p = 0.033).

Conclusion

This feasibility study demonstrates that eye-tracking can be used to analyze experts and non-experts gaze patterns during POCUS. Although, in this study the experts had a longer fixation time in the defined AOIs compared to non-experts, further studies are needed to investigate if eye-tracking could improve teaching of POCUS.

Peer Review reports

Introduction

Eye-Tracking is an objective tool, which delivers insights into gaze behavior of the operator during medical procedures [1,2,3]. With this innovative technology the visual focus of a healthcare provider during performance of medical tasks can be determined [4,5,6]. Although gaze behavior cannot be equated with cognition, it enables a better understanding of the thought processes of operators with various levels of expertise. Thus, differences in gaze patterns between experts and novices have been reported in several studies. Experts mainly remain longer in predefined areas of interest and have a clear focus to their gaze [7,8,9]. Moreover, studying an expert’s gaze behavior may even play a role in medical education for better understanding an expert’s approach and method, especially for challenging tasks (e.g., ultrasonography in critical care medicine).

Due to the technical progress of portable ultrasound devices and appropriate education of nonradiologists and noncardiologists in sonography, point-of-care ultrasound (POCUS) is increasingly used in critical care medicine [10]. POCUS provides real-time diagnosis and supports physicians in decision making to develop treatment strategies [11]. As clinicians with insufficient knowledge may harm patients with inaccurate diagnosis, training to acquire adequate competence is essential. However, the optimal training (e.g., course design, practical training) for becoming competent and the best way in which experts can deliver support for novices during POCUS is unknown.

A handful studies investigated the technology of eye-tracking in medical education for training, assessment as well as a feedback tool [4, 8, 12, 13]. Several studies reported that gaze trained novices demonstrated patterns closer to experts [12, 14,15,16]. Furthermore, in the educational setting eye-tracking can even be used as a feedback tool. This allows trainees to focus on the area, where training and improvement is required [12]. Due to these findings, insights into gaze behavior between experts and novices may be a useful step for teaching POCUS. Nevertheless, eye-tracking has not been studied during echocardiography.

Therefore, the aims of this study were to investigate technical feasibility as well as usability of eye-tracking during simulation-based echocardiography and to evaluate differences of gaze patterns between experts and non-experts.

Methods

Study design

This feasibility study was approved by the data protection committee of the Medical University of Vienna, Austria (05/2021) and was performed according to the Declaration of Helsinki guidelines regarding research on human subjects. Due to the manikin-based simulation character of this study the ethics board gave this study an exempt status. This study was performed between September 2021 and March 2022 at the Medical University of Vienna.

Participants

In this study, nine clinicians with expertise on critical care echocardiography and six residents with limited expertise were recruited to voluntarily participate in the study. Informed written consent was obtained from all participants before beginning. Experts were specialists from the Department of Internal Medicine as well as from the Department of Anaesthesia, Intensive Care Medicine and Pain Medicine with diploma and regular use of focus assessed transthoracic echocardiography (FATE) for more than two years. As non-experts, six residents of the Department of Anaesthesia, Intensive Care and Pain Medicine with theoretical courses, but low practical expertise on echocardiography, were included. All residents participated in an 1 h online repetition on the basics of echocardiography prior to the study on the basics of echocardiography.

Experimental setup

Prior to the study initiation, six pre-installed medical cases of the simulator U/S MENTOR™ (3D Systems, Airport City, Israel) were selected by the study team. For each, scenario participants were given details about history and clinical symptoms of the patient (supplementary data 1). To ensure reproducibility and to offer optimal conditions for the interpretation of ultrasound images, a standardized set-up for simulator, ultrasound monitor, chair and ambient lighting was provided and checked before each scenario. Participants were equipped with mobile Tobii Pro 2.0 eye-tracking glasses (Tobii Pro, Stockholm, Sweden). Calibration of the eye-tracking glasses was performed according to the manufacturer’s instruction before each scenario. During echocardiography, the participants described orally at which anatomical structure they were looking at. Audio and eye movements at a rate of 50 Hz were recorded with the eye-tracking glasses. For analysis, the Tobii Pro Glasses Analyzer Software (Tobii Pro, Stockholm, Sweden) was used. The areas of interests (AOIs) were determined by the study team for each of the standard FATE views (parasternal long axis, parasternal short axis, apical and subxiphoidal) for every scenario in accordance with the fixation patterns observed from the first three experts. The sequence of the scenarios was randomized using latin squares to account for habituation and learning effects.

Outcome measures

Feasibility was defined as the accordance of the orally described area of visual attention with the area of attention marked by the eye-tracking glasses (n = 12). For differences between experts (n = 6) and non-experts (n = 6), the relative dwell time (focus on a specific AOI) as well as the absolute time to first fixation in the case specific AOI, and total duration of echocardiography were assessed. The usability of the eye-tracking glasses was evaluated by all participants (n = 15) after the last scenario using a questionnaire with 11 items about comfort and distraction (5-point Likert scale for each item) which was also used in a previous feasibility study [17]. Baseline characteristics included age, gender, medical background, years of professional experience as well as years of experience in echocardiography.

Sample size calculation

For this feasibility study, sample size was based on previous work by Borg et al., who analyzed the feasibility of eye-tracking glasses during ultrasound-guided regional anaesthesia with 5 experts and 5 non-experts [6]. For determination of the case specific AOIs it was suggested that in total 3 experts were sufficient.

Statistical analysis

Descriptive statistical analysis was performed using SPSS Version 27 (IBM, Armonk, NY, USA), while feasibility, usability, and the comparison between experts and non-experts was analyzed using SAS 9.4 (SAS Institute, Cary, NC, USA, 2016). Demographic data are presented as median (minimum and maximum) due to an asymmetric distribution of some variables. The accordance of the verbally stated area of visual attention with the marked area are reported as percentage of the correctly stated areas of the total number of verbalized areas of visual attention. Relative dwell time, as well as first time of fixation and duration of echocardiography, are described as median (minimum and maximum) for each group (experts; non-experts). Differences between these groups were estimated using a linear mixed model with fixed effects group, case, and period, and a random proband effect to adjust for period effects and dependence within probands. The conditional model residuals were successfully checked for approximate normal distribution and for outliers and influential observations. The conditional R-squared measure is calculated using the function r.squaredGLMM in the R package ‘MuMln’ Version 4.3.0 [18]. Time to first fixation was log-transformed to stabilize the residual distribution (group differences are back-transformed to result in a relative group difference); an additional random effect for view within proband was added. Usability of eye-tracking glasses are reported as mean for each group.

Results

Participants’ characteristics

Demographic data of the participants are depicted in Table 1.

Table 1 Demographic data
Full size table

Technical feasibility

During the entire duration of all 72 recordings, the area of attention was marked by the eye-tracking glasses. In total 1366 areas were orally explained. In 96% of these, the orally described area was in accordance with the marked area of attention.

Differences between experts and non-experts

Differences between experts and non-experts are depicted in Table 2. Although not statistically significant, experts had a longer relative dwell time in the defined AOI than non-experts (raw data across all cases: 50.6% versus 38.4%; adjusted difference: 9.4%-points, 95% confidence interval (CI): -0.9 to + 19.6, p = 0.072, R2 0.51). Furthermore, experts performed a faster ultrasound examination (median raw data across all cases:138 s versus 227 s; adjusted difference: 68 s, 95% CI: -5 to + 142, p = 0.068, R2 0.72). The median time to first fixation in the AOI (supplementary data 2) was significantly lower in the expert group compared to the non-expert group (median raw data across all cases and views: 5 s versus 10 s; adjusted relative difference: 45% reduced fixation time, 95% CI: 68% to 5%, p = 0.033, R2 0.18). Differences of gaze patterns between experts and non-experts are demonstrated in Fig. 1.

Table 2 Differences between experts (n = 6) and non-experts (n = 6)
Full size table
Fig. 1
figure 1

Heatmap of gaze patterns (case 2 pulmonary embolism) between experts (n = 6) and non-experts (n = 6). Intensity of redness correlates with the dwell time in the area

Full size image

Usability of eye-tracking glasses

At the end of all 6 scenarios each participant evaluated the subjective experience with the eye-tracking glasses (Table 3). 13 of 15 participants denied that eye-tracking glasses distracted them from performing echocardiography. Most participants disagreed with discomfort due to eye-tracking glasses. Whereas all non-experts would wear eye-tracking glasses during a real scenario, only 2 of 9 experts considered this option.

Table 3 Usability of eye-tracking glasses
Full size table

Discussion

To our knowledge, this is the first study evaluating eye-tracking during simulation-based echocardiography. The findings of this pilot study demonstrated that eye-tracking during transthoracic echocardiography was technically feasible, relatively comfortable for the performer, and suggested that non-experts and experts have different gaze patterns.

Eye-tracking and assessment of gaze behavior holds a great potential for educational purposes, as well as for understanding and quantifying the development of expertise in POCUS [6, 19]. In previous studies, eye-tracking was investigated while participants were looking at prepared sonography images (e.g., trauma sonography, regional anaesthesia, coronary angiogram) and demonstrated that analysis of gaze patterns is even feasible in case of moving POCUS examinations [6, 19,20,21]. Due to the development of portable ultrasound machines, POCUS is not only reserved to specialists (e.g. cardiologists) anymore. POCUS gains in importance in emergency medicine, even in out-of-hospital scenarios as it may be useful for urgent triage, guidance of procedural interventions, evaluation of clinical symptoms, therapeutic monitoring as well as during cardiopulmonary resuscitation [22, 23]. Although, training in POCUS is recommended for medical students as well as for residents and specialists, the most effective training method for POCUS is unknown. Interpretation of gaze behavior during POCUS examinations, which was demonstrated in this trial to be technically feasible, opens new horizons for educators and trainees as eye-tracking can be used for training, assessment as well as a feedback tool to improve individual’s learning curve.

In this pilot study experts and non-experts had different gaze patterns, although only the duration to first fixation in the AOI was statistically significant (5 s versus 10 s, p = 0.033). These findings are comparable with previous studies. Differences in gaze fixation behavior between experts and non-experts were detectable for image interpretation of focused assessment with sonography for trauma, ultrasound-guided regional anaesthesia as well as for coronary angiogram [6, 19,20,21]. Several studies demonstrated that experts spend less time in identifying anatomy and thus pathologic findings were focused more quickly [6, 24]. In the field of surgery, novices were trained with gaze patterns of experts leading to faster and more accurate performance [8]. Analyzing gaze paths recordings of experts could even be useful for novices trained in POCUS to visualize regions of interests in cases with different pathologies.

This pilot study carries some limitations. First, due to the small sample size interpretation and generalization of the findings of this study should be done with caution. The 95% confidence interval for the percentage of correctly identified AOIs (92.3% to 98.9%) is fully above 90%, which was the acceptable lower bound defined in the study protocol for establishing feasibility. A formal sample size calculation has not been performed for this study, which could have led to insufficient data. However, the low width of the confidence interval demonstrates sufficient power for this feasibility study, though only in a post-hoc manner. Moreover, the non-experts in this study had a median experience of echocardiography of 5 (0.5–7) years and thus cannot be classified as complete novices.

Conclusion

In conclusion, the findings of this study support the feasibility of eye-tracking during POCUS examinations and suggest that interpretation of gaze patterns has the potential to become a tool for clinical education. For trainees, analysis of gaze behavior may facilitate the evaluation of learning and may provide valuable feedback about ultrasound performance. Future studies are recommended to explore this technology and its relevance for educational purposes of POCUS.

Availability of data and materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request. Exemplary recordings are available from the corresponding author on request.

References

  1. Katz TA, Weinberg DD, Fishman CE, Nadkarni V, Tremoulet P, Te Pas AB, et al. Visual attention on a respiratory function monitor during simulated neonatal resuscitation: an eye-tracking study. Arch Dis Child Fetal Neonatal Ed. 2019;104:F259–64.

    Article  Google Scholar 

  2. Law BHY, Cheung P-Y, Wagner M, van Os S, Zheng B, Schmölzer G. Analysis of neonatal resuscitation using eye tracking: a pilot study. Arch Dis Child Fetal Neonatal Ed. 2018;103:F82–4.

  3. Wagner M, Gröpel P, Bibl K, Olischar M, Auerbach MA, Gross IT. Eye-tracking during simulation-based neonatal airway management. Pediatr Res United States. 2020;87:518–22.

    Article  Google Scholar 

  4. Tien T, Pucher PH, Sodergren MH, Sriskandarajah K, Yang G-Z, Darzi A. Eye tracking for skills assessment and training: a systematic review. J Surg Res United States. 2014;191:169–78.

    Article  Google Scholar 

  5. Brunyé TT, Carney PA, Allison KH, Shapiro LG, Weaver DL, Elmore JG. Eye movements as an index of pathologist visual expertise: a pilot study. PLoS ONE. 2014;9: e103447.

    Article  Google Scholar 

  6. Borg LK, Harrison TK, Kou A, Mariano ER, Udani AD, Kim TE, et al. Preliminary Experience Using Eye-Tracking Technology to Differentiate Novice and Expert Image Interpretation for Ultrasound-Guided Regional Anesthesia. J Ultrasound Med. 2018.

  7. Wilson M, McGrath J, Vine S, Brewer J, Defriend D, Masters R. Psychomotor control in a virtual laparoscopic surgery training environment: gaze control parameters differentiate novices from experts. Surg Endosc. 2010;24:2458–64.

    Article  Google Scholar 

  8. Wilson MR, Vine SJ, Bright E, Masters RSW, Defriend D, McGrath JS. Gaze training enhances laparoscopic technical skill acquisition and multi-tasking performance: a randomized, controlled study. Surg Endosc. 2011;25:3731–9.

    Article  Google Scholar 

  9. Kelm DJ, Morrow MM, Kennedy CC, Beckman TJ. Feasibility and Utility of an Eye-Tracking Device for Assessing Teachers of Invasive Bedside Procedures. Mayo Clin proceedings Innov Qual outcomes. 2020;4:339–44.

    Article  Google Scholar 

  10. Díaz-Gómez JL, Mayo PH, Koenig SJ. Point-of-Care Ultrasonography. N Engl J Med United States. 2021;385:1593–602.

    Article  Google Scholar 

  11. Moore CL, Copel JA. Point-of-care ultrasonography. N Engl J Med United States. 2011;364:749–57.

    Article  Google Scholar 

  12. Ashraf H, Sodergren MH, Merali N, Mylonas G, Singh H, Darzi A. Eye-tracking technology in medical education: A systematic review. Med Teach England. 2018;40:62–9.

    Article  Google Scholar 

  13. Harvey A, Vickers JN, Snelgrove R, Scott MF, Morrison S. Expert surgeon’s quiet eye and slowing down: expertise differences in performance and quiet eye duration during identification and dissection of the recurrent laryngeal nerve. Am J Surg. 2014;207:187–93.

    Article  Google Scholar 

  14. Vine SJ, Masters RSW, McGrath JS, Bright E, Wilson MR. Cheating experience: Guiding novices to adopt the gaze strategies of experts expedites the learning of technical laparoscopic skills. Surgery. 2012;152:32–40.

    Article  Google Scholar 

  15. Soh BP, Reed WM, Poulos A, Brennan PC. E-tutorial improves students’ ability to detect lesions. Radiol Technol. 2013;85:17–26.

    Google Scholar 

  16. Sodergren MH, Orihuela-Espina F, Froghi F, Clark J, Teare J, Yang GZ, et al. Value of orientation training in laparoscopic cholecystectomy. Br J Surg. 2011;98:1437–45.

    Article  Google Scholar 

  17. Wagner M, den Boer MC, Jansen S, Groepel P, Visser R, Witlox RSGM, et al. Video-based reflection on neonatal interventions during COVID-19 using eye-tracking glasses: an observational study. Arch Dis Child Fetal Neonatal Ed. 2022;107:156–60.

    Article  Google Scholar 

  18. Johnson PC. Extension of Nakagawa & Schielzeth’s R(2)(GLMM) to random slopes models. Methods Ecol Evol. 2014;5:944–6.

    Article  Google Scholar 

  19. Bell CR, Szulewski A, Walker M, McKaigney C, Ross G, Rang L, et al. Differences in Gaze Fixation Location and Duration Between Resident and Fellowship Sonographers Interpreting a Focused Assessment With Sonography in Trauma. AEM Educ Train. 2021;5:28–36.

    Article  Google Scholar 

  20. Brunyé TT, Nallamothu BK, Elmore JG. Eye-tracking for assessing medical image interpretation: A pilot feasibility study comparing novice vs expert cardiologists. Perspect Med Educ. 2019;8:65–73.

    Article  Google Scholar 

  21. Lee WF, Chenkin J. Exploring Eye-tracking Technology as an Assessment Tool for Point-of-care Ultrasound Training. AEM Educ Train. 2021;5: e10508.

    Article  Google Scholar 

  22. Lee L, DeCara JM. Point-of-Care Ultrasound. Curr Cardiol Rep. 2020;22:149.

    Article  Google Scholar 

  23. Ultrasound Guidelines: Emergency, Point-of-Care and Clinical Ultrasound Guidelines in Medicine. Ann Emerg Med. 2017;69(5):e27–e54. https://doi.org/10.1016/j.annemergmed.2016.08.457.

  24. Harrison TK, Kim TE, Kou A, Shum C, Mariano ER, Howard SK. Feasibility of eye-tracking technology to quantify expertise in ultrasound-guided regional anesthesia. J Anesth. 2016.

Download references

Acknowledgements

We gratefully thank Lisa Berger, Lisa Kumer and Lukas Zwingl for technical support with the eye-tracking glasses.

Funding

This study was not funded.

Author information

Authors and Affiliations

  1. Department of Anaesthesia, General Intensive Care and Pain Medicine, Medical University of Vienna, Spitalgasse 23, 1090, Vienna, Austria

    Christina Hafner, Vincenz Scharner, Martina Hermann, Philipp Metelka, Benedikt Hurch, Daniel Alexander Klaus, Wolfgang Schaubmayr, Harald Willschke & Thomas Hamp

  2. Ludwig Boltzmann Institute Digital Health and Patient Safety, Vienna, Austria

    Christina Hafner, Vincenz Scharner, Martina Hermann & Harald Willschke

  3. Department of Pediatrics, Comprehensive Center for Pediatrics, Medical University of Vienna, Vienna, Austria

    Michael Wagner

  4. Center for Medical Statistics, Informatics, and Intelligent Systems, Medical University of Vienna, Vienna, Austria

    Andreas Gleiss

  5. Emergency Medical Service Vienna, Radetzkystraße 1, 1030, Vienna, Austria

    Thomas Hamp

Authors
  1. Christina Hafner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Vincenz Scharner
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Martina Hermann
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Philipp Metelka
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Benedikt Hurch
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Daniel Alexander Klaus
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Wolfgang Schaubmayr
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Michael Wagner
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Andreas Gleiss
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. Harald Willschke
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Thomas Hamp
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Christina Hafner, Vincenz Scharner, Martina Hermann, Michael Wagner, Andreas Gleiss, Harald Willschke and Thomas Hamp designed the study. Christina Hafner, Vincenz Scharner, Philipp Metelka and Benedikt Hurch participated in data collection. Andreas Gleiss did statistical analysis. Christina Hafner, Daniel Klaus, Wolfgang Schaubmayr, Michael Wagner, Harald Willschke and Thomas Hamp analyzed and interpreted the data. Benedikt Hurch worked as a diploma student in the context of his thesis on this study. All authors critically revised the manuscript for important intellectual content and approved its current version. All authors agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The manuscript has not been previously published and is not under consideration for publication in the same or substantially similar form in any other peer-reviewed media.

Authors’ information

Benedikt Hurch worked as a diploma student in the context of his thesis on this study. Preliminary work of this study was presented as a poster at the AIC 2022, Bregenz, Austria.

Corresponding author

Correspondence to Thomas Hamp.

Ethics declarations

Ethics approval and consent to participate

This feasibility study was approved by the data protection committee of the Medical University of Vienna, Austria (05/2021) and was performed according to the Declaration of Helsinki guidelines regarding research on human subjects. Due to the manikin-based simulation character of this study the ethics board gave this study an exempt status. Informed written consent was obtained from all participants before beginning.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:

Supplementary Data 1. History and clinical symptoms of the scenario. Supplementary Data 2. First fixation time in areas of interest.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Hafner, C., Scharner, V., Hermann, M. et al. Eye-tracking during simulation-based echocardiography: a feasibility study. BMC Med Educ 23, 490 (2023). https://doi.org/10.1186/s12909-023-04458-z

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12909-023-04458-z

Keywords

BMC Medical Education

ISSN: 1472-6920

Contact us