Relationship between demographic and social variables and performance in virtual reality among healthcare personnel: an observational study

Abstract

Background

Virtual reality is emerging as an important component of medical education. Although the benefits of virtual reality are apparent, the optimal strategy to orient to or differentiate learners in the virtual space have not been delineated. The purpose of this study was to investigate the relationships between demographic variables, social variables, and self-perceived comfort with technology to performance on a standardized non-medical virtual reality experience.

Methods

This observational study was performed at the International Meeting on Simulation in Healthcare in 2022. This conference includes medical and non-medical attendees. Participants provided demographic information and participated in a scored non-medical VR experience due to the heterogeneity of the sample. Participants then completed a System Usability Index and NASA Task Load Index form. Participants were dividedintolow scoring, medium scoring, and high scoring groups according to their final game score for further analysis.

Results

95 participants were included in final analysis. 55 (57.9%) of participants had prior virtual reality experience. Higher scores were associated with younger age (11.09, p < 0.001), identifying as male (11.09, p < 0.001), and a higher frequency of playing video games in the past (18.96, p < 0.001). The high score group was more likely to report comfort with virtual reality (6.29, p = 0.003) as well as comfort with new technology (4.61, p = 0.012). NASA Task Load Index scores trended down and System Usability Index scores trended up with increasing score. Being a nurse was a positive predictor of a higher score when compared to physicians in the multivariate analysis.

Conclusion

Performance during an immersive virtual reality experience was most closely related to age, gender, and frequency of playing video games. Self-perceived comfort with virtual reality was more predictive of score than prior virtual reality experience.

Introduction

Virtual Reality (VR) has emerged as a popular teaching modality throughout medical education, where it has served to supplement and/or replace traditional teaching modalities, including high fidelity simulation (HFS) [1,2,3]. Other benefits of VR when compared to HFS include a high level of immersion associated with head-mounted VR, lower resource utilization including facilities, equipment, and staff, lower costs, and more flexibility in location and repeatability of training modules [1, 4]. While VR may provide benefits to medical educators, the innovative technology does have potential pitfalls. The novelty of VR could lead to resistance in adopting the technology by users with no prior experience [5, 6]. This may be due to it being an unfamiliar platform or lack of confidence in navigating the new technology. Given the cost of VR, experiences outside of the education space have been limited to enthusiasts only, which has limited uptake. This has changed significantly in the last two years with the advent of all-inclusive stand-alone hardware with lower entry costs, but VR is still not considered “mainstream” [7]. Finally, VR sickness, or motion sickness associated with VR use, may limit the utilization of VR as this is a known adverse effect of immersive VR [8, 9].

As utilization increases, educators and developers must consider how best to orient learners of all types not only to the educational experience, but to VR itself. Current literature linking demographic variables, prior technological experience, and scored-based outcomes is lacking. Furthermore, there is no current literature on the connection between prior technology and VR experience or self-perceived level of technology proficiency and success in VR environments. Finally, most prior works focused on one specific medical or surgical discipline per study, limiting generalizability [3, 4, 10]. Therefore, the purpose of this study was to investigate the connections between demographic variables such as age and gender, social variables such as level of education and prior use of videogames, along with self-perceived comfort with technology to performance on a standardized non-medical VR experience.

Methods

Study design

This study was performed in line with the principles of the Declaration of Helsinki and institutional review board approval was granted through the Mount Sinai Program for the Protection of Human Subjects Institutional Review Board (STUDY-21-01752). Written informed consent was obtained from all participants. The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Participants attending the International Meeting on Simulation in Healthcare (San Diego CA USA) who attended the Technology Experience Area (TEA) were invited to participate. Meeting attendees share a common interest in simulation in healthcare and occupations represented include simulation administrators, operations specialists, engineers (hardware, software, biomedical), emergency medical technicians, nurses, nurse practitioners, physician assistants, respiratory therapists, physicians and military personnel. Attendees with medical backgrounds also come from a variety of specialties including but not limited to emergency medicine, trauma surgery, general surgery, urology, anesthesiology, internal medicine (and subspecialties) and others, making it one of the most diverse healthcare conferences. A convenience sample size was used given the research setting for this observational study. Advertisements for the TEA session were found in the conference handbook and announcements on social media were made prior to each of the sessions. The TEA was offered on 3 different conference days for two sessions each day, one from 9am to 12pm and one from 1pm to 4pm. Prior registration for this experience was not required, and all subjects were “walk-ins”. Only adult subjects (> 18 years old) were recruited to participate in this study.

After obtaining informed written consent, participants completed a demographic survey including information about their prior experiences with technology. This included information regarding age, gender, race/ethnicity, occupation, highest level of education, and prior experiences with technology and virtual reality using both multiple-choice questions and VAS 0-100 mm scales calculated to the nearest millimeter. They were then escorted to a VR station. Each station was a 10ft x 10ft square box with an Oculus Quest 2 (Meta Technologies, Menlo Park CA USA) headset, 2 hand-held VR controllers, and a haptic vest (Bhaptics, Daejeon South Korea). The hardware was preset with the Synth Riders (Kluge Interactive, Los Angeles CA USA) game. This is a rhythm-based experience whereby participants are asked to touch color-coded orbs, which correspond to musical notes, using one or both hands. With each correct note, the controllers and haptic vest provided haptic feedback through vibrations. TEA staff proctored the session only to aid participants in getting into and out of the headset as well as ensure that the area remained clear of any obstructions. TEA staff were instructed not to answer questions about the experience or provide hints to participants as to how to play the game. Participants were not offered a tutorial on the software. The same song (“Carol of the Bells Remix”) on the easy setting was used. This song was selected as it is a commonly recognizable song to the general public and was a relatively short experience (under 3 min). This non-medical game was selected due to the wide variety of occupations and knowledge base of potential participants at this conference, which allowed for a standardized assessment of success in virtual reality. At the end of the session, the final game score was recorded by the proctor and participants then completed a System Usability Index and NASA-Task Load Index form (NASA-TLX). These previously validated, publicly available tools test 10 domains on a scale of 50 and 6 domains on a scale of 600 respectively. A low final game score was determined as 0-100,000, a medium final game score as 100,001 to 300,000, and a high final game score was any score over 300,000. The thresholds for these groups were agreed upon by the investigators prior to data analysis based on pilot data.

Statistical analysis

Statistical analysis was performed on SPSS® IBM (Aramonk NY USA) Version 24. All continuous variables were tested for normality via Shapiro-Wilk test as well as visual inspection of distributions. Normal distributed variables are reported as mean (SD) and non-normal distributions as median [IQR]. Differences and impacts for demographic variables are reported as F statistics and p-values. A p-value of <0.05 was deemed significant. Categorical comparisons were performed with Chi-Square or Fischer’s Exact test as needed. Participants were placed into 3 groups based on their scores (low, medium, high). Statistics involving three groups where multiple comparisons were performed Bonferroni correction was applied. Analysis of associations between score and other variables were performed via linear regression. Due to the exponential nature of the scoring system, a natural log transformation was performed on the score data to create a normal distribution of scores to fulfil the requirements for linear regression. As such, output coefficients from the regression analysis should be interpreted as proportional changes. Finally, multivariate regression was performed with all variables included as they were deemed by the researchers as potential confounders. Groups with less than 10 subjects were combined when applicable to stabilize the model. Co-linearity was examined with an a priori threshold of variance inflation factor (VIF) of 5.0 as determining co-linearity. Co-linearity statistics can be seen in Supplemental Table 1.

Results

Participant characteristics

A total of 97 participants participated in the study, 2 of which were eliminated from the data set due to missing scores as well as large portions of the surveys missing (> 70%), leaving 95 participants for analysis. Demographic information as well as prior technology and VR experiences are displayed in Table 1, which is stratified according to low, medium, and high scores. Of note most participants reported prior experiences with VR prior to participation in the study 55 (57.9%) and reported a modest degree of comfort in VR (60 [21.75-86]) based on a 0-100 mm VAS scale, however the variation was large. The most popular prior VR headset used was the Quest platform (1 or 2) for which 41 (74.5%) of participants with prior experience had utilized. This was then followed by the HTC platform 17 (30.9%), Windows Mixed Reality headsets 7 (12.7%), the Valve Index 6 (10.9), and the HP Reverb (G1 or G2) 5 (9.1%). 6 (10.9%) participants were unsure which headset was used in their prior experience. Prior usage of video games in the past was also highly variable (47.7%-78.3%). Of the 26 participants who reported never playing videogames the most common reason was that video games did not exist when they were younger 14 (53.8%). This was followed by a general aversion to video games 7 (26.9%), no financial resources to play video games 3 (11.5%), and parental disallowance of video games 1 (3.8%).

Table 1 Demographic and prior technology and virtual reality experiences

Analysis of performance

Stratifying participants by score category revealed some significant associations. Higher scores were associated with younger age (11.09, p < 0.001), identifying as male (11.09, p < 0.001), and a higher frequency of playing video games in the past (18.96, p < 0.001). The high score group was also more likely to report comfort with VR (6.29, p = 0.003) as well as comfort with new technology (4.61, p = 0.012) and were less likely to read instruction manuals when confronted with new technology (5.46, p = 0.006). Score values as well as System Usability and NASA-TLX values are presented in Table 2. In general, usability scores were high albeit higher in the higher scoring groups. This was also mirrored in the NASA-TLX scores which trended down as user score increased. NASA-TLX scores separated by item are seen in Fig. 1.

Table 2 System usability and NASA-TLX values

Fig. 1

NASA-TLX scores for each domain stratified by low, medium, and high scores

Univariate analysis via linear regression is found in Table 3. The strongest predictors of score were age (-0.031 [-0.046 - -0.017], p < 0.001) and gender (-1.081 [-1.41 - -0.747], p < 0.001). Occupation and prior use of VR were also significant independent predictors. Prior use of video games, either daily or weekly, were associated with higher scores (Fig. 2) as was self-reported comfort in VR or with new technology as well as comfort level with using a television.

Table 3 Univariate analysis via linear regression

Fig. 2

Age of subjects stratified by frequency of video game use in the past (p < 0.001)

Multivariate analysis is found in Table 4. In the multivariate analysis age (-0.032 [-0.046–0.017], p < 0.001) and gender (-1.098 [-1.484 - -0.713], p < 0.001) continued to be the strongest predictors of score. Being a nurse was a positive predictor of a higher score when compared to physicians in the multivariate analysis, the opposite of the univariate analysis. Self-assessed comfort with VR remained a significant predictor, however, comfort with new technology became a negative predictor in the multivariate analysis. Comfort with other technologies such as smart phones, computers, and televisions were no longer predictive of score in the multivariate model.

Table 4 Multivariate analysis to identify predictors with higher scores

Discussion

The utilization of VR for educational purposes is growing [4]. There is an imminent need to differentiate learners in virtual environments in order to prevent frustration and disengagement, both from situations when content is too easy or too challenging. This is especially true for VR where the learning experience may be their first experience with VR as a modality. As a comparator, assessments of reading skills (written material) or watching skills (video material) are not required given that in these spaces, learners are healthcare workers and have decades of experience with these modalities. High fidelity simulation might be similar in this regard. Although one would imagine the best predictor of success in a virtual environment would be prior experience in VR, 21 (47.7%) of our participants in the lowest scoring group had prior experience in VR. As such, participants may find themselves being overconfident in their first VR experiences which could lead to poor performance and may not result in success. Unfortunately, we do not have detailed information on the prior VR experience of participants, as we may have found a difference with novice versus experienced VR users in this study.

Our findings support some differentiators present in the literature. Gender has been reported in several works as a differentiator, with female participants scoring lower than male counterparts [11, 12]. One of the original arguments about the difference in performance in digital games such as VR between gender dealt with the idea that digital games and VR was seen mainly as a “male past time” with 62% of gamers reported as male in 2006 [13]. Over the last 15 years, this has evened out with only 55% of gamers identifying as males in 2021 [13]. In our sample, there was no difference in gender to video game exposure as a binary variable, however, male gamers were more likely to play games daily [17 (31.4%) vs. females 1 (2.4%) p = 0.006], but not weekly [15 (27.7%) vs. 11 (26.8%) p = 0.21]. Prior experience in VR was also no different between genders (Chi-Square: p = 0.207). The impact on prior gaming experience is discussed further below.

Another hypothesis centered around differing spatial perceptions between men and women, reporting that women have a higher incidence of “VR sickness” or motion sickness in virtual environments when compared with men [11, 14]. This has been further elucidated in studies examining differences in inter pupillary distance (IPD) and incidence of VR related motion sickness, citing that headsets may not have IPD settings optimized for women [11]. Newer headsets, such as the one used in this study, have IPD adjusters to address this issue. Likewise, system usability score (p = 0.530) as well as NASA-TLX scores (p = 0.462) were no different between genders. Finally, no participants in the study reported motion sickness after the experience, but this may vary depending on the content and length of the virtual experience.

Other factors not measured in this study may contribute to the gender disparity in virtual reality success. Identifying these factors is vital to allow a more equitable level of success in virtual reality educational materials. As the utilization of or potential reliance on virtual reality for medical education increases, it becomes even more vital that medical education sets all learners up for success.

Age was also identified as being inversely related to score, which has also been demonstrated in prior works. Xu et al. examined gameplay uncertainty, display type, and age on performance in exergames and found that middle aged adults performed worse than younger participants [15]. Age related declines in working memory, grip strength and muscle mass were cited as potential drivers. Although our experience was physical and required motion, it was likely a less vigorous experience than seen in the study by Xu et al. [15]. In our study, age and NASA-TLX physical demand scores were very weakly correlated but missed the significance threshold (Pearson Correlation Coefficient: 0.180, p = 0.08). Similarly, age distributions between those with and without prior VR experience were not different (Mann-Whitney U Test, p = 0.405). Our investigation did identify a difference in age distributions stratified by frequency of game use (Fig. 2, p < 0.001). Of 26 participants who reported never playing video games in the past, 14 (53.8%) reported the primary reason being that they did not exist at that time, supporting the prior data regarding age and video game experience.

Prior use of video games have also been shown to correlate with success in VR [16]. Playing video games improves spatial performance and cognition, an important skill for success in virtual environments. Our study confirms these prior findings with those playing video games daily or weekly having improved scores (Table 3). This difference was ablated in our multivariate analysis when controlling for age and gender. Although co-linearity statistical analysis was normal (Supplementary Table 1), it may still be that age, gender, and frequency of video game use are too related to be teased out in our dataset. Our sample size was too small to perform subgroup analysis when stratified by multiple parameters. Further work in this area is needed.

Our study demonstrated self-perceived comfort with VR and new technology were also predictive of success in VR. This supports the common notion that those who are “tech savvy” are more likely to be successful with new technologies such as virtual reality. These self-perceived factors may allow instructors to stratify VR participants by comfort with VR and new technology and focus orientation and training activities on those with low comfort. This can be done through utilization of tutorials or instruction manuals for the associated hardware or software.

Our study has several limitations. Although our sample was random, the study population are all individuals attending a simulation conference, making our results less generalizable to the general public. Additionally, because of the heterogeneity of our study population, our test experience was chosen to be a non-medical experience in order to allow varying levels of medical knowledge to not impact the study findings. Because of this, the use of a non-medical experience may limit the generalizability to a medical VR experience. Our test experience, although standardized, was a single experience, and as such our results could have been different if a different experience had been chosen. Another limitation includes the low incidence of video game use, which could have impacted our results because subjects needed to adapt to VR and video game use. This could have potentially inflated the impact of prior video game use on VR performance and may not apply to medical VR simulation. Self-selection bias may have also impacted our study, meaning subjects with less experience may have been less likely to volunteer to participate, which could have impacted our final results. It was also brief, and as such may underestimate the impact of motion or VR sickness that may be accompanied by longer experiences.

In conclusion, our study demonstrated that performance during an immersive VR experience was mostly correlated with age and gender. Self-perceived comfort with VR was also predictive of performance and may serve as a better screening question to stratify participants in VR experiences instead of a binary response to prior VR experience. Future studies evaluating this topic in medical simulation and in the general population are needed to further expand our understanding of VR adaptation.