The effect of assessor visibility on student stress and anxiety in emergency care simulation assessments

Abstract

Background

Simulation assessment can result in anxiety for students. These assessments involve being observed by assessors, so there is a significant socio-evaluative stress component which may cause greater anxiety if assessors are visible to students during assessments. The aim of this study was to determine the effect of assessor visibility on biological and physiological markers of stress and levels of anxiety experienced by pre-hospital emergency care students in a simulation assessment environment.

Methods

A paired comparison, pre-post test research design was used with two conditions; an assessor visible condition with simulation assessors visible to students in the room and an assessor not visible condition with assessors not in the room but connected via video link. With a sample of 29 emergency care students anxiety was measured with the State-Trait Anxiety Inventory and stress was measured with salivary cortisol and heart rate variability (HRV).

Results

Differences in state anxiety scores, HRV variables and salivary cortisol suggested lower stress and anxiety in the assessor visible group. Only heart rate was significantly different between the groups (p = 0.016), with a higher heart rate in the assessor not visible group.

Conclusions

Greater stress and anxiety may be experienced by students during emergency care simulation assessments when assessors are not visible to students.

Background

Simulation has been broadly implemented in emergency care education, both for teaching of clinical patient care and for assessment of clinical competence [1,2,3,4,5,6]. For the purposes of assessment, simulation offers a number of advantages including ethical appropriateness [7]. Using simulation as an assessment approach may also enhance the reliability of assessments because a single clinical case can be given to a group of students. In South Africa, simulation is widely used for assessment across a range of pre-hospital emergency care qualifications.

Stress and anxiety are part of the experience of pre-hospital emergency care, due to the nature of emergencies. However, beyond this inherent characteristic of emergency care, assessments are associated with anxiety in their own right [8]. Like all assessments, simulation assessments have the potential to provoke anxiety amongst students. However, the simulation assessment environment is associated with unique characteristics that may enhance anxiety compared to other forms of assessment. These include making patient care decisions and performing procedures in real time, especially when the simulated cases involve emergencies. Feedback in the form of clinical signs or prompts can further complicate management of the case, sometimes suggesting to students that they may have made an incorrect decision or performed an inappropriate intervention – adding further pressure.

In addition to the factors mentioned above, simulation assessments always involve observation and judgement of performance and thus at least some degree of socio-evaluative stress – stress associated with a student’s self-identity being judged negatively [9]. Distress, and the affective state of anxiety associated with it, is thought to arise from a situation where an individual evaluates the demands required of a specific situation and their own personal resources and determines that demands outstrip current available resources [10]. Socio-evaluative stress is thought to be a likely stimulus for distress amongst those whose performance is judged [9, 11]. Socio-evaluative stress also activates the physiological stress response systems as measured through cardiovascular and cortisol changes [11]. Thus, in the context of simulation assessments, socio-evaluative stress is a strong potential stimulus for student feelings of anxiety. In the present context, anxiety is defined as the feelings of insecurity which can be induced even without the presence of a stressor (for example, in anticipation of it) whereas stress is defined as an external manipulation or application of a stressor.

In simulation assessment practice, two forms of student observation are possible. Students may be directly observed by assessors situated in the same assessment room. In this case, student awareness of assessor observation is heightened as students can see the assessors and may occasionally make eye contact with them during an assessment. In an alternative approach, assessors are situated in a separate room (or behind a one-way viewing window in the same room) and observe students via a live video and audio stream (or through a one-way viewing window). In this case, student awareness of assessor observation may be diminished because assessors are not directly visible. The former approach (‘assessor visible’) is a low technology and thus cost-effective way of approaching simulation assessment while the latter approach (‘assessor not visible’) requires a more sophisticated environment and technology, with associated extra cost.

A previous study by Horsley and Wambach [12] investigated anxiety experienced by nursing students during simulation assessments when assessors were both visible and not visible to students. Although no significant differences in anxiety scores was found between the two groups in this study, the authors still recommended using an assessor not visible configuration seemingly based on pre-post differences in anxiety observed in the group to whom assessors were visible. Likewise, a study be Mills et al. [13]. found no significant difference in physiological stress response between instructor present and absent conditions, but still recommended removing instructors on the basis of subjective feedback from students.

With very limited and equivocal data available, the aim of this study was to determine the effect of assessor visibility on biological and physiological markers of stress and the perception of anxiety experienced by emergency care students in a simulation assessment environment. Considering the prominent theoretical role played by socio-evaluative stress and anxiety, and the principle that this is likely heightened when assessors are visible to students, we hypothesized that there would be a difference between assessor visible and not visible conditions and that the assessor visible condition would be associated with significantly increased physiological and biological stress markers and state anxiety scores.

Methods

This study followed a paired comparison, pre-post test design with two conditions; an ‘assessor visible’ condition with simulation assessors in the assessment room and visible to students and an ‘assessor not visible’ condition with assessors situated in another room and not visible to students.

Population and Sample

The population of interest was all students registered for the Bachelor of Health Sciences in Emergency Medical Care programme in the Department of Emergency Medical Care at the University of Johannesburg, in 2019. This is a four-year professional degree programme leading to registration with the Health Professions Council of South Africa as an Emergency Care Practitioner, the top tier of pre-hospital emergency care provider in South Africa. The population size was 101 students.

A non-random sample of students in the second through fourth academic year of study were recruited. A sample size calculation based on variance estimates using existing and published data (heart rate variability, cortisol and State-Trait Anxiety Inventory scores) and a medium effect size of 0.51 yielded a required sample of 31 participants with α = 0.05 and statistical power of 0.8 for a paired samples t-test. Students in all academic years were informed of the research verbally and those who gave informed consent comprised the sample.

Exclusion criteria for sample selection were medication that may have affected cardiovascular and autonomic nervous system function, including alpha- or beta-adrenergic blockers and stimulants such as methylphenidate, known endocrine disorders involving the adrenal cortex and cortisol secretion, known disorders of the salivary glands, corticosteroid medication and any diagnosed anxiety disorders.

Setting

All simulation assessments were conducted in the simulation laboratory situated in the Faculty of Health Sciences at the University of [redacted].

Simulation assessments

At the time the research was conducted, the default configuration for all simulation assessments in the Department of Emergency Medical Care was the assessor visible configuration. As the effect of a different configuration (the assessor not visible configuration) on student anxiety was unknown, and because of the potential for anxiety to affect student performance it was decided to expose the students to mock simulation assessments rather than real assessments (i.e., real assessments meaning assessments that resulted in marks and thus academic consequences). In all other respects the mock simulation assessments were conducted under conditions identical to those of real assessments. Each simulation assessment varied in duration from approximately 15 min to approximately 25 min, with longer duration assessments for third and fourth year students.

Simulation Assessment Case Selection

As simulation assessments are normally designed for learning outcomes in a specific academic year of study, three different simulation assessment cases were selected for each year in each group. Cases used in the previous three years, considered to be representative of assessments in each year, were selected by lecturers who normally design and administer these assessments. Because all cases had been previously used in real simulation assessments, all of them had been reviewed by an external moderator prior to final approval.

In order to mitigate the potential bias of simulation case complexity on stress and anxiety experienced by students in assessor visible and assessor not visible conditions, a case complexity rubric was designed. The rubric took into consideration patient presentation, procedures or skills involved and clinical decision-making with each of these scored in one of three levels of complexity and summed for an overall complexity score ranging between 3 and 9 points. Two independent external raters, selected based on a minimum of three to five years of simulation education and emergency care experience, were asked to independently assign a complexity score to each of the two simulations from each academic year of study using the rubric. Intraclass correlation of ratings was calculated and an intraclass correlation coefficient (ICC) of at least 0.6 was considered adequate agreement between raters. A difference of 1–2 points between complexity scores of simulations in the assessor visible and assessor not visible conditions was considered adequate similarity of complexity for the simulations to be included. A short summary of the simulation cases that were selected is given in Table 1 (for these cases, ICC = 0.828, p = 0.012).

Table 1 Simulation assessment case summaries

Scoring

Scoring of simulation assessments used a tool that has been in place in the Department for several years and used in a previous study [14]. The tool comprised of a set of assessment outcomes chosen by the lecturer who compiled the assessment in each year. These outcomes were weighted by three independent experts and the average of the three was used as the final weighting for each assessment outcome. An assessment rubric was constructed with a set of scores and descriptors for each assessment outcome. Score categories were best practice, competent, omitted, not yet competent, minor harm and major harm. Between two to three independent external assessors used the score sheet and the assessment rubric to assign scores for each assessment outcome but were blinded to the outcome weightings. A final mark for the simulation assessment was calculated by summing the weighted scores [15].

Equipment and setup

For all simulation assessments, a high-fidelity simulator mannequin (SimMan 3G or SimBaby, Laerdal Medical, Stavanger Norway) was placed in the simulation venue with medical equipment routinely used for simulation-based assessments. Participants had access to a full range of emergency care equipment relevant to their level of practice in a given academic year of study. The simulation venue also contained all electronic equipment routinely used in simulation-based assessments, such as notebook computers, video cameras, and electronic devices connected to the simulator. A second student volunteer was present for each assessment whose role was to assist the assessed student with equipment or procedures but who otherwise did not make any decisions and was not allowed to contribute to decision-making. Students were provided with a brief summary of relevant dispatch information (what type of incident they were dispatched to, the time of day, their response time and travel time to the closest appropriate medical facility).

Assessor visible simulation assessments

Each of the simulation assessors sat at a desk roughly 4–6 m from the simulator and participants, and at different locations allowing each assessor a unique view of the simulator and student. Assessors were drawn from the pool of assessors normally used for simulation assessments and were all external to the Department. An examiner, responsible for prompting the participants and operating the simulation equipment, was also present in the assessment room. Examiners were lecturers from each academic year of study who normally fulfil this function in real assessments and typically had a relevant clinical qualification and quite extensive experience of simulation and simulation assessment.

Assessor not visible simulation assessments

Simulation assessors were situated in a different room and viewed live video and audio footage of the assessment room recorded from two different angles by high-definition video cameras. A separate live video stream showed the patient monitor linked to the mannequin with clinical data such as ECG, respiratory rate, SpO₂ and blood pressure. The examiner was still situated in the assessment room as described above.

Tests

State-Trait Anxiety Inventory (STAI)

The STAI is a validated test of state and trait anxiety [16]. State anxiety refers to anxiety experienced in response to a specific situation in time while trait anxiety refers to a longer-term, more stable sense of anxiety as an individual trait. The STAI consists of 40 questions with responses on a five-point Likert scale. The STAI scores range from a minimum score of 20 to a maximum score of 80 on both the state and trait subscales. The STAI-Y form Y of the STAI was used in this study. Only the state anxiety subscale scores were used.

Heart Rate Variability (HRV)

HRV was used as a physiological measure of stress experienced during simulation assessments and is obtained through analysis of an electrocardiographic (ECG) recording. HRV is controlled by the autonomic nervous system and is subject to a dynamic balance between sympathetic and parasympathetic tone depending on the degree of stress experienced [17]. Three HRV variables were analysed - the standard deviation of all ECG complex intervals (SDNN) is an index of heart rate variability in general, while the square root of the mean of the sum of the squares of differences between adjacent ECG complex intervals (RMSSD) provides an estimate of parasympathetic nervous system (PNS) activity. The high-frequency (HF) spectral component of HRV frequency domain analysis is also an index of PNS activity [17, 18]. There is currently no reliable single HRV variable that can be used as an index of sympathetic nervous system activity alone [18]. Finally, heart rate (HR) in beats per minute was also derived from the ECG.

A small, self-contained portable heart rate monitor (Actiheart, CamNtech, Cambridge, United Kingdom) was used for ECG recordings during simulation assessments. ECG data were downloaded to a notebook computer after recording and analysed using a heart rate variability analysis application (Kubios HRV Standard, Version 3.1, Kubios Oy, Finland).

Salivary cortisol

Cortisol is a biological marker of short-term sympathetic nervous system activity in response to stress, with a diurnal release pattern peaking early in the day and reducing to lower levels at night. Cortisol secretion in response to acute stress increases independent of this diurnal pattern [19, 20]. Salivary cortisol is reliably correlated to serum cortisol levels [21] and was used in this study because it can be non-invasively collected. Participants rinsed their mouths with water 10 min before sample collection which was done by unstimulated passive drool into a collection aid until a sufficient volume of saliva had been collected. After recording the participant number, group, date and time all samples were frozen at -20 °C until laboratory analysis using a commercial assay kit (Salivary cortisol assay, Salimetrics LLC, California, USA).

Data Collection

Data for each academic year group and for the control and experimental conditions in each were collected on different days giving six separate days of data collection. Dates were based on convenience and availability of participants and assessors and data was not collected in counterbalanced order. Eligible and consenting students were requested to not consume alcohol for at least 12 h, not vigorously exercise for four hours and not eat, consume caffeine, or smoke (if applicable) for two hours prior to each assessment. All participants underwent both assessor visible and not visible conditions.

Before each assessment, participants first completed the STAI and then provided a saliva sample. This was followed by fitting of the heart rate monitor and commencement of ECG recording. Participants then completed the assessment and on exiting the assessment room the heart rate monitor was removed and ECG data downloaded. This was followed by completion of the STAI and collection of another saliva sample approximately 5–10 min after the end of the assessment.

Ethical approval was obtained from the University of Johannesburg Faculty of Health Sciences Research Ethics Committee (REC-01-177-2018) for this research. Participation was voluntary and all students gave prospective informed consent for participation. All research methods complied with local research ethics guidelines and regulations.

Data Analysis

Interrater reliability (2.3 above) was assessed by two independent raters by calculating intraclass correlation for all simulation difficulty ratings. Both between condition (control vs. experimental) and within condition (pre- vs. post-assessment) differences were assessed for all dependent variables (STAI state anxiety scores, HRV variables and salivary cortisol levels) with paired t-tests, respectively, when data were normally distributed and with Wilcoxon signed rank tests when not. Data were combined across all academic years. A value of p < 0.05 was considered significant for all hypothesis tests, and IBM SPSS (version 25.0, IBM Corporation, New York) was used for all statistical analyses.

Results

A total of 31 students consented to participate in this study. Ten students were from the third-year group and ten from the fourth-year group. Eleven students were from the second-year group. The data of two students were withdrawn due to non-attendance of all data collection days leaving a total of 29 student participants (94%). Seventeen (59%) were male and 12 (41%) were female. Descriptive data for all groups are shown in Table 2, giving the mean where data where normally distributed and the median where not.

Table 2 Descriptive statistics: all variables

In general, pre- and post-test state STAI score differences were associated with small effect sizes. The assessor visible pre-test score was very similar to the assessor not visible post-test score with assessor visible pre- vs. post-test scores showing a decrease while the opposite pattern was observed with assessor not visible pre- vs. post-test scores (Table 3). While none of the paired differences were significant (Table 3), the greatest effect size was observed in post-tests between control and experimental groups.

Table 3 Significance tests: Assessor condition and Pre-/Post-test

Within the assessor visible group, post-test salivary cortisol levels were slightly higher than pre-test values. In the assessor not visible group, this difference was doubled in the same direction and was significant (Table 3). Neither pre- or post-test differences between these control and experimental conditions were significant, however the post-test comparison was associated with a larger effect size.

All of the HRV variables showed values in the not visible condition suggesting lower heart rate variability (SDNN) and parasympathetic withdrawal (lower mean RMSSD and HF values) in this condition compared to the assessor visible condition, although none of these differences were significant. The only significant difference was for heart rate, where the mean heart rate was significantly higher when assessors were not visible (Table 3).

Discussion

In this study investigating the effect of assessor presence on emergency care simulation assessment stress and anxiety, only one physiological variable - mean heart rate - was found to differ significantly between experimental and control conditions. A significantly increased mean heart rate in the assessor not visible condition suggests that students in this condition experienced more stress than those in the assessor visible condition. While significant differences between conditions in the other physiological variables were not observed, the differences observed do not contradict this difference in heart rate and are consistent with possibly higher stress in the experimental condition, as effect sizes were similar. These include salivary cortisol, which was higher in the post-test assessor not visible condition with a moderate effect size, and all the included HRV variables which indicated lower heart rate variability and parasympathetic tone in the assessor not visible condition, albeit with smaller effect sizes. The above results were unexpected and opposite to what we had hypothesized - increases in stress and anxiety in the assessor visible condition due to socio-evaluative stress from assessors being visible during assessments.

State STAI scores

Mean state STAI scores across both conditions and pre-/post-tests were within the normal range for college students (36.47 ± 10.02 for males and 38.76 ± 11.95 for females) [16] These scores were lower than state STAI scores recorded in previous research with a very similar population and under the same conditions where the median post-test assessor visible state STAI score was 60 (a difference of 23.03) [22]. The only plausible explanation for this difference is that the simulation assessments in our study were mock assessments and did not have any implications for overall academic success or failure while the previous study’s assessments did have real implications. Mean state STAI scores were also marginally lower than those reported in the literature for other research where state STAI was measured during emergency care simulation assessments with a mix of paramedic and emergency medicine or surgical resident participants [23,24,25].

State STAI score pre- and post-test trends in our study were different to those observed in another study designed to assess the effect of assessor visibility on anxiety experienced by nursing students during simulation assessments. Horsley and Wambach observed an increase in post-test scores when assessors were visible but a decrease in post-test scores when assessors were not visible [12]. A study previously conducted with a similar population to ours found a post-test state STAI increase when assessors were present, similar to that of Horsley and Wambach [22]. Another randomized study with nursing students as participants, but not using the STAI for measurement of anxiety, documented a post-test reduction in groups directly observed by assessors and where simulation assessments were video recorded [26].

Pre-test state STAI scores can be taken to infer participants anticipation-related anxiety of a simulation assessment. This may reflect anxiety based on prior assessment experiences, feelings related to their own level of preparation for a particular assessment or just general anxiety based on anticipation of socioevaluative stress and the unknown case that they will be expected to manage. Post-test scores may reflect an extension of the anxiety experienced during the assessment and may be modulated by a participant’s own evaluation of their performance and the consequences that this may have in the future. Consequently, considering the results above and the lack of a consistent pattern in pre- and post-test response across the literature, it is not possible to reach any firm conclusions about the effect of assessor visibility on anxiety. Differences in state-anxiety scores between our study and those discussed above may reflect differences in methodology.

Salivary cortisol

Salivary cortisol levels followed a similar trend in both the assessor visible and not visible conditions with post-test levels in both conditions being higher than pre-test, but only significantly higher in the assessor not visible condition. Although this was similar to the trend observed with pre- and post- test state STAI scores in the assessor not visible condition, it was the opposite of the trend observed for state STAI scores in the assessor visible condition. Despite the significant change between pre- and post-test salivary cortisol levels in the assessor not visible condition, neither test was significantly different compared to those of the assessor visible condition although the post-test difference was associated with a moderate effect size. These results are suggestive of greater stress experienced by participants in the assessor not visible condition but remain inconclusive.

Salivary cortisol has been associated with varied results in research investigating simulation-associated stress [24, 27, 28]. In studies measuring salivary cortisol levels before and after simulations, results generally show trends similar to those in our study – an increase in post-simulation cortisol levels, some of which are significant [24, 27,28,29]. However, these studies did not evaluate the effect of assessor visibility on cortisol levels and were different in other aspects such as the types of simulations used and the clinical experience of participants. It is therefore difficult to directly compare any of these results to those obtained in our study.

Heart rate variability

HRV and HR data showed that participants in the assessor not visible condition experienced significantly higher heart rate, and therefore potentially greater stress, than in the assessor visible condition. Changes in other HRV variables between the two conditions – suggesting decreased parasympathetic tone and decreased overall heart rate variability - corroborated this although none of the other changes were significant.

One other study, by Mills et al., evaluated heart rate in a group of paramedic students participating in simulations with assessors either visible or not visible to students [13]. A higher mean heart rate was observed in the assessor not visible condition although this was not significantly different compared to the assessor visible condition. This was despite a subset of participants indicating in post-simulation interviews that having assessors present during simulations provoked greater stress. Comparison with the two other studies [12, 26] investigating the effect of assessor presence in simulation assessments is not possible as heart rate was not recorded in these studies.

Visible vs. not visible assessors, stress and anxiety

Our observations in this study, that students completing simulation assessments may have experienced greater stress and anxiety when assessors were not visible to them, requires explanation. An important consideration is that students completing simulation assessments when assessor were not visible would still have been aware that they were being observed. Thus, socio-evaluative stress as a factor could still be assumed to be operative under these conditions. However, there is a possibility that being able to see assessors while being assessed may have a moderating effect on the stress and anxiety experienced by students. Alternatively, in the absence of being able to see assessors, students may have their attention diverted from the case at hand by wondering what the assessors reactions might be to their performance particularly when their own self-evaluation is negative. Whether these speculative explanations for our observations are due to characteristics of students or characteristics of the assessors or an interaction of these is difficult to establish without further research.

Limitations

It is possible that observed effects on stress and anxiety were influenced by differences in simulation assessment cases between assessor visible and not visible conditions. Even the simplest simulation assessment cases are complex in nature and it is not possible to match cases exactly in order to eliminate such biases. However, we made significant attempts to limit bias introduced by differences in simulation assessment case complexity in this study between conditions through use of the simulation complexity scoring. The sample size used in this study was based on a calculation that utilised estimates of variance for STAI scores, HRV variables and salivary cortisol that were available at the time. The effect sizes observed ended up being smaller than those calculated a priori from these estimates, which may have increased the probability of a Type II error, however, replication of this study with a larger sample would help to confirm the present preliminary findings. This study utilised a convenience sample of participants from one university assessed using a specific methodology and this limits how generalisable the results may be to different settings due to specific teaching and simulation assessment methodologies. Lastly, the simulation assessments in this research did not carry any academic consequences for students and this may have influenced effects on stress and anxiety, meaning that any changes or the magnitude of responses observed was not as large as might be observed in response to real assessments. However, we used this design due to the ethical implications of conducting research on students during real assessments.

Conclusions

This study investigating the effect of assessor visibility on stress and anxiety amongst emergency care student during simulation assessments has found some evidence suggesting that greater stress and anxiety is experienced when assessors are not visible to students. While there was some evidence of this pattern in all variables measured, only one (heart rate) showed a significant increase in the assessor not visible condition. These results require corroboration with further research before they can be considered as evidence for a recommendation regarding situation of assessors during simulation assessments.