A mixed-methods program evaluation of the Alda Healthcare Experience- a program to improve healthcare team communication

Background Communication among interprofessional healthcare worker teams is critical to ensure a thriving and resilient workforce. We will evaluate the implementation and effectiveness of the Alda Healthcare Experience (AHE), a novel medical improvisation (improv) workshop designed to improve interprofessional communication skills among healthcare professionals. The AHE workshop includes a two-hour experiential training workshop led by an improv specialist and a clinical co-facilitator. In July 2022 we began implementing the AHE workshop by training 18 clinical co-facilitators who will co-facilitate the workshops for 550 healthcare workers from five hospital departments at Stony Brook University Hospital over the course of a year and a half. Using mixed-methods, we will conduct an Effectiveness-Implementation Hybrid Design project that includes an outcome evaluation (effectiveness) and a process evaluation (implementation). Methods Our outcome evaluation will assess the impact of the AHE workshop on short- and long-term improvement in interprofessional communication, stress, and professional fulfillment. The process evaluation component will examine programmatic, organizational, and individual facilitators or barriers to effective implementation of the AHE workshop. Qualitative methods will include dimensional analysis employing individual interviews of 20–40 AHE Project Participants, 5–10 Selected Informants, and all the clinical co-facilitators. Quantitative methods will use a quasi-experimental longitudinal design with an intervention group and surveillance of a control group (wait-list) and repeated assessments using validated instruments measuring communications skills, professional fulfillment, stress, burnout, uncertainty tolerance, and teamwork. Discussion Effective and efficient communication within healthcare teams is fundamental to building team cohesion that, in turn, supports individual resilience and builds positive organizational culture. The AHE program is an innovative approach to improve interprofessional healthcare communication and reduce healthcare worker burnout. In addition to institutional buy-in, rigorous evaluations of medical improv programs are necessary as a critical step in making such programs scalable. Trial registration N/A

is feasible, acceptable, engaging, and effective in generating expected participation outputs and achieving its immediate outcomes [28,29]. However, rigorous evaluations of medical improv programs for interprofessional healthcare workers have not been published in the peerreviewed literature. Evaluating programmatic aspects of such medical improv workshops (process evaluation) and of its impacts (outcome evaluation) are necessary critical steps in making such programs scalable.

Project aims & design
Grounded in implementation science and program evaluation [30], the present project will evaluate the implementation and effectiveness of the AHE, a medical improv workshop designed to improve communication skills among healthcare professionals. Specifically, using mixed methods, we will conduct an Effectiveness-Implementation Hybrid Design project [31] that includes [1] an outcome evaluation to determine the impact of this workshop on communication skills among healthcare professionals (effectiveness), and [2] a process evaluation to explore interprofessional communication and its effects on AHE implementation fidelity (implementation).
Our outcome evaluation will assess the impact of the AHE workshop on short-and long-term improvement in several relevant outcomes, including interprofessional communication, stress, and professional fulfillment. We will use an embedded experimental explanatory sequential mixed methods design for implementation and evaluation [32,33]. The evaluation will use participatory action research as a guiding framework and will include both qualitative methods (e.g., dimensional analysis employing individual interviews) and quantitative methods (e.g., quasi-experimental longitudinal [34][35][36]). In the first phase (see Fig. 1), we will deliver the AHE workshops and conduct a quantitative prospective outcome evaluation to assess the intervention effects using a quasi-experimental approach with an intervention group and surveillance of a control group (wait-list).
Next, our mixed-methods process evaluation will examine programmatic and individual barriers or facilitators to effective implementation of the AHE medical improv communication workshop. This evaluation will consist of qualitative interviews and quantitative analyses to explore organizational-, departmental-, and individual-level factors, related to interprofessional communication and its effects on delivery, fidelity, and effectiveness of the intervention. We will use our quantitative findings to inform the selection of AHE Informants (key AHE Stakeholders and AHE Project Participants) for qualitative in-depth interviews focused on interprofessional communication and the AHE workshop experience. In addition, we will conduct repeated brief interviews with clinical co-facilitators after AHE workshop sessions. Qualitative interviews will be aligned with dimensional analysis method [37,38].

Alda healthcare experience intervention
The AHE workshops will target approximately 550 healthcare professionals and nurses-both practicing professionals and trainees. Physicians, resident physicians and fellows, and practicing nurses will be drawn from five Stony Brook University Hospital Departments (Surgery, Medicine, Obstetrics/Gynecology, Family Medicine, and Pathology) whose leaders have indicated their support for this project. Participation in the AHE workshop will be mandatory and free; participants will also receive continuing education credits (e.g., Continuing Medical Education, Continuing Nursing Education) for their attendance. All personnel trained in the AHE workshop will be eligible to participate in the program evaluation.
As suggested in the literature [14], each workshop is co-facilitated by two leaders, an improv-trained facilitator and a clinically trained facilitator. Using a train-the trainer approach, clinical professionals will be nominated by leadership from each participating department to receive specialized training by the Alda Center team to enable them to co-facilitate the workshops and be ambassadors of change in their department. Approximately 18 clinical co-facilitators will be trained by going through 2 3-hour group sessions followed by 1-hour (maximum) 1:1 coaching. The clinical co-facilitator training includes experiencing the AHE workshop as a participant (novel to the AHE). Participation in the evaluation component, in addition to the training, will be voluntary.
The Alda Center for Communicating ScienceⓇ has assigned three improvisation facilitators to this project. All three have extensive experience designing and facilitating experiential learning using the Alda MethodⓇ, and have taught prior versions of this curriculum. In addition to facilitating the workshops, one facilitator Based on the fundamentals of medical improv and experiential learning, the AHE program includes a twohour in-person workshop comprised of a series of improvisational exercises that focus on specific communication skills (recognizing the give and take of relational communication, listening to collaborate, reflecting values that are shown through passion, thinking creatively and generativity about circumstances) followed by a discussion about practical application in healthcare [14][15][16]. The exercises are immersive and participatory -participants work in pairs or groups to complete each exercise, and then engage in a full group discussion guided by openended questions from the facilitators. At key points in the workshop, participants are given written prompts to reflect. Follow-up one-hour virtual workshops will be conducted 1-3 months after the workshop in groups of up to 30 aimed to refresh the principles of improvisation to the workshop participants and introduce to a new tool to help them strategize a process to start challenging interprofessional conversations with colleagues.
Based on pilot data, to ensure maximum participation, groups will include 7-10 healthcare professionals, an improv facilitator, and a clinical co-facilitator. There will be approximately 68 separate AHE in-person workshops with a total of n = 100 healthcare professionals trained by the end of 2022, n = 350 more in 2023, and n = 100 more in the first quarter of 2024.

Participants
Three types of participants will be involved in the present evaluation: AHE participants and stakeholders, clinical co-facilitators, and improv facilitators (see Table 1). AHE participants who choose to participate in the evaluation will be asked to join the project and complete online questionnaires. AHE Project Participants will receive $5 gift cards for each completed questionnaire. We expect 85% of the healthcare professionals that take part in the AHE workshop to agree to become AHE Project Participants for a baseline of N = 468 (n = 234 in each group). In addition, 20-40 AHE Project Participants and AHE Stakeholders will be chosen to be AHE Informants and take part in qualitative interviews. Of the AHE Informants, 5-10 will be chosen as Selected Informants and participate in a subsequent focus group. AHE Informants will be purposefully sampled based on individual characteristics and informed by the quantitative findings. AHE Informants will be chosen to reflect different professions, career levels, departments, racial and ethnic backgrounds, and gender identities. They will receive a $15 gift card for their time. As part of their role, all clinical co-facilitators will be interviewed following each AHE workshop they lead. With their agreement, these interviews will be used in real-time to improve the facilitation process as well as be used for evaluation purposes. Each co-facilitator will be eligible to receive a $3000 stipend for their roles (e.g., training, co-facilitating 4-6 AHE workshops, interviews). Improv facilitators will complete a brief online questionnaire following each AHE workshop session they lead. Their hourly pay is commensurate with experience.
The project was reviewed by the Stony Brook University IRB (IRB2022-00231) and approved as a project that is not considered human subject research based on Common Rule determination. Nonetheless, prior to completing the baseline survey, AHE workshop participants were presented with a brief description of the evaluation project, their involvement, confidentiality, and compensation. Participants who were interviewed were presented similar information about their involvement in the evaluation verbally as well.

Procedures-quantitative
For the AHE Project Participants, evaluation procedures include online surveys collected during key time points for the AHE (see Fig. 1). AHE participants who are Hourly interested, will be asked to become AHE Project Participants and complete four online surveys over the course of 2 years (see Table 2) that will be administered through Qualtrics. Surveys will take approximately 3-5 minutes to complete and will repeatedly assess constructs related to teamwork, burnout, stress, well-being, and communication. The following previously validated measures will be included in the surveys:

Communication skills
Communication will be assessed at all time points using the eight item Perceived Interpersonal Communications Skills Scale (PICSS) that was previously developed by the Alan Alda Center for Communicating Science [28]. Each item assesses a different skill related to flexible empathic communication (e.g., "Beginning a conversation with an open mind"). Participants are asked to rate their skill level for each communication item on a Likert-type scale ranging from 1= "not skilled at all" to 5 = "very skilled". This scale has shown validity in previous pilot studies (Cronbach's alpha is between 0.89 and 0.91; 28). The overall score is represented as the mean of responses.
Higher scores indicate higher levels of communication skill.

Cohesion
Closeness with the healthcare team will be measured at time points 1, 3, and 5 using the Inclusion of Other in the Self Scale (IOS; 39). On a single item, respondents choose one of seven pairs of circles, labeled "Self " and "Other", with differing amounts of overlap (1 = "no overlap" to 7 = "most overlap"). Higher scores indicate greater selfperceived cohesion between participants and others in their healthcare team.

Teamwork and response to errors
Teamwork and response to errors will be measured at time points 1, 3, and 5 using the Surveys on Patient Safety Culture questionnaire (SOPS; 40). Two subscales of the SOPS will be included in the current evaluation: three items measuring Teamwork (e.g., "In this unit, we work together as an effective team") and four items measuring Response to Errors ("When an event is reported in this unit, it feels like the person is being written up, not the problem"). Participants are asked to respond to each statement on a Likert-type scale ranging from 1 = "strongly disagree" to 5 = "strongly agree". The Cronbach's alpha of the Teamwork subscale is 0.76. The Cronbach's alpha of the Response to Errors subscale is 0.83. Higher scores indicate greater teamwork and effective responses to errors.

Uncertainty tolerance
The Uncertainty Tolerance Scale (UTS; 41,42) will be used to assess uncertainty tolerance at time points 1 and 5. The UTS includes two subscales: five items measuring Uncertainty Tolerance (Cronbach's alpha is 0.70) and three items measuring Need for Predictability (Cronbach's alpha is 0.73; 28). Respondents indicate the degree to which they agree or disagree with each of eight statements (e.g., "I like change and excitement"), on a scale ranging from 1 = "strongly disagree" to 6 = "strongly agree". Higher scores indicate a greater need for predictability of a greater intolerance of uncertainty.

Stress
The Short Perceived Stress Scale (PSS-4; 43) will be used to assess self-reported stress at time 1, 3, and 5. Respondents will be asked to indicate how often each of four stated situations had occurred within the past month (e.g., In the last month, how often have you felt that you were unable to control the important things in your life?) on a Likert-type scale ranging from 0 = "never" to 4 = "very often". Cronbach's alpha for the PSS-4 is typically between 0.60 and 0.82 [44]. Higher scores indicate greater perceived stress.

Work exhaustion and professional fulfillment
Work exhaustion and professional fulfillment will be measured at time 1, 3, and 5 using the Stanford Professional Fulfillment Index (PFI; 45). We will use two subscales included in the PFI. The first subscale is Burnout which includes four items related to work exhaustion (e.g., "Physically exhausted at work"). The second subscale is Professional Fulfillment which includes five items related to satisfaction with work (e.g., "I feel happy at work"). Participants respond to each statement on a Likert-type scale ranging from 0= "not at all true" to 4 = "completely true" or 0= "not at all" to 4 = "extremely. " The Cronbach's alpha of the Burnout subscale is 0.92 and the Cronbach's alpha of the Professional Fulfillment subscale is 0.91 [45]. Higher scores on the Burnout subscale indicate greater work exhaustion. Higher scores on the Professional Fulfillment subscale indicate greater fulfillment.

Well-being
The PROMIS Meaning and Purpose self-report scale [46] will be used to measure well-being at time points 1, 3, and 5. Responses to each of four items (e.g., "My life has meaning") are rated on a likert-type scale ranging from 1 = "strongly disagree" to 5 = "strongly agree", or 1= "not at all" to 5 = "very much"). The Cronbach's alpha for this scale is 0.90 [46]. Higher scores indicate greater wellbeing.

Excitement about AHE workshop
The extent to which AHE Project Participants are excited about participating in the workshop will be measured at time point 1 using a Visual Analogue Scale. Participants are asked to indicate "Using the scale, please indicate how excited you are about participating in the Alda Healthcare Experience communication workshop?" on a sliding scale from 0 = not excited at all to 100 = Extremely excited.

Engagement in workshop
The extent to which AHE Project Participants were actively engaged in the workshop will be measured at time 2 for Wave 1 and time 4 for Wave 2 using a Visual Analogue Scale. Participants are asked to indicate "How engaged you were in today's workshop" on a sliding scale from 0 = not engaged to 100 = fully engaged.

Background data
AHE Project participants' sociodemographic background (age, gender identification, race and ethnicity, degree/ credentials, and career level) will be assessed at time point 1. In addition, participants will be asked whether they have any prior experience with improv, and asked to indicate what kind of experience they have.

Employee engagement
Department-level employee engagement will be indexed by absenteeism (e.g., works days missed) and retention (e.g., turnover rates) every 6 months, starting with the 6 months before T1 and ending 6 months after the T5.

Debrief Improv facilitators surveys
Improv facilitators will complete a brief (2 minutes) online survey at the end of each AHE workshop session to document the workshop environment (e.g., distractibility, disruptions, perceived cohesion, initiation of questions, willingness to engage). Several outcomes (i.e., group size, disruptions, perceived cohesion, and engagement) will be collected, using original measures designed for this project.

Procedures-qualitative
All qualitative data collection and management procedures will be carried out using institutionally secured and password protected video-conferencing and data storage software by a team of three to five project team members trained and supervised by the project qualitative lead co-investigator. Qualitative interviews will be conducted virtually by a project team member trained in qualitative data collection. Interviewers will use semi-structured interview and discussion guides to ensure data elicitation on topics central to the aims of the project, which will be iteratively refined in alignment with emerging concurrent analysis and theoretical sampling. Once completed and audio-recorded, each interview will be transcribed using an institutionally approved transcription software service, and at least two project team members will independently align audio-recordings with their corresponding transcripts to ensure verbatim accuracy and remove identifiable information.
For the qualitative process evaluation of the AHE, we will use theoretical sampling to select 20-40 individuals for in-depth individual interviews, from the pool of AHE Project Participants and AHE Stakeholders that agree to being contacted to participate in interviews. Should we face recruitment challenges resulting in fewer than expected interviews, theoretical sampling will enable sufficient depth of data for analysis through application of it during interviewing by "steering questions in the direction of emergent theorizing, " as well as after data collection is complete by theoretically sampling "within the data" [32 , p. 8]. An initial portion of the AHE Informants will be selected from the Wave 1 department to allow a critical mass for analysis (between T1-T3), then additional interviews will be conducted with Wave 2 AHE Project Participants and AHE Stakeholders (between T4-T5) to ensure sample heterogeneity in terms of clinical department, as well as overall data saturation. Each interview will last approximately 60 minutes and interview questions will focus specifically on participants' experiences of interprofessional communication, burnout, and the AHE workshop.
For the qualitative stakeholder participatory validation (after T5), we will sample 5-10 Selected Informants from the AHE Informants to participate in a focus group interview. The focus group will last approximately 60 minutes and will aim to establish participatory validation of the theoretical findings generated from individual interviews as well as any proposed modifications to the AHE rooted in these findings or the quantitative outcome evaluation.
For the clinical co-facilitator analysis (between T1-T5), each clinical co-facilitator will participate in brief semistructured interviews after every AHE workshop they facilitate. These individual interviews will last approximately 10-30 minutes and are designed to meet multiple project-related goals. First, the interviews will function as a practical opportunity to learn of implementation issues requiring troubleshooting prior to subsequent AHE workshop sessions, across all clinical and improv co-facilitators. Second, the interviews are guided by the underlying goal of understanding the longitudinal experiences of clinical co-facilitators involved in delivering improv-based communication workshops.

Analysis plan-quantitative
Power analysis for the quantitative methods is based on our ability to conduct Structural Equation Modeling (SEM) among all participants and to detect differences between Wave 1 and Wave 2. For SEM we will need a final sample of n = 200, therefore, having a baseline sample of n = 234 AHE project participants (85% of all AHE participants in each Wave) in each Wave will ensure a sufficient final sample size of n = 210 in each wave (10% attrition over time). Having n = 234 in each Wave will also give us enough power to compare the two groups after controlling for individual and program level factors using techniques such as Hierarchical Linear Modeling.
Quantitative outcome analyses will examine interindividual changes over time among AHE participants as well as departmental-level differences between intervention and control groups using independent and dependent t-tests. We will also examine sequential processes in outcomes using SEM. Our quantitative process evaluation will examine moderators for AHE impacts using PROCESS macros. Effects of individual (e.g., age, staff position), program (e.g., participants' engagement, facilitator-rated group dynamics, time of day), and organizational (e.g., department, team structures) level factors on program outputs will be tested using nested techniques such as Hierarchical Linear Modeling. To reduce bias associated with attrition from the study over time, we will analyze missingness patterns and handle any missing data accordingly (e.g., pairwise deletion, multiple imputation).

Analysis plan-qualitative
Inductive analysis of all qualitative data will be guided by dimensional analysis, an interactionist methodology [37,38]. Each interview transcript will be coded over multiple passes by members of the project team trained in qualitative data analysis. First pass coding will involve a Gestalt read of the data and analytic memo writing. In subsequent passes, the data will be analyzed by generating open, axial, and theoretical codes. Constant comparison technique [47], dimensionalization [37,38], and theoretical sampling [32] will guide each phase of iterative analysis, facilitated by regular dialogic engagement sessions to ensure research team coherence in analytic procedures.
Inductive analysis of the data collected for the qualitative process evaluation of the AHE will help identify theoretical relationships between key dimensions of healthcare team communication, safety issues, and attitudes towards communication training including workshop groups environment. For the qualitative stakeholder participatory validation, analysis of the data will seek to uncover dimensions relevant for program implementation -utility, feasibility, acceptability, and scalability. Analysis of clinical-co-facilitator interviews, will aim to identify theoretical relationships among concepts central to improv communication training among interprofessional healthcare workers.

Discussion
This mixed-methods process and outcome program evaluation will examine the implementation and the effects of an innovative medical improv workshop. Over 550 healthcare professionals from five hospital departments will participate in the two-hour AHE workshops that will be led by an improv facilitator and a clinical co-facilitator followed by a 1-hour virtual booster session several months later. The robust Effectiveness-Implementation Hybrid Design project will help uncover the effects of the program, under what circumstances and for whom it works best, and what can be done to improve it. Project findings will inform the creation of large-scale implementation and dissemination protocols.
During the Coronavirus Disease 2019 (COVID-19) pandemic, the mental and physical burden placed upon healthcare workers was -and continues to be -heavy. There is evidence that clinicians experienced high rates of stress and burnout even before the pandemic began, suggesting that these concerns will not be fully resolved even if the number of COVID-19 cases is reduced [1][2][3]. However, the adoption of the AHE workshop is unique and innovative because it breaks away from traditional models to address burnout.
The successful implementation and rigorous evaluation of such a program requires institutional buy-in and sponsorship by organizational leaders. Leaders set the example by participating in the workshop as individuals, by providing ongoing collaboration to maximize participation across the organization, and by providing access to data required to rigorously evaluate the impact of the workshop. For this project, our institution's Patient Experience Officer is serving as our executive sponsor, completed clinical co-facilitator training, and is facilitating our team's access to organizational employee engagement data (i.e., rates of retention and absenteeism); furthermore, five department chairs enthusiastically agreed to have their staff participate and nominated co-facilitators to take a more active part in the program. The importance of having clinical co-facilitators lead medical improv workshops has been previously established [14]. From our experience, having the clinical co-facilitators from our institution and from the participating departments is as critical. This increases the authenticity of the workshops, helps the trainees be more at ease, and allows for application of the improv exercises to the institutional and departmental context. Moreover, these trained clinical co-facilitators will be ambassadors for change in communication culture within their department.
While the current project will address short-and longterm outcomes, sustainability of the communication impacts of the AHE workshop are limited to the oneyear follow-up of the planned evaluation. In addition, the workshop is only delivered to five hospital departments and mainly delivered to practicing and training physicians and nurses and do not include ancillary health professionals (e.g., physical therapists, occupational therapists) who often have important roles in healthcare teams. While these five departments represent key departments within the organization and the healthcare professional make up the majority of the healthcare teams, it is unclear if this amount of reach will be enough to create sustainable cultural changes in our institution. One way to address this would be to make the AHE mandatory to all direct care staff and examine its institutional-level effects. However, this might be challenging in a highly matrixed environment where faculty are under different leadership than nursing, trainees, and ancillary staff.
Effective and efficient communication within healthcare teams is fundamental to building team cohesion that, in turn, supports individual resilience and builds positive organizational culture. Healthcare teams are very well trained to communicate well during patient "codes" (e.g., when a patient has a cardiopulmonary arrest requiring a team of healthcare providers to rush to the specific location and begin immediate resuscitation efforts). In contrast, most healthcare professionals graduate from their respective professional schools with little training in the interprofessional team communication skills that are needed to provide high quality and equitable healthcare to patients. Moreover, the majority of current performance expectations and career advancement pathways for healthcare professionals-especially those in academic medical centers-fail to reward collaboration. There have been dramatic changes in the workload and workflow for healthcare professionals over the past 20 years. Healthcare policymakers and organizations now need to invest in the needs of the workforce to ensure a robust and resilient future for us all.