Evaluating multisite multiprofessional simulation training for a hyperacute stroke service using the Behaviour Change Wheel

Background Stroke is a clinical priority requiring early specialist assessment and treatment. A London (UK) stroke strategy was introduced in 2010, with Hyper Acute Stroke Units (HASUs) providing specialist and high dependency care. To support increased numbers of specialist staff, innovative multisite multiprofessional simulation training under a standard protocol-based curriculum took place across London. This paper reports on an independent evaluation of the HASU training programme. The main aim was to evaluate mechanisms for behaviour change within the training design and delivery, and impact upon learners including potential transferability to the clinical environment. Methods The evaluation utilised the Behaviour Change Wheel framework. Procedures included: mapping training via the framework; examination of course material; direct and video-recorded observations of courses; pre-post course survey sheet; and follow up in-depth interviews with candidates and faculty. Results Patient management skills and trainee confidence were reportedly increased post-course (post-course median 6 [IQ range 5–6.33]; pre-course median 5 [IQ range 4.67–5.83]; z = 6.42, P < .001). Thematic analysis showed that facilitated ‘debrief’ was the key agent in supporting both clinical and non-clinical skills. Follow up interviews in practice showed some sustained effects such as enthusiasm for role, and a focus on situational awareness, prioritization and verbalising thoughts. Challenges in standardising a multi-centre course included provision for local context/identity. Conclusions Pan-London simulation training under the London Stroke Model had positive outcomes in terms of self-reported skills and motivation. These effects persisted to an extent in practice, where staff could recount applications of learning. The evaluation demonstrated that a multiple centre simulation programme congruent with clinical practice can provide valuable standard training opportunities that support patient care. Electronic supplementary material The online version of this article (doi:10.1186/s12909-015-0423-1) contains supplementary material, which is available to authorized users.

Results: Patient management skills and trainee confidence were reportedly increased post-course (post-course median 6 [IQ range 5-6.33]; pre-course median 5 [IQ range 4.67-5.83]; z = 6.42, P < .001). Thematic analysis showed that facilitated 'debrief' was the key agent in supporting both clinical and non-clinical skills. Follow up interviews in practice showed some sustained effects such as enthusiasm for role, and a focus on situational awareness, prioritization and verbalising thoughts. Challenges in standardising a multi-centre course included provision for local context/identity.
Conclusions: Pan-London simulation training under the London Stroke Model had positive outcomes in terms of self-reported skills and motivation. These effects persisted to an extent in practice, where staff could recount applications of learning. The evaluation demonstrated that a multiple centre simulation programme congruent with clinical practice can provide valuable standard training opportunities that support patient care.

Mesh terms: Stroke, Patient simulation, Education, Evaluation
Background National clinical guidelines in the UK emphasise the need to establish acute stroke as a clinical priority requiring early specialist assessment and treatment [1]. Management on a specialised acute stroke unit from the time of admission results in 19 % more patients being alive and independent at 1 year [2,3] and 'clot-busting' treatment with thrombolysis within 3 h of stroke onset results in 30 % more patients being alive and independent at 3 months [4]. There has thus been increasing recognition of the importance of timely medical attention in acute stroke management [5][6][7] to facilitate early diagnosis and determination of the aetiology of the stroke (ischaemic or haemorrhagic) in addition to planning treatment strategies aimed at reducing the brain damage caused by the stroke, and preventing complications.

The London stroke strategy
In 2008, a London-specific stroke strategy was published that made a number of recommendations, including implementation of a new model of acute care incorporating eight hyper-acute stroke units (HASUs) that would deliver care in the first 72 h for all suspected stroke patients [8]. The stroke care model was co-created through a series of events with key stakeholders, clinical experts, patients and carers as well as representatives from carer groups. Subsequent to this wide engagement, the new model was introduced in 2010 with each HASU providing: immediate response; specialist assessment on arrival; brain imaging and thrombolysis (if appropriate) within 30 min; high dependency care and stabilisation. Once stable, the patient is transferred to a stroke unit for rehabilitation and discharge to community care.
The centralised model shows early improvement in patient outcomes [9,10]. To support its effectiveness, there was an identified training need for the increased numbers of specialist medical and nursing staff recruited to the HASUs.

The HASU simulation training programme
Following a pilot course, [11] four independent simulation centres provided innovative, multisite training using a standardized protocol-based curriculum based on the London Cardiac and Stroke Network Model [12] and curriculum-mapped against the DoH's Stroke-specific education framework [13]. The training was designed to provide an immersive, dynamic environment in which learners could practice general and stroke-specific skills without risk to patients [14]. Simulation training is established in healthcare as a valid teaching modality for students, trainees and multiprofessional groups [15]. However multiple-site programmes are rare, as is longitudinal follow-up of candidates [16,17].

Aims and objectives
This paper reports on an independent evaluation of the HASU programme. Primary aims were to evaluate design and content, delivery, impact upon learners and transferability to the clinical environment, including making recommendations for faculty development and course improvement.
The main evaluation questions were: What were the reported behavioural outcomes from the course?
What evidence is there for sustained effect over time? What recommendations can be made with respect to delivery/evaluation of similar courses?

Conceptualising the training intervention
Michie et al. [18] outline a model, the Behaviour Change Wheel (BCW), for designing and evaluating effective interventions. The 'wheel' involves determinations about target behaviours (hub), identification of intervention functions (inner ring) and consideration of policy context (outer ring). Specific behaviour change techniques (BCTs; [19]) are conceptualised as the 'active ingredients' by which an intervention achieves its aims. Intervention functions (e.g. training, education) are understood both in relation to the behaviours they target and the policy contexts (e.g. guidelines, regulatory aspects) within which they take place.
In the present study, this model allowed for: a) making the service-provision context explicit b) conceptualising target behaviours; c) studying the behaviour change techniques applied; and d) describing the modes of delivery and findings of the evaluation. Table 1 shows a model of the intervention using the framework. Each simulated exercise and 'debrief' was rich in facilitated Behaviour Change Techniques (BCTs); these main agents of change are illustrated together with indications of evaluation metrics. Table 2 shows specific procedures undertaken as they relate to various components of the conceptual evaluation model. Table 2 shows a mixed methods design including before and after survey sheet for trainees (see Additional file 1) and follow-up interviews with staff and faculty. Interviewees were randomly contacted from an attendee list, stratified for basic/advanced course, profession (doctor/ nurse) and time passed since attendance at the course (<3 months; >6 months). Interviewed faculty were chosen purposively, forming a criterion-based sample [20] able to reflect on the design and delivery of the course and its outcomes.

Evaluation procedures
All participants gave prior written informed consent to be contacted for follow-up interview and for survey data to be aggregated for research purposes in accordance with the terms of the Data Protection Act 1998. Ethical approval was given by the Hospital Research Ethics Committee (South London REC 3; approval ref 09/28), under the terms of the UK NHS Research Ethics Service.
All interviews were recorded with permission using a digital voice recorder. The interviews were then transcribed verbatim for data analysis.

Survey tool
All candidates were given a pre-and post-course questionnaire using 7-point Likert scaled items, adapted from a standard satisfaction measure [21], and some open ended questions. Three scaled items on communication skills, leadership skills and confidence in managing emergency situations were asked identically both before and after the course. Post course perceptions were also gathered on aspects such as course enjoyment and the most valuable learning outcomes.

Interview tool
Candidate interviews were conducted by telephone and were progressively cued to move from general perceptions to an exploration of specific topics of interest: post-course perceptions; reflections on how the learning objectives were met; what information had been retained; and outcomes in terms of knowledge, skills, personal development/motivation and specific descriptions of patient care episodes.
Faculty interviews were conducted face-to-face according to a semi-structured, topical interview protocol that focused on behavioural needs, training design, delivery  and modes of facilitation in the simulated learning environment. The mean interview length was 21 min (candidates) and 29 min (faculty), with a range 16 to 38 min.

Observation
Observational data were gathered to support the investigators in achieving a complete sense of the scope, scale, and overall experience of the course. Observational data were gathered in three ways: 1. Principal investigators [AR, GR] attended two basic and two advanced courses at multiple training centres and observed all activities. 2. Security permission was established to access audio/ video data at one of the centres for the purpose of detailed post-hoc analysis. Data were held on a secure stand-alone drive to protect confidentiality. 3. Secure audio files from a second participating centre were accessed to allow for detailed post-hoc analysis of 6x simulation 'debriefs'.

Analysis
Analysis of pre-and post-course survey data took place using appropriate analysis of variance techniques in IBM SPSS v22.0. Simulation scenarios and debriefs were observed and analysed using SMOTS (Scotia Medical Observation and Training System). Qualitative data from direct observations and interviews and were analysed thematically using HyperRESEARCH 3.5.2 data analysis software. Coding frames were developed from learning objectives and iterated inductively as data were gathered, with discussion of routine and exceptional responses to ensure reliability of cross-coding.

Simulation training procedures
The collaboratively-developed multiprofessional programme operated as a 'basic' and 'advanced' course based on simulated scenarios using a manikin (with computer-controlled vital signs that allowed changes in patient characteristics to be simulated) and/or standardized patient actors. Attendees directly participated in at least one scenario and watched others via a live video-feed. Each simulated scenario lasted up to 15 min and was followed by a group debriefing session lasting approximately 40 min which followed the SaIL debrief diamond model [22] of description, analysis and application to practice [23]. Table 3 shows clinical scenarios employed and specific learning objectives for the basic and advanced courses.
Although specific clinical competencies were included, the main learning objectives were more general psychological and physical capabilities (see conceptual model in Table 1): knowledge and understanding (e.g. of stroke signs, symptoms and 'mimics', and timeframes for treatments); patient management (e.g. communication skills, team working skills, acting on risk assessment results), and motivational aspects (empowering/enabling staff to increase their confidence in their own professional capabilities).

Candidates
Seventy-seven candidates attended the Basic HASU course. These were 38 doctors (1-7 years post-qualification) and 39 Registered Nurses at various career stages. Seventy-five candidates with a similar range of seniority attended the Advanced HASU course: 32 doctors and 41 nurses (two missing). All candidates filled in surveys (n = 152) but most items have a small amount of missing data.

Candidate experiences
Overall, candidates enjoyed the course and felt it was relevant to their clinical practice (both items median rating 7/7; IQ range 6-7). Enjoyment and relevance were closely related (Spearman's rho = .712**; p < .001).
Doctors were more likely to rate the course as enjoyable (z = 1; n = 137; NS), and find it relevant to practice (z = 1.2; n = 129; NS), than nurses, but a Mann Whitney test for independent groups shows these differences were not significant.
However, this multiprofessional interaction with the scenarios also tended to arise in debrief and in interview. Observations showed nurses having some difficulty in following their usual protocol for assessing patients when the manikin cannot move limbs, does not have a grip response etc. Doctors tended to interact more by talking/taking history (the manikin has voice functionality), by directing treatment, and by reference to notes. This holistic assessment seems more amenable to modelling via simulated practice via manikin than the more direct 'caring' provided by nurses (see Discussion). All participants recognised that the manikin gives limited biofeedback cues in providing the context for training realistic stroke care: because stroke is so dependent on clinical sign things […] is there facial weakness, is this arm moving or not […] it just makes it a little bit false […] I think the thing with an actor is you can replicate stuff a lot […] (doctor 14); If you look at somebody, you eyeball them, you can see the difference, you can't do that with a dummy (doctor 16); it's very hard to look at the symptoms in the manikin […] which is not really manifesting the right things (nurse 11).

Capability, motivation and opportunity
Reported competency outcomes were assessed via the survey on three seven-point scale items given before and after the courses: How good are your clinical communication skills?; How good are you leadership skills?; How confident do you feel managing emergency situations? (reliability analysis: Cronbach's alpha = .897). Figure 1 shows a comparison of composite scores for these three 'before and after the course' items. Figure 1 shows that these competencies were rated higher after the course.
There were no significant differences (interactions) for course (basic or advanced) or by profession (doctor or nurse) on any of these reported improvements.
Overall there was also a slight increase post-course in the perceived usefulness of particular 'early warning' scoring systems employed during the training (for those who n = 111, z = 6.42, P < .05).
Qualitative data from survey, interview, and video observations were examined to explore this reported learning further, and to look specifically at the behavioural change techniques employed. Behaviours identified can be grouped thematically into five specific areas: verbalising thoughts; calling for help; teamwork; assertiveness; and situational awareness. Table 4 shows these five main behavioural themes and change techniques employed, with examples of selfreported outcomes synthesised from the survey, interview, and video/audio file observation of the training episodes (all quotations are verbatim). The main general mechanism for addressing behaviour comes via the post-exercise 'debrief ', facilitated by senior faculty using the events that have just been observed as a platform. Post-hoc descriptions of scenario timelines and/or use of video playback are used to facilitate peergroup discussions of strategies employed, alternative approaches, personal experiences, emotional aspects and action points to take away. One nurse stressed the importance of the video playback and reflective 'debrief ' mechanisms: We had the scenario played back [and could see] there was a period where we all in the midst of trying to get things done and there was no focus on the patient themselves. So it was quite interesting seeing that because obviously when you're doing something you just focus on whatever you're doing, […] you can't see it from the outside until it's actually played back. I think playback is quite useful. (Nurse 13) Anxiety was felt variably by candidates, but for most was formative, in that it allowed for practice in what would be a stressful situation in real life: it's good to be put in that situation I guess, because when you're in a real life scenario that's what you need to be able to do sometimes (Doctor 17). This is a behavioural change agent in itself, via the regulation of negative emotions.

Delivery and context
The course was designed and marketed as a training event with educational content, however it can also be seen that there is a persuasive element (many discussions focused on how people felt, what they thought patients would feel) and a modelling element (senior faculty provide an example to aspire to). In part, what people 'took away' depended on their prior expectations. People who came looking for detailed clinical knowledge of stroke medicine tended to ask clinical questions, take notes, request protocols etc. and thus formed outcomes in their own ideal, taking away technical/skill-based messages ("noticing blood pressure changing, […] whether to give Labetalol, when to put on the Alteplase, we were experiencing different ways that we deal with this"). Others who were experienced stroke practitioners tended to have more non-clinical discussions about teamwork and situational awareness. For example senior nurses said of the course: "made you think on your feet"; "was more about effective communication […] than actual stroke care"; "it was more about managing situations to me"; "it was just reinforcing to be clear and focused on what you were doing […]".
Faculty raised a number of issues that apply to similar courses that seek to work in a standardized way across multiple centres, including the balance between overall standardization/reliability and courses being tailored to fit in with the ethos, facilitation style and corporate identity of each individual centre.

Application to post-course practice
Follow up interviews were intended to explore whether there was any reported transference of simulation-based knowledge and skills to practice. Good application of learning in practice was reported, with candidates specifically recalling: refocusing on 'door to needle' time from presentation to treatment; prioritizing during a thrombolysis call;  One of the learning points is just how difficult it is for telephone conversations to provide useful results to both sides (Doctor AV) The communication skill for a rapid interaction has to be borderline pedantic (Doctor I); Communications skills is really, really important, and someone has to listen and someone has to lead (Nurse I)

Managing and planning
Modeling/peer review/problem solving/coping planning/feedback on behaviour

Timelines of scenarios/identification of exemplars/elicitation of strategies employed in practice
The A&E and the stroke team can actually work as a team to actually achieve that door to thrombolysis time of 10 min… To change the practice I would probably get the A&E consultants and the A&E matron to actually be involved in this management of stroke so that the delivery of care can be given within the target time (Doctor I); I've got this new mindset of going in, that I want to go in and it's about being mentally prepared for any situation (Nurse I); it's quite difficult to (plan ahead) because you have your own patient to look after, and at the same time co-ordinate the ward (Nurse AV); You need to know when to call for help, and when you are at the limit of what you can do on your own (Doctor AV)

Breaking down institutional barriers
Restructuring social environment/ self-affirmation/reframing/identity/ emotional consequences/pros and cons/social support/feedback on behaviour the pertinence of the stroke course to using stroke scoring systems; confidence in initiating stroke management; and verbalizing or vocalizing thoughts in stressful situations.
Most candidates were enthusiastic about the experience and its general motivational effects (I think it's a very good experience […] it stimulates you and gets you to get everyone else enthusiastic, Nurse 2).
More recent attendees were quite explicit about enhanced capability (technical and psychological skills). As one nurse attending a thrombolysis call reported: I was just imagining the situation from [simulation], so that made it really quite, it made me feel quite confident to do it, because I knew exactly what I was looking for in terms of […] watching out for the signs of anaphylaxis and then monitoring throughout, just to ensure there was no deterioration.
[…] Just a general sense of, I've done this, I just did this in simulation. I can do it again… I was ready for the situation, do you know what I'm saying? If ever they had had a reaction, I was really clear in my mind of how I would actually react to that (Nurse 3). A doctor had a similar feeling post-course after having helped in a critical care situation: Yesterday's case we had in, in resus-it was pertinent having been on the course, getting the stroke team down quickly and starting the scoring system and whether the patient would be thrombolysis or not (Doctor 9).

Sustained effects
Qualitative follow-up data show that the course was a driver for ongoing reflective practice, even when, at around 6 months post-course, the ability to recall specific messages or learning 'on the day' was seen to degrade. In one instance, a nurse explained how she and her colleagues had instituted changes to their practice for stroke patients after first one, and then several, of them had attended the course and subsequently had time to compare their experiences: It's assessing, assessing how we can get our time down, but still getting everything done […] trying to get the 'door to needle' time down, but also not missing anything, because you still also need to get your patient's history […] (nurse 2).
Overall, confidence was reportedly increased in the months following the course and reflected the various behavioural themes in Table 4 such as verbalising thoughts (I think, sometimes when you vocalise aloud your thoughts, I think, you know, even if you don't have the answer you might trigger a thought process from somebody else on the team; Doctor 7), managing situations (because of the course I felt more confident in initiating management that maybe other house officers wouldn't have been able to do; Doctor 1); situational awareness (it's being aware of my surroundings, knowing who I've got, just making sure… you've got to be specific and use the people that you've got there and according to the skills that they have; Nurse 13).

Discussion
This paper reported on a structured evaluation of a multi-centre simulation training programme for hyper acute stroke medicine.

Post-simulation effects
Candidates rated content and design highly in mixed response survey sheets. Candidates reported increased confidence after training. This has been consistently reported across a range of clinical scenarios and specialties [24]. Open-ended responses show nurses reported specific learning based on assertiveness, and were receptive to discussions about social barriers to communication in multiprofessional teams. This training encouraged them to be aware of situations where information flow may be restricted, leading to takeaway messages about what has been termed 'flattened hierarchy' [25].
We have reported some general positive evaluations both in post-course survey and follow up interviews, and some specific relation of the intervention and its mechanisms of change to improved behaviours in the new service delivery environment.
Following up after time has elapsed in important, because transfer to practice [26] and the sustainability (or decay) of training-acquired skills or knowledge over time [27] has been a relatively neglected area of simulation research [28,29]. Thus, "[…] some of the challenges that still exist in simulation-based medical education include […] measuring the effect of simulation and the transference of knowledge from the simulated environment to real life" [30].

Capability, motivation and opportunity
The evaluation was designed to study how the programme actively addressed capability, opportunity and motivation. We have reported that learner expectations vary with respect to whether they are learning skills or acquiring knowledge. Motivational effects (e.g. reported change of 'mindset') also emerge during post-course reflection. Despite growing use of simulated modalities and positive evaluations, relatively few simulation studies have used a theoretically driven evaluation within which intended outcomes can be framed. It is important that trainers are clear whether each episode is targeting physical/technical and/or psychological skills, motivation, opportunity (including social) or combinations of all three.
In particular, some candidates (those less experienced in clinical management of stroke) expected knowledge-based outcomes and others (experienced) expected to focus on extended skills. It is important that simulation designers and facilitators anticipate multiprofessional differences in receptivity and that this feeds back into design so that learning objectives are tailored to specific student needs [31]. Learning from participatory simulation must address contextual and systems factors, which in turn give rise to emergent outcomes [32]. There are ongoing discussions in the literature about compatibility of different learning outcomes that tend to be mixed in simulation, based on knowledge or skill acquisition and/or the aim to provide transformative personal experience [33].
Further, we have identified some important behavioural change techniques that recur in simulated performance (and in particular in peer 'debriefing'). These include forming good habits such as verbalisation and taking 'time outs' , and social restructuring around professional hierarchies, to encourage speaking out about safety.
However we also identified variance in debriefing styles and approaches, despite standard materials and learning outcomes. Cantrell (2008) reports that different styles per se are not problematic [34] as long as, as in this study, debriefing takes place immediately following scenarios while perceptions are still salient. However active engagement of candidates is key [35], and some faculty reports suggested some styles were more didactic in nature. Results also showed that there were also some senior clinical faculty who were not trained specifically in simulation debrief (especially with relation to nonclinical or 'non-technical skills') and this has been recognised as important for best practice [36].
Michie et al. [19] conclude that further 'elucidation of how content, mode, and context of delivery interact in their impact on outcomes is a key research goal for the field of behavioral science'. The physical environment in this mode of training delivery is not 'the same for everyone'. Observations and interviews showed an interaction whereby fidelity was reportedly more of an issue for nurses. Nurses had more difficulty in following their usual protocol for assessing patients; the manikin cannot move limbs, does not have a grip response etc. Doctors tend to perform in a space less contiguous to the manikin via taking history (the manikin has voice functionality), directing treatment and referring to notes. It may seem self-evident that "fidelity is the degree to which a simulation replicates or approaches reality" [37], but there are various social and psychological dimensions that need to be taken into account and it is not easy in applied courses of this type to assess fully whether thresholds for fidelity are being met [38].

Strengths and limitations
Results in this paper are supportive of standardised multiprofessional training for stroke medicine and indicate benefits. Reported post-course confidence ratings, for example, are increased and internally reliable (as in previously reported studies [39]) but these are insufficient in themselves as evidence for patient benefit. We have triangulated findings with follow-up interviews on reports about the use of these skills in practice.
The next stage might be to examine, for example using case comparison, patient outcomes in units using standard educational modalities against simulation training. This in effect is a complex intervention and attribution of variance in outcomes to specific training events is difficult, but empirical tests of context-mechanism-outcome configurations have been recommended [40].