Table 8 Benefits and limitations of ISS reported in the included studies

Realism: Real setting enabling teams to perform with actual equipment and resources

Locate and test equipment

Facilitates safe transitions to new facilities

Possibility of selection bias / lack of randomisation of participants

Releasing participants from other clinical duties while undertaking ISS may limit generalisability to the clinical setting

Possibility of training effect for pre-announced ISS: enabling participants to prepare (as opposed to unannounced ISS)

Observers and video reviewers are unblinded to the type of participant and setting

Lack of usual clinical distractions and lack of assessment over the full 24-h period may limit generalisability

High cancellation rate in high acuity areas

Fidelity issues in key components of task completion (lack of adequate visual cues regarding patient output, monitor function and appearance)

Scenarios can be rated independently my numerous assessors

Small sample sizes and inadequately powered studies prevents formal statistical analysis

Problems with recruitment

Use of non-validated assessment tools

Confounding factors: unable to capture all of the complex all factors which contribute to outcomes in a changing climate of practice

Some tasks are capable of high fidelity and reproducibility

Inadequate collection of participant demographic data which may impact the findings (e.g., number of shifts worked or days off before the data collection, participation in more than one scenario, prior simulation training)

Assessment of tasks with clearly defined and established standards

Potential ‘refresher effect’ if participants repeatedly engage in ISS simulations

Efforts to standardise ISS activities may limit including variation between scenarios and tasks

Evaluation of ISS assessment in one setting reduces generalisability to the wider context

Identified opportunities for improvement in the clinical setting

Lack of formal measures to translate the findings into practice and inform action plans

Enables more team members to participate compared to off-site training

Variation in teams when evaluating pre / post assessments over longer follow-up periods

Measuring communication in an established team maybe difficult as the need for communication decreases

Lack of availability of experienced non-technical skills assessors

Maintaining participant anonymity in smaller sites / studies

Performance anxiety, reluctance to participate