- Research article
- Open Access
- Open Peer Review
Anaesthesiology students’ Non-Technical skills: development and evaluation of a behavioural marker system for students (AS-NTS)
BMC Medical Educationvolume 19, Article number: 205 (2019)
Non-Technical Skills (NTS) are becoming more important in medical education. A lack of NTS was identified as a major reason for unsafe patient care, favouring adverse events and team breakdown. Therefore, the training of NTS should already be implemented in undergraduate teaching. The goal of our study was to develop and validate the Anaesthesiology Students’ Non-Technical Skills (AS-NTS) as a feasible rating tool to assess students’ NTS in emergency and anaesthesiology education.
The development of AS-NTS was empirically grounded in expert- and focus groups, field observations and data from NTS in medical fields. Validation, reliability and usability testing was conducted in 98 simulation scenarios, during emergency and anaesthesiology training sessions.
AS-NTS showed an excellent interrater reliability (mean 0.89), achieved excellent content validity indexes (at least 0.8) and was rated as feasible and applicable by educators. Additionally, we could rule out the influence of the raters’ anaesthesiology and emergency training and experience in education on the application of the rating tool.
AS-NTS provides a structured approach to the assessment of NTS in undergraduates, providing accurate feedback. The findings of usability, validity and reliability indicate that AS-NTS can be used by anaesthesiologists in different year of postgraduate training, even with little experience in medical education.
In patient care, both Technical Skills (TS) and Non-Technical Skills (NTS) are necessary to maintain best practice as well as reach a high level of expertise . TS are routinely taught in trainee programs. However, evaluation and assessment of NTS have been missing for a long time [2, 3]. NTS are defined as “the cognitive, social and personal resource skills that complement technical skills and contribute to safe and efficient task performance” .
Adverse events in high-risk settings often take place due to deficiencies in NTS , which has been shown in various fields such as aviation and nuclear energy [6,7,8] and has also been confirmed for medical care: up to 70% of adverse events are due to human errors [9,10,11]. In order to reduce medical errors, good NTS and improved teamwork are essential .
NTS interventions and mostly feedback on NTS have shown to have positive effects on team performance, concluding that good patient care requires TS and NTS, which have been found to correlate in crew resource management [13, 14] and to foster improved clinical performance like quicker problem solving in simulated operating theatre environment . The positive effects of NTS also encompass enhanced patient safety. Salas et al. showed in a meta-analysis the positive effects of NTS on team members’ reactions and attitudes related to teamwork safety . Other studies pointed out positive effects of NTS on clinical performance (TS) and on patient outcome like surgical complications and morbidity .
Knowledge of necessity and benefits of NTS lead many institutions to emphasize the importance of interpersonal skills [13, 16,17,18,19,20,21] and these departments implemented crew-resource training programs in their curricula, primarily focussing on NTS during postgraduate training .
The training of NTS should not only be focused in postgraduate training - undergraduate curricula and education should already integrate the concept of “patient safety”, directly addressing, teaching and assessing NTS [23, 24]. Only a few studies have investigated the effect of teaching NTS in undergraduates. Hagemann et al. showed that even one brief seminar had positive effects on undergraduates’ NTS .
The German Association for Medical Education has acknowledged the repeatedly expressed need of NTS implementation in undergraduate education by publishing a “Learning Objective Catalogue for Patient Safety in Undergraduate Medical Education”, which has the aim to unify the curricular targets in German medical faculties . However, a concrete curriculum, implementation or teaching strategy for NTS in undergraduate education is not given yet. In addition, due to a missing conventional rating tool a structured assessment of NTS in undergraduates is often lacking.
To create and realize an implementation and teaching strategy for NTS in undergraduates, at first the structured assessment of NTS with a robust method is necessary , in order to provide specific and formative feedback and to monitor the learning progress in undergraduates.
Several rating tools for assessing NTS in medical professionals are available. They are helpful to provide feedback which is not based on “gut feeling” and to speak “the same language” during the feedback process [26,27,28,29,30,31]. However, the existing rating tools, such as the Anaesthetists’ Non-Technical Skills (ANTS) , are very complex and not designed for undergraduates or junior residents, as ANTS is developed for experienced anaesthesiologists to rate trainees who have reached certain TS, which limits its broad application in undergraduate education. A feasible application is further limited as for raters a two-day training with the rating scheme is required. The use of ANTS delays the feedback loop, as the NTS ratings are based on video clips of the training sessions, which are evaluated after the training .
The goal of this study was to develop a rating tool to assess NTS in undergraduate education in emergency medicine and anaesthesiology: Anaesthesiology Students’Non-Technical Skills (AS-NTS). The tool is supposed to be feasible and easily handled without the necessity for video recording or extended instructions and trainings for the user.
This study was performed at the Department of Anaesthesiology in the University Medical Center Hamburg-Eppendorf, Germany. The study was conducted in the period of Janurary 2017 to December 2017. Undergraduates and residents in anaesthesiology participated in this study with a stepwise design in order to develop and validate a rating tool for NTS in undergraduates in anaesthesiology. The development took place in four steps (Table 1), empirically grounded on qualitative and quantitative research methods:
Review of published literature (expert group)
Focus group and half-structured interviews
Implementation and validation
Table 1 shows a scheme of the conducted developmental steps and underlying research methods.
A detailed explanation of the development is given in the Additional file 1.
Study setting: assessment of NTS during emergency and anaesthesiology training sessions
The undergraduate curriculum of the Medical Faculty of Hamburg has implemented emergency training sessions in nearly every semester, in order to experience the students in emergency medicine. We use high fidelity simulators (Rescue Anne Laerdal) which are suitable for training technical skills such as endotracheal intubation, defibrillation or drug administration.
NTS were assessed in four different training sessions (Advanced cardiac life support I, II, III and operation room simulation) of four different semesters. In each training session a pre-existing set of standardized simulation scenarios were used (13 in total, a detailed description of the simulation scenarios is provided in the Additional file 1).
The simulation scenarios are standardised and solely for each type of training session. For example, the training session “Advanced cardiac life support II (ACLS II)”, which is held in the 3rd year of undergraduate education, includes the scenarios: “Hyperkalaemia”, “Hypothermia” and “Aspiration”.
In each training session every student is assigned to a small group which rotates through each simulation scenario.
With each following semester, the simulation scenarios require more advanced TS and NTS. In order to rule out that low NTS skills are due to technical skills being not proceduralized we decided to test our rating system in students who had already passed the basic life support training in following training sessions:
○ Advanced cardiac life support I (ACLS I: 2nd or 3rd year undergraduates, pre-existing simulation scenarios: 2; number of rated simulation scenarios for interrater agreement analysis: 20)
○ Advanced cardiac life support II (ACLS II: 3rd year undergraduates, pre-existing simulation scenarios: 3; number of rated simulation scenarios for interrater agreement analysis: 24)
○ Advanced cardiac life support III (ACLS III: 4th year undergraduates, pre-existing simulation scenarios: 5; number of rated simulation scenarios for interrater agreement analysis: 23)
○ Operation room (OR) simulation (3rd or 4th year undergraduates, pre-existing simulation scenarios: 3; number of rated simulation scenarios for interrater agreement analysis: 31)
In each training session, the undergraduates are divided into groups of three. Each of these groups rotates through the simulation scenarios of the training session. In each simulation scenario one student takes the role of the physician, the other two that of paramedics or anaesthetic co-workers. The student in the role of the physician leads the team and delegates basic tasks such as establishing the monitoring, preparing defibrillation and other required medical procedures to the other team members. Therefore, only this student was evaluated by the two or three supervising anaesthesiologists, using the AS-NTS.
Raters and interrater reliability
Twenty-one anaesthesiologists (Table 2) conducted the training sessions during the study period. In 67 emergency simulation scenarios two of them rated the students independently, in the 31 operating room simulation scenarios three raters were involved. The raters who rated the same simulation scenario, did not discuss their results while rating, in order to rule out cognitive bias.
The rater teams changed frequently based on the teaching schedule. The raters all received a five-minute introduction into the AS-NTS.
The interrater-reliability was investigated using a two-step approach.
In the first step a classical analysis of interrater-reliability was conducted, analysing data from rating pairs. To rule out agreement by chance, the intraclass correlation (ICC) from six pairs of raters were calculated, which had rated at least six simulation scenarios together. The first analysis included 67 of the total of 98 simulation scenarios. In five of the six pairs the first author (R1) took part (Table 3).
In the second step of the interrater-reliability analysis, the whole data set from the 98 simulation scenarios was analysed. To rule out that either the strong involvement of R1 in the development process of AS-NTS or the medical training had an effect on the interrater reliability, data was aggregated across raters being in the same year of training. This allowed us to investigate the relationship between medical expertise and rating agreement (Table 4).
Statistical analysis was performed using IBM SPSS Statistics Version 23.0. Intraclass correlation (ICC) was used for ordinally scaled data and Cohens Kappa for nominally scaled data to calculate interrater reliability. We used the one-way random effects model to calculate the ICCs . Values of ICC and kappa below 0.40 are interpreted as poor correlation, between 0.40 and 0.59 as fair correlation, between 0.60 and 0.74 as good correlation and between 0.75 and 1.00 as excellent correlation .
Development of the AS-NTS assessment tool
The literature search resulted in 12 different NTS important in anaesthesiology (Table 5). The discussions in the focus- and expert group revealed, that not all of these NTS are highly important for undergraduates. During the field observations some NTS were difficult to observe. Using the results of the focus group discussions we defined new dimensions specifically for undergraduates, symbiosing some pre-defined NTS:
Planing tasks, prioritising and conducting
Teamwork: exchanging information and leading the team
Table 5 displays the created list of the NTS and the further conducted steps which were decisive for the inclusion of each skill. The last column illustrates which NTS is part of ANTS and AS-NTS. Figure 1 displays the definition of the NTS.
The first dimension of AS-NTS:
“Planning tasks, prioritizing and problem solving” resulted as a compound, mainly formed by pre-defined NTS dimensions “Decision making” and “Task management” (Fig. 2). The elements that were considered important in undergraduates and therefore created the basis to define the first dimension of AS-NTS are highlighted.
“Coordinating team members”, “communication” and “Leadership” were regarded as highly important in the focus group and performance could be observed in different levels during the field observation, therefore these elements created the basis for dimension two of ANAESTHESIOLOGY STUDENTS’ NON-TECHNICAL SKILLS: “Teamwork and leadership”.
Leadership, defined as the skill of directing others, coordinating, managing workload and motivating others  is often separated into two independent dimensions allowing for the assessment of different leadership styles  distinguishing between task orientation and team orientation. In this leadership model, “Task orientation” is closely related to our first two AS-NTS dimensions, therefore we decided to add “Team orientation” as third and final dimension of the AS-NTS. “Teamwork and leadership” emphasizes the collaborative processes to perform a task, whereas “Team orientation” focuses on the collaborative processes to build a team.
In contrast to the ANTS, performance is rated in the AS-NTS on the three dimensions and not on the level of skills. However, the underlying skill structure was used to give behaviorally anchored rating examples to clarify what a “good” or “poor” performance on each dimension might look like. In the final AS-NTS assessment tool (Fig. 3), a five-point Likert scale was used for each dimension, although the ANTS system has a four-point scale . Cook et al. could show that, in regard to reliability and interrater reliability, there are no differences in 5- and 9- point scales in mini-clinical evaluation exercise .
Feasibility and content validity of the scoring system
The interviews with eight anaesthesiologists in their first year of residency, who used both the AS-NTS and ANTS in simulation training (including video tapings), showed that no further dimension had to be added to the AS-NTS rating tool (step 4). Furthermore, they confirmed the feasibility of AS-NTS and concluded that in undergraduates, as well as in the first 2 years of residency in anaesthesiology, the ANTS system is too complex.
Without video tapings it is nearly impossible to complete ANTS, due to time shortness. This was already pointed out by the developers [5, 32]. The eight anaesthesiologists discovered the rating of the videos to be very time consuming and delaying the feedback loop.
These anaesthesiology trainees decided to continue their postgraduate training curriculum using AS-NTS, rather than ANTS, for their first 2 years of residency.
The results from an additional evaluation questionnaire, completed by 21 anaesthetits, who had used the rating tool at least three times in undergraduate medical education, confirmed that the AS-NTS was feasible and practical (Additional file 1: Table S1). Additionally, they rated the importance of each dimension of AS-NTS.
The content validity index for each dimension was calculated, reflecting the proportion of relevance . The calculated content validity index for the first dimension of AS-NTS was 0.9, for the second dimension 0.95 and for the third dimension 0.8. A content validity index of 0.75 or higher is considered as “excellent” .
The interrater reliability reached high levels of agreement (Table 6), except for dimension two, in the group of 3rd vs. 5th year residents (fair correlation). The ICC indicated a high rater agreement regardless of educational experience, training in anaesthesiology or familiarity with the AS-NTS rating tool.
The development of the AS-NTS was performed in a stepwise approach, beginning with a review of pre-existing literature, continuing with focus group analysis and field observation, and ending with implementation and validation.
During the field observations some skills proved to be difficult to observe and excluded from ANAESTHESIOLOGY STUDENTS’ NON-TECHNICAL SKILLS, based on developmental guidelines of assessment tools described by Abell et al., who recommend items to be excluded, if they are not observable in at least 50% of field observations .
Nonetheless, the excluded skills are part of most existing NTS taxonomies and regarding the importance of these skills, one might argue that they should still be taught and addressed in undergraduate education.
However, acquiring and refining NTS is an individual and ongoing process . Therefore, in undergraduate training pre-cursors of some NTS should be assessed and evaluated. Further, most of the taxonomies from which the NTS list was extracted, are developed for postgraduate training- focussing on specialist level, which makes these skills not one to one transferable to undergraduates.
Those skills should be focused in more advanced educational levels, mostly in postgraduate training. Nevertheless, the aim of the study was to include as many NTS as possible into the ANAESTHESIOLOGY STUDENTS’ NON-TECHNICAL SKILLS, in order to assess them in undergraduates to provide accurate feedback, enhancing the learning process.  For this goal, skills were redefined during the development of ANAESTHESIOLOGY STUDENTS’ NON-TECHNICAL SKILLS, symbiosing some pre-defined NTS and focusing more on pre-cursors and underlying elements of skills. This adaptation process was not solely based on the expert- and focus groups- but was supported by literature and resulted in the new dimensions of ANAESTHESIOLOGY STUDENTS’ NON-TECHNICAL SKILLS, specifically designed for undergraduates. The adaptation step was necessary, as some NTS are highly important but not fully developed in undergraduates.
Transferred to the first dimension of ANAESTHESIOLOGY STUDENTS’ NON-TECHNICAL SKILLS, two main dimensions of described NTS (“Decision making” and “Task management”) were symbiosed to the first AS-NTS dimension “Planning tasks, prioritizing and problem solving”.
This might lead to the assumption a specific assessment of these skills is not possible, as they are assessed in the same dimension of performance and in pre-existing rating tools, they are separately assessed.
This objection can be warded by focusing on the developmental rational and existing literature:
“Decision making” is a complex skill which is divided into subskills and rated separately by some behavioral taxonomies [28, 29, 32]. Flowerdew et al. pointed out that it is not only making the decision which is of great importance, but also following the effects caused by the decision, like planning and prioritizing tasks to conduct the decision . Here, “Decision making” is directly linked to the dimension “Task management”, which includes the elements: “Planning and preparing, prioritizing, providing and maintaining standard, identifying and utilizing resources”. Conducting the elements of “Task management” is the following consequence after “a decision is made”.
The comprehensiveness of “Decision making” and “Task management” regarding the training level of undergraduates was pointed out repeatedly, leading to the exclusion of these dimensions and focusing on some elements of these skills.
Identifying options is necessary to solve a problem- in the decision making loop the risks and benefits of the solving strategy are re-evaluated – this concept of decision making is applicable in more complex scenarios than in undergraduate simulation training. Therefore, the focus group discussed and agreed to include “problem solving” as a less complex proxy of decision making into the first dimension of AS-NTS.
The strength of this study was scrutinizing the interrater reliability from different viewpoints. The interrater reliability was not only defined by a few designated raters, as in classical approaches. A two-step approach was chosen to analyse rater agreement, simultaneously examining if personal background (a.e. year of anaesthesiology training or experience in medical education) might influence ratings. First, agreement of rater pairs were analysed with a sufficient number of ratings, excluding agreement by chance, then data aggregation of the full sample was conducted based on anaesthesiology training, to calculate the interrater reliability. AS-NTS achieved excellent Interrater reliability, only within the group of 5th year vs. 3rd year anaesthesiology residents, the ICC and Cohens Kappa were “good” and only “fair” for dimension two of AS-NTS.
Data aggregation in the full sample, supports the result that the rating agreement is detached from anaesthesiology training and experience in medical education, fostering the usability of AS-NTS.
The strong involvement of Rater 1 in the assessment of the interrater reliability might lead to the assumption one rater could influence all the other raters. Regarding our results, this is not the case, as data aggregation across all raters, in which Rater 1 is not represented predominantly, showed high agreement on ratings as well.
The dimensions of the final version of AS-NTS achieved excellent content validity indexes according to a guideline for evaluating standardized assessment instruments . However, one weakness of the study is that the calculation of the content validity index was only calculated from the evaluation of twenty-one anaesthesiologists. Although there is no predefined sample size required to establish content validity , the effect of agreement by chance is higher in a small sample.
The AS-NTS has high potential to improve NTS assessment in undergraduate education and ultimately patient safety, because a lack of NTS leads to adverse events in high-risk settings . A recent study by Hagemann et al. showed that NTS in undergraduate students are improved after only one seminar . Due to its good feasibility, the AS-NTS could be applied to all students as a standardised assessment and feedback tool.
The AS-NTS has only been tested in German language at one institution with a limited number of teachers. Further studies should be conducted to establish the validity, reliability and feasibility of the English version.
AS-NTS provides a structured approach to the assessment of NTS in undergraduates, providing accurate feedback. The findings of usability, validity and reliability indicate that the AS-NTS can be used by anaesthesiologists in different year of postgraduate training, even with little experience in medical education.
Availability of data and materials
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.
Anaesthesiology Non-Technical Skills
Anaesthesiology Students’Non-Technical Skills
Content Validity Index
Weinger MB. Experience≠ ExpertiseCan simulation be used to tell the difference? The Journal of the American Society of Anesthesiologists. 2007;107(5):691–4.
Gaba DM, DeAnda A. The response of anesthesia trainees to simulated critical incidents. Anesth Analg. 1989;68(4):444–51.
Fletcher G, McGeorge P, Flin RH, Glavin RJ, Maran NJ. The role of non-technical skills in anaesthesia: a review of current literature. Br J Anaesth. 2002;88(3):418–29.
Flin RH, O'Connor P, Crichton M. Safety at the sharp end: a guide to non-technical skills: Ashgate Publishing, Ltd.; 2008.
Flin R, Patey R, Glavin R, Maran N. Anaesthetists’ non-technical skills. Br J Anaesth. 2010;105(1):38–44.
Salas EBC, Bowers CA. Team training in the skies: does crew resource management (CRM) training work? Hum Factors. 2001.
Flin R, Martin L, Goeters K-M, Hormann H, Amalberti R, Valot C, et al. Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Human Factors and Aerospace Safety. 2003;3:97–120.
Flin R. Safe in their hands?: licensing and competence assurance for safety-critical roles in high risk industries; 2005.
Pierre MS, Hofinger G, Buerschaper C. Human Factors und Patientensicherheit in der Akutmedizin: Springer; 2014.
Hoffmann B, Siebert H, Euteneier A. Patient safety in education and training of healthcare professionals in Germany. Bundesgesundheitsbl Gesundheitsforsch Gesundheitsschutz. 2015;58(1):87–94.
Müller MP, Hänsel M, Stehr SN, Fichtner A, Weber S, Hardt F, et al. Six steps from head to hand: a simulator based transfer oriented psychological training to improve patient safety. Resuscitation. 2007;73(1):137–43.
Risser DT, Rice MM, Salisbury ML, Simon R, Jay GD, Berns SD. The potential for improved teamwork to reduce medical errors in the emergency department. Ann Emerg Med. 1999;34(3):373–83.
O'Connor P, Campbell J, Newon J, Melton J, Salas E, Wilson KA. Crew resource management training effectiveness: a meta-analysis and some critical needs. Int J Aviat Psychol. 2008;18(4):353–68.
Riem N, Boet S, Bould MD, Tavares W, Naik VN. Do technical skills correlate with non-technical skills in crisis resource management: a simulation study. Br J Anaesth. 2012;109(5):723–8.
Doumouras A, Hamidi M, Lung K, Tarola C, Tsao M, Scott J, et al. Non-technical skills of surgeons and anaesthetists in simulated operating theatre crises. Br J Surg. 2017;104(8):1028–36.
Salas E, Wilson KA, Burke CS, Wightman DC, Howse WR. Crew resource management training research, practice, and lessons learned. Reviews of human factors and ergonomics. 2006;2(1):35–73.
Schmutz J, Manser TD. Do team processes really have an effect on clinical performance? A systematic literature review. Br J Anaesth. 2013;110(4):529–44.
Cannon-Bowers JA, Salas E. Team performance and training in complex environments: recent findings from applied research. Curr Dir Psychol Sci. 1998;7(3):83–7.
Flin R, Patey R. Improving patient safety through training in non-technical skills. British Medical Journal Publishing Group. 2009;339:b3595.
Gordon M, Darbyshire D, Baker P. Non-technical skills training to enhance patient safety: a systematic review. Med Educ. 2012;46(11):1042–54.
Hagemann V, Herbstreit F, Kehren C, Chittamadathil J, Wolfertz S, Dirkmann D, et al. Does teaching non-technical skills to medical students improve those skills and simulated patient outcome? Int J Med Educ. 2017;8:101.
Gaba DM, Howard KJ, Gaba SKDM, Fish KJ, Howard SK. Crisis management in anesthesiology 1994.
Walton M, Woodward H, Van Staalduinen S, Lemer C, Greaves F, Noble D, et al. Republished paper: the WHO patient safety curriculum guide for medical schools. Postgrad Med J. 2011;87(1026):317–21.
der Medizinischen Wissenschaften SA. Projekt „Zukunft Medizin Schweiz “–Phase lll–Aus-und Weiterbildung in Patientensicherheit und Fehlerkultur. Unter Mitarbeit von Barbara Gassmann, Jacques Haller, Martin Täuber, Peter M Suter Schweizerische Akademie der Medizinischen Wissenschaften. 2007.
Kiesewetter J, Gutmann J, Drossard S, Salas DG, Prodinger W, Mc Dermott F, et al. The learning objective catalogue for patient safety in undergraduate medical education–a position statement of the Committee for Patient Safety and Error Management of the German Association for Medical Education. GMS journal for medical education. 2016;33:1.
Kim J, Neilipovitz D, Cardinal P, Chiu M, Clinch J. A pilot study using high-fidelity simulation to formally evaluate performance in the resuscitation of critically ill patients: the University of Ottawa critical care medicine, high-Fidelity simulation, and crisis resource management I study. Crit Care Med. 2006;34(8):2167–74.
Andersen PO, Jensen MK, Lippert A, Østergaard D, Klausen TW. Development of a formative assessment tool for measurement of performance in multi-professional resuscitation teams. Resuscitation. 2010;81(6):703–11.
Yule S, Flin R, Paterson-Brown S, Maran N, Rowley D. Development of a rating system for surgeons’ non-technical skills. Med Educ. 2006;40(11):1098–104.
Thomas E, Sexton J, Helmreich R. Translating teamwork behaviours from aviation to healthcare: development of behavioural markers for neonatal resuscitation. BMJ Qual Saf. 2004;13(suppl 1):i57–64.
Healey A, Undre S, Vincent C. Developing observational measures of performance in surgical teams. BMJ Qual Saf. 2004;13(suppl 1):i33–40.
Cooper S, Cant R, Porter J, Sellick K, Somers G, Kinsman L, et al. Rating medical emergency teamwork performance: development of the TEAM emergency assessment measure (TEAM). Resuscitation. 2010;81(4):446–52.
Fletcher G, Flin R, McGeorge P, Glavin R, Maran N, Patey R. Anaesthetists’ non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003;90(5):580–8.
Lynn MR. Determination and quantification of content validity. Nurs Res. 1986.
Koch GG. Intraclass correlation coefficient. Encyclopedia of statistical sciences; 1982.
Cicchetti DV. Guidelines, criteria, and rules of thumb for evaluating normed and standardized assessment instruments in psychology. Psychol Assess. 1994;6(4):284.
Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R. Assessment of clinical performance during simulated crises using both technical and behavioral ratings. The Journal of the American Society of Anesthesiologists. 1998;89(1):8–18.
Salas E, Burke CS, Stagl KC. Developing teams and team leaders: strategies and principles. Leader development for transforming organizations: growing leaders for tomorrow; 2004. p. 325–55.
Flowerdew L, Brown R, Vincent C, Woloshynowych M. Development and validation of a tool to assess emergency physicians’ nontechnical skills. Ann Emerg Med. 2012;59(5):376–85.e4.
Mishra A, Catchpole K, McCulloch P. The Oxford NOTECHS system: reliability and validity of a tool for measuring teamwork behaviour in the operating theatre. BMJ Qual Saf. 2009;18(2):104–8.
Frankel A, Gardner R, Maynard L, Kelly A. Using the communication and teamwork skills (CATS) assessment to measure health care team performance. Jt Comm J Qual Patient Saf. 2007;33(9):549–58.
Guise JM, Deering SH, Kanki BG, Osterweil P, Li H, Mori M, et al. Validation of a tool to measure and promote clinical teamwork. Simulation in healthcare: journal of the Society for Simulation in Healthcare. 2008;3(4):217–23.
Flin R, Martin L. Behavioral markers for crew resource management: a review of current practice. Int J Aviat Psychol. 2001;11(1):95–118.
Kines P, Andersen LP, Spangenberg S, Mikkelsen KL, Dyreborg J, Zohar D. Improving construction site safety through leader-based verbal safety communication. J Saf Res. 2010;41(5):399–406.
Fernandez R, Kozlowski SW, Shapiro MJ, Salas E. Toward a definition of teamwork in emergency medicine. Acad Emerg Med. 2008;15(11):1104–12.
Mitchell L, Flin R. Non-technical skills of the operating theatre scrub nurse: literature review. J Adv Nurs. 2008;63(1):15–24.
Ottestad E, Boulet JR, Lighthall GK. Evaluating the management of septic shock using patient simulation. Crit Care Med. 2007;35(3):769–75.
Mickan SM, Rodger SA. Effective health care teams: a model of six characteristics developed from shared perceptions. Journal of interprofessional care. 2005;19(4):358–70.
Boreham N, Shea C, Mackway-Jones K. Clinical risk and collective competence in the hospital emergency department in the UK. Soc Sci Med. 2000;51(1):83–91.
Cosby KS, Roberts R, Palivos L, Ross C, Schaider J, Sherman S, et al. Characteristics of patient care management problems identified in emergency department morbidity and mortality investigations during 15 years. Ann Emerg Med. 2008;51(3):251–61. e1.
Kachalia A, Gandhi TK, Puopolo AL, Yoon C, Thomas EJ, Griffey R, et al. Missed and delayed diagnoses in the emergency department: a study of closed malpractice claims from 4 liability insurers. Ann Emerg Med. 2007;49(2):196–205.
Kaissi A, Johnson T, Kirschbaum MS. Measuring teamwork and patient safety attitudes of high-risk areas. Nurs Econ. 2003;21(5):211.
Schenkel SM, Khare RK, Rosenthal MM, Sutcliffe KM, Lewton EL. Resident perceptions of medical errors in the emergency department. Acad Emerg Med. 2003;10(12):1318–24.
Stella D, Hendrie J, Smythe J, Graham I. Critical incident monitoring is a useful quality improvement tool for the emergency department. Emergency Medicine Australasia. 1996;8(4):215–9.
White AA, Wright SW, Blanco R, Lemonds B, Sisco J, Bledsoe S, et al. Cause-and-effect analysis of risk management files to assess patient care in the emergency department. Acad Emerg Med. 2004;11(10):1035–41.
Senior B, Swailes S. Inside management teams: developing a teamwork survey instrument. Br J Manag. 2007;18(2):138–53.
Malec JF, Torsher LC, Dunn WF, Wiegmann DA, Arnold JJ, Brown DA, et al. The mayo high performance teamwork scale: reliability and validity for evaluating key crew resource management skills. Simul Healthc. 2007;2(1):4–10.
Morgan PJ, Pittini R, Regehr G, Marrs C, Haley MF. Evaluating teamwork in a simulated obstetric environment. Anesthesiology: The Journal of the American Society of Anesthesiologists. 2007;106(5):907–15.
Cooper S, O’carroll J, Jenkin A, Badger B. Collaborative practices in unscheduled emergency care: role and impact of the emergency care practitioner—qualitative and summative findings. Emerg Med J. 2007;24(9):625–9.
Apker J, Mallak LA, Gibson SC. Communicating in the “gray zone”: perceptions about emergency physician–hospitalist handoffs and patient safety. Acad Emerg Med. 2007;14(10):884–94.
Blake RR, Mouton JS. Grid® principles versus Situationalism: a final note. Group & Organization Studies. 1982;7(2):211–5.
Cook DA, Beckman TJ. Does scale length matter? A comparison of nine-versus five-point rating scales for the mini-CEX. Adv Health Sci Educ. 2009;14(5):655.
Martuza VR. Applying norm-referenced and criterion-referenced measurement in education: Allyn & Bacon, Incorporated; 1977.
Hitzier R, Honer A, Maeder C. Die institutionalisierte Kompetenz zur Konstruktion von Wirklichkeit.
Asbury J-E. Overview of focus group research. Qual Health Res. 1995;5(4):414–20.
Basch CE. Focus group interview: an underutilized research technique for improving theory and practice in health education. Health Educ Q. 1987;14(4):411–48.
Morgan D. The focus group guidebook: sage publications; 1997.
Pelz C, Schmitt A, Meis M, editors. Knowledge Mapping als Methode zur Auswertung und Ergebnispräsentation von Fokusgruppen in der Markt-und Evaluationsforschung. Forum Qualitative Sozialforschung/Forum: Qualitative Social Research; 2004: Deutschland.
Bogner A, Littig B, Menz W. Introduction: expert interviews—an introduction to a new methodological debate. Interviewing experts: Springer; 2009. p. 1–13.
Jun M, Peterson RT, Zsidisin GA. The identification and measurement of quality dimensions in health care: focus group interview results. Health Care Manag Rev. 1998;23(4):81–96.
Dorussen H, Lenz H, Blavoukos S. Assessing the reliability and validity of expert interviews. European Union Politics. 2005;6(3):315–37.
Zwick MM, Schröter R. Konzeption und Durchführung von Fokusgruppen am Beispiel des BMBF-Projekts „Übergewicht und Adipositas bei Kindern, Jugendlichen und jungen Erwachsenen als systemisches Risiko “. Fokusgruppen in der empirischen Sozialwissenschaft: Springer; 2012. p. 24–48.
Abell N, Springer DW, Kamata A. Developing and validating rapid assessment instruments. Oxford: Oxford University Press; 2009.
Stewart GL, Barrick MR. Team structure and performance: assessing the mediating role of intrateam process and the moderating role of task type. Acad Manag J. 2000;43(2):135–48.
Sevdalis N, Hull L, Birnbach D. Improving patient safety in the operating theatre and perioperative care: obstacles, interventions, and priorities for accelerating progress. Br J Anaesth. 2012;109:i3–i16.
Goeters K-M. Evaluation of the effects of CRM training by the assessment of non-technical skills under LOFT. Human Factors and Aerospace Safety. 2002;2(1).
Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007;30(4):459–67.
The authors would like to thank all medical teachers from the Department of Anaesthesiology, University Medical Center Hamburg-Eppendorf.
This study was not supported by any funding.
Ethics approval and consent to participate
The assessment of NTS in simulation scenarios in undergraduate education was performed after approval by the local ethics committee in January 2017 (Ethikkommission der Ärztekammer Hamburg, Hamburg, Germany; Correspondence PV5563). Written informed consents were obtained from the study participants.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
AS-NTS. (DOCX 93 kb)