- Research article
- Open Access
- Open Peer Review
Assessing the similarity of mental models of operating room team members and implications for patient safety: a prospective, replicated study
BMC Medical Educationvolume 16, Article number: 229 (2016)
Patient safety depends on effective teamwork. The similarity of team members’ mental models - or their shared understanding–regarding clinical tasks is likely to influence the effectiveness of teamwork. Mental models have not been measured in the complex, high-acuity environment of the operating room (OR), where professionals of different backgrounds must work together to achieve the best surgical outcome for each patient. Therefore, we aimed to explore the similarity of mental models of task sequence and of responsibility for task within multidisciplinary OR teams.
We developed a computer-based card sorting tool (Momento) to capture the information on mental models in 20 six-person surgical teams, each comprised of three subteams (anaesthesia, surgery, and nursing) for two simulated laparotomies. Team members sorted 20 cards depicting key tasks according to when in the procedure each task should be performed, and which subteam was primarily responsible for each task. Within each OR team and subteam, we conducted pairwise comparisons of scores to arrive at mean similarity scores for each task.
Mean similarity score for task sequence was 87 % (range 57–97 %). Mean score for responsibility for task was 70 % (range = 38–100 %), but for half of the tasks was only 51 % (range = 38–69 %). Participants believed their own subteam was primarily responsible for approximately half the tasks in each procedure.
We found differences in the mental models of some OR team members about responsibility for and order of certain tasks in an emergency laparotomy. Momento is a tool that could help elucidate and better align the mental models of OR team members about surgical procedures and thereby improve teamwork and outcomes for patients.
Effective teamwork is essential for patient safety [1–5]. Failures in teamwork and communication are common in the operating room (OR) and often lead directly to compromised patient care and reduced productivity [6–11]. The tasks of the surgical, anaesthetic, and nursing subteams are closely interlinked and inter-dependent [5, 12]. Members of the OR team should have a common understanding of the plan for patient management  and of the roles and responsibilities of each individual. However, the composition of OR teams changes frequently, members come from different professional backgrounds, and decisions may be needed under time pressure, sometimes with ambiguous or incomplete clinical information. Differences in understanding of the situation, the plan, and the key roles and responsibilities of individual team members may arise, and may impact on patient outcomes, particularly in crises, when time is severely limited [14, 15].
Humans function on the basis of their personal understanding of their situation at any time, which is likely to be unique and to represent reality to a varying degree. Apparently bizarre accidents can often be explained on the basis of discrepancies between this internal view of the world, and the facts that actually pertained at the time [9, 16]. This internal representation of reality has been named the person’s “mental model” . Within a team, each member will have his or her own mental model of the situation and the plan, and of when and by whom various tasks should be done . The extent to which these models overlap (like the common intersect of several circles on a Venn diagram) has been called the teams’ “shared mental model” [19, 20]. In practice, the degree of overlap or “sharing” may vary between different subsets of the team, and may change, dynamically, over time. There is likely to be a core set of information that must be shared by all key players if teamwork is to be effective on a regular basis , and for team members to be able to adapt to unexpected situations and predict each other’s actions and needs [22, 23]. Substantially shared mental models are presumed to be the cognitive basis of the smooth and effortless coordination observed in many expert teams working in high-intensity environments  such as the OR.
Research on the extent of sharing, or similarity, of mental models in healthcare teams is scarce. Most prior investigations have been conducted within the military domain [24–27] or in laboratory settings often involving students in pairs exposed to computer simulations [28, 29]. Studies in the OR have tended to focus on individual professional groups or “silos”, such as anaesthetists [30–32].
Therefore, we aimed to explore the degree of similarity between the mental models of members of multidisciplinary OR teams regarding the key tasks in an upcoming surgical procedure. Our specific questions were: 1) to what degree do OR team members share a mental model of task sequence (i.e., when the tasks should be done) prior to the procedure; and 2) to what degree do OR team members share a mental model of responsibility for task (i.e., who is primarily responsible for each task) prior to the procedure. To this end we began by developing a card sorting tool for quantification of the degree of similarity (or “sharedness”) of mental models concerning 20 key tasks related to two common clinical scenarios.
This study was conducted in the context of the Multidisciplinary Operating Room Simulation (MORSim) project, a larger body of research aimed at examining various aspects of teamwork in the OR (Australia and New Zealand Clinical Trials Registry ID 12612001088831).
We recruited 20 complete OR teams (comprising 120 healthcare professionals in total) for the MORSim project from general surgical ORs at two large teaching hospitals in the Auckland region. Each team included: a consultant and a junior surgeon (surgical subteam); a consultant anaesthetist or senior anaesthetic fellow and an anaesthetic technician (anaesthetic subteam); and two nurses (nursing subteam). In each case these were team members that worked together from time to time within their ORs.
We conducted the MORSim project at the University of Auckland’s Simulation Centre for Patient Safety in a simulated operating room. Each team participated in two simulated abdominal cases, presented in random order to control for time of day and order effects. Scenario 1 was a laparotomy for an abdominal stab wound and scenario 2 was a laparotomy for a perforated viscus. We developed a card sorting tool, named Momento, which can be easily customised for other clinical scenarios. Items in Momento for this work were developed from the clinical requirements of the cases presented to participants.
Our approach was informed by several studies [31, 33–35]. In addition, we observed ten relevant clinical cases, and identified a number of tasks for possible inclusion in Momento. We then selected a group of expert OR clinicians, including at least two representatives from each OR professional group (surgery, anaesthesia and nursing). We asked these experts to select tasks relevant to our two study scenarios according to the following criteria: the tasks should be clinically important, and occur routinely in all general surgical laparotomies, and they should have the potential to make a difference to patient care. This produced a list of 20 tasks for each scenario (Table 1), 18 generic and two scenario-specific. These tasks were captured on a set of electronic cards, for presentation in random order on a computer screen. We designed the screen layout with the help of human factors and IT specialists. When using Momento, participants are provided with a summary of an upcoming clinical scenario and are then presented with the relevant set of electronic cards. They are asked to arrange these in the order in which they should be done and to identify the subteam (nursing, surgery or anaesthesia) primarily responsible for ensuring that they are done (as opposed to simply doing the tasks) (Fig. 1). Participants can sort cards in parallel if they think the tasks need to be done at the same time. They can exclude a card, if they believe the associated task is not required for the successful management of the patient in the scenario.
We conducted a pilot of the final version of Momento using members of our research team supplemented by several other clinicians, and made adjustments to the software to optimise its ease of use and functionality.
Conduct of study day
Before starting each case scenario, participants received a demonstration on how Momento computer application works by observing the researcher working through a simple example of a card sort. They were then given a written brief about the upcoming case scenario and the time to read it. Participants were then asked to use Momento to sort the items related to that case. Within each team, participants performed the card sort simultaneously, but independently on individual computers, and without discussion. They were given as much time as they required to complete the card sort. Upon completion of the card sort, participants were shown into the simulated operating room and the scenario commenced.
For each participant, Momento produces a set of numerical ranks for order of the 20 tasks and a subteam category for responsibility for each task. We conducted all possible pairwise comparisons of individual ranks and categories within the team to calculate the degree of similarity of the mental models.
We calculated similarity scores for the sequence of each task between pairs of subteam members as the absolute difference between the ranks assigned to the task divided by the maximum possible score for any given task (19 is the maximum possible score for any given task, given a list of 20 tasks; a maximum score at this stage of the calculation implies the least possible agreement). We subtracted the result from one. Thus perfect agreement would produce a score of 1–(0/19) = 1, and the worst possible agreement would produce a score of 1–(19/19) = 0. These scores were then expressed as percentages.
We calculated similarity scores for responsibility for each task between pairs of subteam members as ‘1’ if the two participants agreed and ‘0’ if they disagreed and then calculated the mean similarity score for each subteam, expressed as a percentage.
We calculated similarity scores for the multidisciplinary OR team for each task as the mean of all possible pairwise combinations within the OR team. For each task and type of mental model we then calculated the mean similarity score for 20 participating OR teams.
Further details of these calculations can be found in Additional file 1.
One hundred-and-twenty participants (20 consultant and 20 junior surgeons; 20 consultant anaesthetists or senior anaesthetic fellows and 20 anaesthetic technicians; and 40 nurses) completed the card sort related to scenario 1. One participant, a nurse, was subsequently excluded because she misunderstood the instructions. The card sort for scenario 2 was completed by 119 participants because a junior surgeon had to leave the course early for personal reasons.
Within each OR team, there were more females (62.5 %) than males. Participants’ self-reported clinical experience ranged between 1 (6.7 %) and over 21 years (16 %), and experience in the OR ranged between less than 6 months (2.5 %) and over 21 years (9.2 %). Most participants reported having between 3 and 7 years of clinical (34.2 %) and OR experience (32.5 %).
Participants took an average of 11.34 min (range = 4.46–34.48 min) to complete the card sort for scenario 1, and an average of 13.13 min (range = 3.47–35.10) to complete the card sort for scenario 2.
For the whole team, the overall mean similarity score for both scenarios was 87 % (range 57–97 %, median 86 %) for task sequence. This score exceeded 80 % for all but two tasks in one of the scenarios (Table 2). The overall mean similarity score for both scenarios was 70 % (range 38–100 %, median 70.5 %) for task responsibility and more than half of the items (26 out of 40) scored less than 80 %.
For both scenarios, ‘making a surgical incision’ was the task with the highest mean similarity score both for task sequence and for responsibility (for which the score was 100 %). The lowest mean similarity score for sequence was given for when to inform the intensive care unit during a laparotomy for an abdominal stab wound (scenario 1) (57 %), followed by when to confirm estimated blood loss in both scenarios (72 and 80 %). The lowest score for responsibility was given for checking for optimal patient positioning (38 and 39 %, respectively) closely followed by estimating blood loss (39 % in both scenarios). All participants indicated that the anaesthesia subteam is primarily responsible for performing a rapid sequence induction in scenario 1, and administering anaesthesia induction drugs in scenario 2.
Mean similarity scores on task sequence were largely consistent across the three subteams (see Table 2). Within some subteams mean similarity scores on responsibilities for some of the tasks in the procedure were lower than in other subteams (see Table 3). For example, mean similarity scores were lower among surgical subteams than the anaesthesia and nursing subteams for who should be primarily responsible for ensuring patient warming devices are in place. Similarly, the mean similarity score was higher in anaesthesia subteams than the surgical subteams for who they thought should be primarily responsible for ensuring appropriate antibiotic prophylaxis.
Frequency histograms in Fig. 2 show the spread of ranks in the task sequence assigned by individual team members (20 within each of the six member categories) to individual tasks on which there was lowest (57 %–Fig. 2a) and highest average agreement (97 %–Fig. 2b) within OR teams. We also provided an example of a plot showing ranks assigned to a task with a mid-range agreement (80 %–Fig. 2c) for illustrative purposes. The plots demonstrate that the lower the OR team mean similarity score for the sequencing of a task in the procedure, the greater the spread of ranks assigned by team members to that task, with the lowest mean score (in the case of informing the intensive care unit in scenario 1) generating the greatest spread of ranks. By contrast, the highest OR team mean score (in the case of making a surgical incision in scenario 2) had the most unified rankings, regardless of team member. In the case of informing the intensive care unit (Fig. 2a), the majority of surgeons thought this task should be performed at the start of the procedure, while the majority of nurses and anaesthesia subteams believed it should be done sometime in the second half of the procedure.
Table 4 shows the mean percentage of times each subteam believed which of the three possible OR subteams was primarily responsible for a task. On average, the anaesthesia subteam believed they were responsible for ten out of 20 tasks in scenario 1, and nine out of 20 tasks in scenario 2. The surgical subteam believed their subteam was responsible for nine out of 20 tasks in both scenarios, while the nursing subteam believed they were responsible for half the tasks in each scenario. For 14 out of 20 tasks, the majority of each subteam agreed on the same subteam as being primarily responsible for a task. For three tasks (insert urinary catheter; initiate time out; and check drains are turned on), the majority of the subteams agreed that the primary responsibility should be split between the nursing and the surgical subteam. For two tasks (ensure patient warming devices are in place; and provide handover on intraoperative events to PACU staff) responsibility was split between the anaesthesia and the nursing subteam. Subteams produced variable responses for the tasks with lower mean similarity score on responsibilities. For example, all three subteams chose their own subteam as being primarily responsible for checking for optimal patient positioning on the OR table. While the majority of anaesthesia and surgical subteam members agreed that providing handover on intraoperative events to PACU (post-anaesthesia care unit) staff was primarily the responsibility of the anaesthesia subteam, most nurses thought they should be primarily responsible for this task. For confirming estimated blood loss, one of the tasks with lowest similarity scores for both task sequence and responsibility for task, there was also a split within the subteams, with the nurses splitting primary responsibility between all three subteams.
We found that there was poor agreement between OR team members on responsibility for task for half the tasks in each procedure. This has potentially important and concerning implications for safe and efficient team work. OR team members had largely similar understandings of when tasks should be done in an upcoming procedure for all but two tasks, which was more reassuring. Members of the three OR subteams believed their own subteam was primarily responsible for around half the tasks in each procedure.
The relationship between the extent of similarity of mental models within a team and the team’s performance has not been well defined . Furthermore, it is still not clear what constitutes an optimal degree of shared understanding either conceptually , or in relation to our similarity scores. Important decisions in an organisational setting presumably warrant a very strong degree of agreement among the decision-makers , but there is no empirical evidence to guide the quantification of this in multidisciplinary healthcare teams. It has been suggested that agreement over who is responsible for what may be more important for team performance than agreement over the sequence in which tasks should be done . Redundancy in perceived responsibility for a task may be seen as making that task less likely to be forgotten, but may also result in the (possibly unjustified) assumption that it can be left to others. Furthermore, it would seem to be less efficient to have more than one person taking responsibility for a task, especially in a crisis where time and resources are precious.
Our new Momento tool could be used to identify the potential differences between members of OR teams in their understanding of the key tasks and related responsibilities regarding the upcoming surgical procedure. Making team members aware of the extent of the discrepancies in individual mental models prior to embarking on a case gives them an opportunity to regroup and address the gaps in shared understanding, to make sure all team members are “on the same page” as to who should be responsible for which crucial tasks and when. Providing time for team members to agree, perhaps through a briefing session or in relation to the WHO Surgical Safety Checklist ahead of a procedure, on the order of crucial tasks and on who should be responsible for each task, may help clarify potential ambiguities and better align mental models [39, 40]. The level of disagreement seen in this study for some tasks reinforces an increasing body of evidence supporting a pre-procedure briefing to align understandings and circumvent intra- and postoperative complications and reduce wasted time [41–44].
A secondary outcome of our study is that we have produced a measurement tool for evaluating the similarity of mental models of clinical teams in the OR. This could be easily modified for use in different clinical scenarios. A standard method for evaluating the similarity of mental models that would be applicable in wider research settings has not been reported in literature to date. The choice of a technique to elicit and represent a mental model is considered a key issue in mental model research , and the slow progress of empirical work on the sharing of mental models has largely been blamed on the lack of measurement tools [35, 46]. A multi-database search of academic literature, online search engines, and a manual search of associated bibliographies, produced only three studies [31, 34, 35] published in peer-reviewed journals that have combined the ease of use of sorting tasks into a predefined chronological sequence and the ability to simultaneously capture declarative or descriptive knowledge (i.e., the knowledge of what constitutes a task) and procedural knowledge (i.e., knowledge of how to perform the task) . In all three studies, ad hoc teams completed the sorting exercise specifically designed for that study.
A strength of this study lies in the inclusion of different OR professional groups within complete OR teams who have worked together in the past. Mental models have not previously been assessed directly in the context of established, multidisciplinary healthcare teams. The study by Burtscher et al.  remains so far the only previous study to quantify mental models in the OR context. These authors, however, only focused on mental models of two-person anaesthesia teams. Larger teams, and especially multidisciplinary ones, may have different dynamics than smaller ones. They engage in more complex processes than smaller teams and they have a greater diversity of view points and expertise due to multiple team members [48–50] Thus, the focus on larger multidisciplinary teams in the current project adds to the previous studies that have focused on mental models of small teams [34, 35], or of only part of the team .
Another strength lies in the fact that all teams scored the identical scenario–this was made possible by the use of simulated cases that participants were about to undertake, rather than clinical cases. The fact that participants were preparing to actually manage the cases (albeit in simulation) is likely to have increased their engagement in the process of reading the briefs and responding to the card sorting task, in comparison with the alternative possibility of just providing teams with hypothetical scenarios.
Our card sorting tool, Momento, with modification for particular procedures, could now be used to identify dissimilarities of mental models for various procedures in clinical practice. Given its ease-of-use and customisability, our tool also has the potential to be used in domains reliant on teamwork other than healthcare.
One limitation of our study is that the lack of previous validation of our card sorting tool. While validity was built in to the tool through a rigorous approach to the selection of items for inclusion, further evidence of validity would be desirable.
Other potential limitations of this study include the possibility that there were inaccuracies in card sorting by participants. Also, participants did not have the option of choosing more than one OR subteam as responsible for a task, as the exercise used a forced choice design. The split within OR teams on who should be primarily responsible for certain tasks found in this study might therefore reflect that there should be joint responsibility for those tasks. Future versions of the card sorting exercise should be upgraded to allow for the selection of multiple subteams for those tasks for which there is likely to be joint responsibility.
Our study was limited to cases requiring laparotomy. Future research should extend this work to other types of surgery and surgical specialties.
We measured the similarity of mental models at the beginning of the two cases. Mental models of surgical procedures are likely to be dynamic and the degree of similarity of mental models may well change as cases progress and team members communicate with each other.
In this study we have not established the accuracy of the participants’ mental models. This can be construed as the degree to which these converge with those of guidelines or subject experts, or more generally the degree to which they are grounded in reality. For many tasks it may be more important that team members agree on what should be done, by whom and when, than that this agreement necessarily reflects received wisdom. In some instances however, accuracy in this latter sense probably does matter – and our results may support the idea that agreement tends to be higher in such circumstances. For example, our mean similarity scores were high for who should make the surgical incision and who should administer the anaesthetic drugs.
The extent to which similarity of mental models influences subsequent team performance and patient outcome was beyond the scope of this study and is an area for further research.
We found differences in the mental models of OR team members about who was responsible for certain tasks, and some variation regarding the order of tasks in an emergency laparotomy. This has implications for effective team function and patient safety. Momento is a tool that could help elucidate and better align the mental models of OR team members about surgical procedures and thereby improve teamwork and outcomes for patients.
Helmreich RL, Schaefer HG. Team performance in the operating room. In: Bogner MS, editor. Human error in medicine. Hillsdale: Erlbaum; 1995. p. 225–53.
Manser T, Howard SK, Gaba DM. Identifying characteristics of effective teamwork in complex medical work environments: Adaptive crew coordination in anaesthesia. In: Flin R, Mitchell L, editors. Safer surgery: Analysing behaviour in the operating theatre. Aldershot: Ashgate; 2009. p. 223–39.
Rosen MA, Salas E, Wilson KA, King HB, Salisbury M, Augenstein JS, et al. Measuring team performance in simulation-based training: Adopting best practices for healthcare. Simul Healthc. 2008;3(1):33–41.
Sexton JB, Thomas EJ, Helmreich RL. Error, stress, and teamwork in medicine and aviation: Cross sectional surveys. BMJ. 2000;320(7237):745–9.
Lingard L, Espin S, Whyte S, Regehr G, Baker GR, Reznick R, et al. Communication failures in the operating room: An observational classification of recurrent types and effects. BMJ Qual Saf. 2004;13(5):330–4.
Bognor M. Human error in medicine. 1st ed. New Jersey: Lawrence Erlbaum Association Inc; 1994.
Helmreich R. Threat and error in aviation and medicine: Similar and different. In: Special medical seminar, lessons for health care: Applied human factors research. Australian Council of Safety and Quality in Health Care & NSW Ministerial Council for Quality in Health Care. 2000.
Reader TW, Flin R, Cuthbertson BH. Communication skills and error in the intensive care unit. Curr Opin Crit Care. 2007;13(6):732–6.
Reason JT. Human error. Cambridge. New York: Cambridge University Press; 1990.
Greenberg CC, Regenbogen SE, Studdert DM, et al. Patterns of communication breakdowns resulting in injury to surgical patients. J Am Coll Surg. 2007;204:533–40.
Mazzocco K, Petitti DB, Fong KT, et al. Surgical team behaviors and patient outcomes. Am J Surg. 2009;197:678–85.
Gaba DM, Howard SK, Fish KJ, Smith BA, Sowb YA. Simulation-based training in anesthesia crisis resource management (ACRM): A decade of experience. Simul Gaming. 2001;32(2):175–93.
Burtscher MJ, Manser T. Team mental models and their potential to improve teamwork and safety: A review and implications for future research in healthcare. Saf Sci. 2012;50(5):1344–54.
Leach LS, Myrtle RC, Weaver FA. Surgical teams: Role perspectives and role dynamics in the operating room. Health Serv Manag Res. 2011;24(2):81–90.
Weller J, Janssen AL, Merry A, et al. Interdisciplinary team interactions: a qualitative study of perceptions of team function in simulated anaesthesia crises. Med Educ. 2008;42:382–8.
Perrow C. Normal accidents: Living with high-risk technologies. Princeton: Princeton University Press; 1999.
Johnson-Laird PN. A computational analysis of consciousness. Cogn Brain Theor. 1983;6:499–508.
Cannon-Bowers JA, Salas E. Cognitive psychology and team training: Shared mental models in complex systems. Paper presented at the Annual meeting of the Society for Industrial and Organizational Psychology, Miami, FL. 1990. August.
Cannon-Bowers JA, Salas E, Converse SA. Shared mental models in expert team decision making. In: Castellan NJ, editor. Individual and group decision making: Current issues. Hillsdale: Lawrence Erlbaum Associates Inc.; 1993. p. 221–46.
Klimoski R, Mohammed S. Team mental model: Construct or metaphor? J Manag. 1994;20(2):403–37.
Salas E, Sims DE, Burke CS. Is there a “big five” in teamwork? Small Group Res. 2005;36(5):555–99.
Orasanu J. Shared mental models and crew decision making (CLS report no. 46). Princeton: Princeton University Cognitive Science Laboratory; 1990.
Serfaty DS, Entin EE, Volpe C. Adaptation to stress in team decision making and coordination. Proceedings of the 37th Annual Meeting of the Human Factors Society. 1993. p. 1228–32.
Entin EE, Serfaty DS. Adaptive team coordination. Hum Factors. 1999;41(2):312–3.
Salas E, Fowlkes JE, Stout RJ, Milanovich DM, Prince C. Does CRM training improve teamwork skills in the cockpit? Two evaluation studies. Hum Factors. 1999;41:326.
Serfaty DS, Entin EE, Deckert JC. Implicit coordination in command teams. In: Levis AH, Levis IS, editors. Science of command and control: Part III coping with change. AIP Information Systems. 1994. p. 87–94.
Swain K, Mills V. Implicit communication in novice and expert teams. DSTO Systems Sciences Laboratory: Australia; 2003. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.535.8330&rep=rep1&type=pdf. Accessed 24 Aug 2013.
Shah J, Breazeal C. An empirical analysis of team coordination behaviors and action planning with application to Human–Robot teaming. Hum Factors. 2010;52(2):234–45.
Mathieu JE, Heffner TS, Goodwin GF, Salas E, Cannon-Bowers JA. The influence of shared mental models on team process and performance. J Appl Psychol. 2000;85(2):273.
Xiao Y, Hunter WA, Mackenzie CF, Jefferies NJ, Horst RL. Task complexity in emergency medical care and its implications for team coordination. LOTAS group. Level One trauma anesthesia simulation. Hum Factors. 1996;38(4):636–45.
Burtscher MJ, Kolbe M, Wacker J, Manser T. Interactions of team mental models and monitoring behaviors predict team performance in simulated anesthesia inductions. J Exp Psychol Appl. 2011;17(3):257–69.
Manser T, Harrison TK, Gaba DM, Howard SK. Coordination patterns related to high clinical performance in a simulated anesthetic crisis. Anesth Analg. 2009;108(5):1606–15.
Novak JD. Concept maps and Venn diagrams: Two metacognitive tools to facilitate meaningful learning. Instr Sci. 1990;19:29–52.
Ellis APJ. System breakdown: The role of mental models and transactive memory in the relationship between acute stress and team performance. Acad Manag J. 2006;49(3):576–89.
Marks MA, Zaccaro SJ, Mathieu JE. Performance implications of leader briefings and team-interaction training for team adaptation to novel environments. J Appl Psychol. 2000;85(6):971–86.
Lim B, Klein KJ. Team mental models and team performance: A field study of the effects of team mental model similarity and accuracy. J Organ Behav. 2006;27(4):403–18.
Boos M. Optimal sharedness of mental models for effective group performance. CoDesign. 2007;3(1):21–8.
LeBreton JM, Senter JL. Answers to 20 questions about interrater reliability and interrater agreement. Organ Res Methods. 2008;11(4):815–52.
Paige JT, Aaron DL, Yang T, Howell DS, Hilton CW, Cohn Jr I, et al. Implementation of a preoperative briefing protocol improves accuracy of teamwork assessment in the operating room. Am Surg. 2008;74(9):817–23.
Paige JT, Aaron DL, Yang T, Howell DS, Chauvin S. Improved operating room teamwork via SAFETY prep: A rural community hospital’s experience. World J Surg. 2009;33(6):1181–7.
Lingard L, Regehr G, Orser B, Reznick R, Baker GR, Doran D, et al. Evaluation of a preoperative checklist and team briefing among surgeons, nurses, and anesthesiologists to reduce failures in communication. Arch Surg. 2008;143(1):12–7.
Haynes AB, Weiser TG, Berry WR, Lipsitz SR, Breizat AS, Dellinger EP, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med. 2009;360(5):491–9.
Fudickar A, Hörle K, Wiltfang J, Bein B. The effect of the WHO surgical safety checklist on complication rate and communication. Dtsch Arztebl Int. 2012;109(42):695–701.
Weller J, Boyd M. Making a difference through improving teamwork in the operating room: A systematic review of the evidence on what works. Curr Anesthesiol Rep. 2014;4:77–83.
Langan-Fox J, Code S, Langfield-Smith K. Team mental models: Techniques, methods, and analytic approaches. Hum Factors. 2000;42(2):242–71.
Mohammed S, Ferzandi L, Hamilton K. Metaphor no more: A 15-year review of the team mental model construct. J Manag. 2010;36(4):876–910.
Cooke NJ, Kiekel PA, Salas E, Stout R, Bowers C, Cannon-Bowers JA. Measuring team knowledge. Group Dyn: Theor Res. 2003;7(3):179–99.
Shuffler ML, Jiménez-Rodríguez M, Kramer WS. The science of multiteam systems. Small Gr Res. 2015;46:659–99.
Cummings JN, Kiesler S, Bosagh ZR, et al. Group heterogeneity increases the risks of large group size: a longitudinal study of productivity in research groups. Psychol Sci. 2013;24:880–90.
Milliken F, Martins L. Searching for common threads: understanding the multiple effects of diversity in organizational groups. Acad Manag Rev. 1996;21:402–33.
We would like to thank the following members of the MORSim research team for their invaluable input into the design of the content and layout of the Momento card sorting tool: Ian Civil, Michael Crossan, Alexander Garden, Abbey Gundesen, Nishanthi Gurusinghe, Wendy Guthrie, Sharee Johnston, Brenda Knowles, Tracey Lee, Andrew MacCormick, Simon Mitchell, Wai Leap Ng, and Jane Torrie. We would also like to thank the clinicians who attended the MORSim pilot study day. We are very grateful to the staff working at the Auckland City Hospital level eight operating rooms for their openness to the presence of IN in their ORs during the in-theatre observation phase of the study. Special thanks are due to Wendy Guthrie and Sharee Johnston for making the in-theatre observations possible.
This work was supported by the Health Workforce New Zealand Innovations Grant.
Availability of data and materials
Raw data for this study can be found in Additional file 2.
IN was involved in conception and design, acquisition of data, analysis and interpretation of data, drafting and revision of the manuscript. JW participated in the conception and design, interpretation of data, and drafting and revision of the manuscript. CW was involved in the design of the study, analysis and interpretation of data, and revision of the manuscript. DC was involved in conception and design, acquisition of data, analysis and interpretation of data, and revision of the manuscript. CF participated in the design of the study and statistical analysis and interpretation of data, as well as revision of the manuscript. MB was involved in conception and design, analysis and interpretation of data, and revision of the manuscript. AM participated I the conception and design, analysis and interpretation of data, and drafting and revision of the manuscript. All authors read and approved the final manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Approval for the study was obtained from the Central Regional Ethics Committee (CEN/12/03/002). Written informed consent for participation in the study was obtained from participants.
Calculating similarity of mental models. An example considering one team and four tasks is provided to demonstrate in more detail how we calculated the similarity scores for mental models of responsibility for task and similarity of mental models of task sequence. (DOCX 20 kb)
Raw data. Ranks assigned by individual team members (A = anaesthetist; AT = anaesthetic technician; Na = nurse 1; Nb = nurse 2; Sa = consultant surgeon; Sb = junior surgeon) to each of the 20 tasks and a subteam category (A = anaesthesia subteam; N = nursing subteam; or S = surgical subteam) they assigned for responsibility for each task. (DOCX 239 kb)