Skip to main content

Table 6 IRR results for all raters or paired raters (Raters 1 + 3)) using ICC, and weighted kappa. The two raters used for the weighted kappa analysis were chosen randomly because weighted kappa can only compare 2 raters at a time

From: A study of validity and usability evidence for non-technical skills assessment tools in simulated adult resuscitation scenarios

Scoring system and NTS category

ICC: Raters 1 + 3

ICC: all three raters

Weighted kappa (Raters 1 + 3 only)

ANTS

Global score

0.73

0.62

0.52

Task Management

0.26

N/Aa

0.10

Teamwork

0.65

0.62

0.64

Situation Awareness

0.79

0.60

0.54

Decision Making

0.64

0.67

0.45

Oxford NOTECHS

Global score

0.71

0.69

0.54

Leadership and Management

0.46

0.50

0.28

Teamwork and Cooperation

0.77

0.76

0.61

Problem Solving and Decision Making

0.81

0.67

0.67

Situation Awareness

0.34

0.51

0.22

OSCAR

Global score

0.80

0.68

0.40

Communication

0.53

0.25

0.25

Cooperation

0.84

0.69

0.54

Coordination

0.72

0.75

0.26

Leadership

0.75

0.67

0.29

Situation Awareness (monitoring)

0.87

0.67

0.73

Decision making

0.64

0.66

0.41

  1. Scores are highlighted (bold and underlined) where good or better agreement occurred and underlined where p = < 0.05
  2. aThe comparison between 3 raters was not possible for the Task Management domain of ANTS because the scores for rater 2 had zero variance