Skip to main content

Advertisement

Springer Nature is making SARS-CoV-2 and COVID-19 research free. View research | View latest news | Sign up for updates

Table 3 Inter-rater Reliability of Assessment of Insightful Practice (AIP) Questions

From: Insightful Practice: a robust measure of medical students’ professional response to feedback on their performance

3a - GROUP 1 (Calibrated Assessors)
  AIP questions 1–3 (engagement, insight and action) 1-7 scale AIP question 4 (global assessment) 1-7 scale AIP question 5 (Dichotomous assessment on suitability for progression recommendation)
Number of Raters Inter-Rater Reliability (G)b Inter-Rater Reliability (ICC)a (G)b (95 % confidence interval)c Inter-Rater Reliability (ICC)a (G)b (95 % confidence interval)c
1 0.76 0.73 (−) 0.75 (−)
2 0.87 0.84 (0.57-0.95) 0.85 (0.59-0.95)
3 0.91 0.89 (0.73-0.96) 0.9 (0.76-0.96)
3b- GROUP 1 (Assessors not calibrated)
GROUP 2 (Assessors NOT Calibrated) AIP questions 1–3 (engagement, insight and action) 1-7 scale AIP question 4 (global assessment) 1-7 scale AIP question 5 (Dichotomous assessment on suitability for progression recommendation)
(n) Raters Inter-Rater Reliability (G)b Inter-Rater Reliability (ICC)a (G)b Inter-Rater Reliability (ICC)a (G)b
1 0.33 0.18 0.16
2 0.5 0.31 0.28
3 0.6 0.4 0.37
  1. aIntraclass Correlation Coefficients (ICCs) are G-coefficients when you have a one facet design (rater)
  2. bInter-rater reliability is the extent to which one rater’s assessments (or when based on multiple raters, the average of raters’ assessments) are predictive of another rater’s assessments
  3. c95 % confidence intervals for reliabilities (ICCs) were calculated using Fisher’s ZR transformation which is dependent on raters (5) with a denominator value of (n-1), so cannot be calculated when only one rater. (Streiner and Norman, [9])