Insightful Practice: a robust measure of medical students’ professional response to feedback on their performance

BMC Medical Education

Table 3 Inter-rater Reliability of Assessment of Insightful Practice (AIP) Questions

3a - GROUP 1 (Calibrated Assessors)
	AIP questions 1–3 (engagement, insight and action) 1-7 scale	AIP question 4 (global assessment) 1-7 scale	AIP question 5 (Dichotomous assessment on suitability for progression recommendation)
Number of Raters	Inter-Rater Reliability (G)^b	Inter-Rater Reliability (ICC)^a (G)^b (95 % confidence interval)^c	Inter-Rater Reliability (ICC)^a (G)^b (95 % confidence interval)^c
1	0.76	0.73 (−)	0.75 (−)
2	0.87	0.84 (0.57-0.95)	0.85 (0.59-0.95)
3	0.91	0.89 (0.73-0.96)	0.9 (0.76-0.96)
3b- GROUP 1 (Assessors not calibrated)
GROUP 2 (Assessors NOT Calibrated)	AIP questions 1–3 (engagement, insight and action) 1-7 scale	AIP question 4 (global assessment) 1-7 scale	AIP question 5 (Dichotomous assessment on suitability for progression recommendation)
(n) Raters	Inter-Rater Reliability (G)^b	Inter-Rater Reliability (ICC)^a (G)^b	Inter-Rater Reliability (ICC)^a (G)^b
1	0.33	0.18	0.16
2	0.5	0.31	0.28
3	0.6	0.4	0.37

^aIntraclass Correlation Coefficients (ICCs) are G-coefficients when you have a one facet design (rater)
^bInter-rater reliability is the extent to which one rater’s assessments (or when based on multiple raters, the average of raters’ assessments) are predictive of another rater’s assessments
^c95 % confidence intervals for reliabilities (ICCs) were calculated using Fisher’s ZR transformation which is dependent on raters (5) with a denominator value of (n-1), so cannot be calculated when only one rater. (Streiner and Norman, [9])

ISSN: 1472-6920