Skip to main content

Advertisement

Table 3 Reliability for Competency Items

From: Construct validation of judgement-based assessments of medical trainees’ competency in the workplace using a “Kanesian” approach to validation

Competency Item Variance Components Variances SEMa Percent of Total Variance of trainees’ scores Individual item Reliability Coefficient (R) NAAMARb
Overall Rating 0.084 0.016 29.4 0.676 10
Communication 0.104 0.017 36.4 0.741 7
Teamwork Skills 0.067 0.014 26.9 0.648 11
Professional Responsibility 0.043 0.013 18.1 0.557 19
Time Management Skills 0.071 0.018 24.6 0.620 13
Medical Records 0.054 0.015 21.5 0.578 15
Knowledge Base 0.045 0.015 18.2 0.527 17
Clinical Skills 0.051 0.014 17.9 0.522 18
Clinical Judgement 0.105 0.022 29.6 0.678 10
Awareness of Limitations 0.046 0.013 18.3 0.561 18
Professional Obligations 0.049 0.012 19.8 0.543 16
Competency Domain Construct 1 2.465   40.0 0.769 6
Competency Domain Construct 2 0.579   31.2 0.664 11
Competency Domain Construct 3 0.180   22.3 0.589 13
  1. a Standard Error of the Measurement
  2. bNAAMAR = Number (rounded to digit) of assessments for adequate minimum acceptable reliability level of R = 0.80 with the NAAMAR calculated form the formula: R (reliability coefficient) = {σ2 subjects /(σ2 subjects + σ2 error /n)}, where n = assessments needed per trainee to attain the desired reliability coefficient