Construct validation of judgement-based assessments of medical trainees’ competency in the workplace using a “Kanesian” approach to validation

Table 3 Reliability for Competency Items

Competency Item	Variance Components	Variances SEM^a	Percent of Total Variance of trainees’ scores	Individual item Reliability Coefficient (R)	NAAMAR^b
Overall Rating	0.084	0.016	29.4	0.676	10
Communication	0.104	0.017	36.4	0.741	7
Teamwork Skills	0.067	0.014	26.9	0.648	11
Professional Responsibility	0.043	0.013	18.1	0.557	19
Time Management Skills	0.071	0.018	24.6	0.620	13
Medical Records	0.054	0.015	21.5	0.578	15
Knowledge Base	0.045	0.015	18.2	0.527	17
Clinical Skills	0.051	0.014	17.9	0.522	18
Clinical Judgement	0.105	0.022	29.6	0.678	10
Awareness of Limitations	0.046	0.013	18.3	0.561	18
Professional Obligations	0.049	0.012	19.8	0.543	16
Competency Domain Construct 1	2.465		40.0	0.769	6
Competency Domain Construct 2	0.579		31.2	0.664	11
Competency Domain Construct 3	0.180		22.3	0.589	13

^a Standard Error of the Measurement
^bNAAMAR = Number (rounded to digit) of assessments for adequate minimum acceptable reliability level of R = 0.80 with the NAAMAR calculated form the formula: R (reliability coefficient) = {σ² _subjects /(σ² _{subjects +} σ² _error /n)}, where n = assessments needed per trainee to attain the desired reliability coefficient

ISSN: 1472-6920