Assessing professional competence in optometry – a review of the development and validity of the written component of the competency in optometry examination (COE)

Table 1 Descriptive statistics for all multiple choice question (MCQ) exam administrations

Administration Round	1	2	3	4	5	6	7	8	9	10	11	12
Examination date	April 2014	September 2014	April 2015	September 2015	April 2016	September 2016	April 2017	September 2017	March 2018	August 2018	March 2019	August 2019
No. of scored questions (total no. of questions)	120 (132)	120 (132)	120 (132)	120 (132)	120 (132)	120 (132)	120 (132)	120 (132)	120 (144)	119^c (144)	119^c (144)	120 (144)
No. of linking questions	23	23	20	19	19	21	26	18	10	19	12	16
No. of candidates	33	25	18	24	13	20	22	20	18	19	22	23
No. of repeat candidates	14	15	5	10	4	4	7	5	4	8	4	8
Mean candidate score (%)	51.5	65.6	66.9	64.5	58.9	66.9	73.3	68.0	59.4	62.7	65.2	62.2
Max total score (%)	74	81	83	86	70	83	82	89	78	81	79	78
Min total score (%)	34	49	44	33	43	37	61	38	36	30	49	32
Mean item facility (%)	– ^d	– ^d	66.8 ± 21.6	64.5 ± 19.7	58.9 ± 23.7	66.9 ± 18.1	73.0 ± 21.4	68.0 ± 20.2	59.3 ± 19.2	62.7 ± 19.1	65.2 ± 21.5	62.2 ± 23.0
Mean discrimination index^a	0.2^e	0.2^e	0.23 ± 0.24	0.26 ± 0.20	0.18 ± 0.33	0.26 ± 0.24	0.17 ± 0.24	0.33 ± 0.20	0.25 ± 0.28	0.29 ± 0.24	0.19 ± 0.23	0.29 ± 0.24
Reliability index^b	0.80	0.79	0.85	0.88	0.78	0.89	0.71	0.93	0.89	0.91	0.79	0.91
Cutscore (%)	50.8	65.8	67.5	61.7	64.6	69.2	70.0	66.7	63.3	66.4	63.9	60.8
Pass rate (%)	54.5	60.0	61.1	58.3	38.5	60.0	63.6	65.0	55.6	47.4	54.9	65.2

Descriptive statistics for every MCQ administration for all candidates between 2014 and 2019. (^a) Mean value for the discrimination index calculated for each item as part of the classical analysis undertaken by EPEC. (^b) Reliability index alpha (KR-20) calculated for the exam as part of the classical analysis undertaken by EPEC. A reliability index greater than 0.80 is considered high paper reliability. (^c) One scored item excluded from analysis following content expert review after being flagged by the preliminary Rasch analysis. (^d) Detailed individual item reports not requested from EPEC consultancy firm prior to April 2015 administration. (^e) Detailed individual item reports not available from EPEC consultancy firm prior to April 2015 administration. The values reported here represent the mean discrimination index provided by EPEC from the classical analysis, rather than an analysis of the individual item point biserial values, therefore no standard deviations showing the spread of the data are available.

ISSN: 1472-6920