Assessing professional competence in optometry – a review of the development and validity of the written component of the competency in optometry examination (COE)

Table 2 Descriptive statistics for all short answer question (SAQ) exam administrations

Administration Round	1	2	3	4	5	6	7	8	9	10	11	12
Examination date	April 2014	September 2014	April 2015	September 2015	April 2016	September 2016	April 2017	September 2017	March 2018	August 2018	March 2019	August 2019
No. of scored questions^a	18	18	18	18	18	18	18	18	18	18	18	18
No. of candidates	32	21	18	23	13	16	24	19	19	18	21	24
No. of repeat candidates	12	11	5	9	4	1	9	4	5	7	3	9
Mean candidate score (%)	60.2	52.4	46.4	46.5	56.8	38.1	57.6	45.5	51.3	49.9	50.7	59.5
Max total score (%)	84.0	70.3	58.0	60.5	67.8	57.1	67.8	63.6	68.9	70.8	68.3	71.9
Min total score (%)	24.4	26.9	22.0	8.0	42.8	5.0	42.5	18.6	27.8	30.0	31.9	31.4
Mean person ability (logits)^b	– ^e	– ^e	– ^e	– ^e	– ^e	– ^e	– ^e	0.154	−0.068	−0.122	0.233	0.102
Item-trait interaction (χ² probability)^c	– ^e	– ^e	– ^e	– ^e	– ^e	– ^e	– ^e	0.93	0.42	0.39	0.76	0.42
Reliability (separation index)^d	– ^e	– ^e	– ^e	– ^e	– ^e	– ^e	– ^e	0.93	0.89	0.84	0.83	0.82
Cutscore (%)	50^f	50^f	50^f	50^f	50^f	50^f	50^f	46.1	49.3	49.2	48.2	57.6
Pass rate (%)	84.4	66.7	61.1	47.8	84.6	25.0	83.3	47.4	63.2	50.0	57.1	66.7

Descriptive statistics for every SAQ administration for all candidates between 2014 and 2019. (^a) Each question consists of multiple sections, adding to a total of 10 marks per question (scored with half marks) and 180 marks per exam. Two questions were used for linking exams from September 2017 onwards. (^b) Mean person ability, in logits, from the Rasch analysis undertaken by EPEC consultancy on behalf of the Optometry Council of Australia and New Zealand (OCANZ) for the Competency in Optometry Exam (COE). The low mean person abilities indicate that the questions are well targeted to the candidates. (^c) The item-trait interaction gives a measure of whether the data fits the Rasch model for discreet groups (class intervals) along the scale. χ² values above 0.05 indicate the data do not significantly deviate from that expected from the model. (^d) Separation index values from the Rasch analysis undertaken by EPEC consultancy on behalf of OCANZ for the COE. These reliability values being greater than 0.8 show good fit of the Rasch model to the scores, indicating the SAQ exams have good power and reliability. (^e) Rasch analysis of the SAQ exam not undertaken prior to the September 2017 running of the exam, therefore no mean person ability or separation index values available to report. (^f) Rasch analysis of the SAQ exam not used to calculate a cutscore until the September 2017 running of the exam. Prior to September 2017 the cutscore was set as 50%.

ISSN: 1472-6920