Comparison of the sensitivity of the UKCAT and A Levels to sociodemographic characteristics: a national study

Data preparation- A level and UKCAT scores

For this analysis data relating to medical and dental applicants to the UKCAT consortia of universities in the 2009 round of applications (for 2010 entry) were used. Of the 23,719 individuals who sat the UKCAT in 2009 only 8,180 had complete data on the six sociodemographic predictor variables (ethnicity, age, school type attended, socioeconomic background, age at which English first spoken and sex). It was only information from these candidates with complete data that were included in the analysis. This approach, using ‘listwise deletion’, was taken so that the final sample investigated remained the same across all analyses.

In terms of managing the educational attainment data; A level grades in general studies and critical thinking were excluded, as were duplicated observations where the subject, grade and candidate unique identifier were identical (535 duplicate exam grades were deleted in the latter case). Examination outcome entries where the subject was the same but the grade differed for a candidate were assumed to be resits. In such cases the lowest grade was retained. This assumption was made for several reasons:- firstly, the dates of sitting of the examinations were unavailable; secondly, the first sitting was assumed to reflect a candidate’s academic potential more accurately than subsequent sittings; and thirdly, medical and dental schools often only accept grades at first sitting as meeting entry requirements. Consequently 810 presumed ‘resit’ exam grades were deleted. The UCAS tariff scores for a candidate’s best three A level grades were summed (that is, A* = 140, A = 120, B = 100, C = 80, D = 60 and E = 40 points). Thus the maximum summed tariff that a single candidate could obtain was 420 points (i.e. A*A*A* grades). Standardised z scores for both ‘best of three’ summed A level tariff and UKCAT total score were also derived (i.e. mean of zero and a standard deviation [SD] of one). This standardisation was intended to permit a certain amount of comparison between UKCAT and A level tariff scores. The distribution of standardised A level tariffs and UKCAT scores were examined graphically using histograms and quantile (Q-Q) plots to assess for degree of normality and to allow selection of appropriate estimation procedures. A Q-Q plot produces a graph of quantiles of the variable against quantiles of the normal distribution, allowing the visual identification of marked departures of a distribution from normality. This approach is recommended over simple reliance on significance tests for normality, such as the Kolmogorov-Smirnov test, which may be overly sensitive in certain circumstances [11].

Data preparation- sociodemographic data

The dichotomisation of sociodemographic variables was guided by previous research on widening participation in medical and dental education [12] and informed by an initial univariate exploration of the dataset. For example, previous research has reported that students educated at state grammar schools (that usually include some element of criterion-based selection for admission, in contrast to other types of state school) do not have poorer performance on the UKCAT, at A level, or in university compared to those receiving an independent (private) schooling [10]. Thus, for the purpose of dichotomising candidates into those from WP categories it seemed reasonable to classify those from state grammar schools in the same group as those from an independent school background. This assumption was supported by an exploratory univariate analysis that did not uncover evidence of disadvantage for grammar school students compared to independently schooled individuals when applying to medical or dental school. Likewise, previous research demonstrated a disadvantage when applying for medical school for those reporting their ethnicity as anything other than ‘White’ [5]. Consequently ‘non-White’ was the WP category created for ethnicity. Some descriptive analysis was conducted using more broadly defined ethnic groups (‘White’, ‘Black’, ‘Chinese’, ‘Asian’, ‘Mixed’ and ‘Other’). However, the numbers in each of these latter ethnic categories was generally too small to adequately power more detailed modelling in relation to these groups. Regarding socioeconomic background, the UKCAT database records socioeconomic status using a simplified version of the socioeconomic classification system utilised by the National Office for Statistics, based on the National Statistics Socio-Economic Classification (NS-SEC) [13]. As in previous research on WP in medical and dental education we classified those who reported a NS-SEC rating of four or greater as being from a non-professional socioeconomic background. Thus this category was equivalent to the classifications of ‘lower supervisory/technical’ and ‘semi-routine/routine’. It should be noted that this method of WP group definition differs from the approach to categorising socio-economic status taken by James et al. [10]. In this latter study the authors used the highest (most advantaged) category of the simplified NS-SEC system, ‘managerial/professional’ as the socio-economic predictor variable. In contrast the present study used the two lowest parental occupation categories to represent relative socioeconomic disadvantage as a predictor of A level and UKCAT performance. We classified those learning English after the age of two years as having English as a second language (EASL).

Data analysis

We used Stata v12 MP for data analysis. The distribution of the "best of three" A level tariff scores appeared to be represented by a roughly normal distribution that was right censored at 420 points as only three grades had been retained (i.e. 3 A*s was the maximum possible attainment- see Figure 1). The Q-Q plot highlighted the censored nature of the tariff score and that those less than two standard deviations away from the mean departed from normality (Figure 2). It was thus considered appropriate to model the data using a Tobit regression that could accommodate the right-censored nature of the tariff score [14] with bootstrapped standard errors that would allow for the moderate departure from normality. On inspection, total UKCAT test scores were normally distributed (Figure 3) although a Q-Q plot highlighted some departure from normality at extreme low scores (Figure 4). However, this slight departure from normality did not appear to justify the use of non-parametric methods of deriving standard errors. Thus linear regression was employed to estimate the influences on UKCAT score. The association between WP group status and UKCAT subtest scores were compared using linear regression. The UKCAT subtest scores were also approximately normally distributed and, given the relatively large number of observations, linear regression was also utilised in their analysis. Thus Tobit and linear regressions were utilised, as appropriate, for both univariable and multivariable models. Scores were compared between the UKCAT and A level tariffs using a Spearman rank correlation (to allow for any non-normality at lower values).

Following univariate regression analyses, multivariable regression analyses were performed. Two sets of regression analyses were conducted using A level tariffs then UKCAT scores as the outcomes of interest, respectively. In both cases the WP sociodemographic categories served as the predictor variables. Only six predictor variables were used in the multiple regression models and the mean variance inflation factor for a model including all of these was relatively small at 1.09. This latter value implies that the standard error for the estimated coefficients would be, on average, increased by approximately 9% due to multicollinearity of the predictor variables. Consequently, rather than taking a step-wise approach to model building, all predictors were included in the multivariable regression models. Where A level summed tariff score was used as the dependent variable the model was estimated using a Tobit regression using bootstrapped derived standard errors. Thus two differing regression methods (Tobit versus linear regression) were used to produce regression coefficients, intercepts and associated 95% confidence intervals (via bootstrapping in the case of the Tobit regression). As the dependent variables used in the multivariable models were z scores the coefficients were produced in the metric of standard deviation units. This allowed, to some extent, an appreciation of the relative magnitude of ability of each sociodemographic category to predict A level achievement and UKCAT score. Nevertheless, it should be noted that the regression coefficients derived from a Tobit regression differ slightly in conceptual terms from those derived from a linear regression; Tobit coefficients are estimated taking into account the probability that a predictor is actually above the observable limit. It should also be noted that where standardised confidence intervals do not overlap there may be a true difference in the comparative sensitivity of the outcome metric (i.e. A level tariff vs UKCAT score) to the sociodemographic variable. Conversely, where there is a modest degree of overlap in confidence intervals one cannot assume that no difference exists at the 95% confidence level (though such a difference would not exist at the 99% confidence level). Thus, some additional caution must be exercised when attempting to make inferences by comparing the confidence intervals for the coefficients from these two different regression methods.

Ethical approval

At UKCAT registration students were informed that the information would be used for educational research and evaluation of the UKCAT and that the results would be published in a form in which individual students could not be identified. Thus the data utilised was routinely contemporaneously collected and anonymised. Consequently, exemption from an external ethical review was confirmed in writing by the Chair of Durham University School for Health Research Ethics Committee.

Study group

Univariate analyses

It should be noted, that when interpreting standardised scores and regression coefficients, for A level tariffs each standard deviation unit (SD) is equivalent to around 45 UCAS points. For the UKCAT scores each SD is worth around 200 points.

Multivariable regression analyses

Further comparison with previous findings

This study builds on the previous work investigating sociodemographic predictors of A level and UKCAT performance [10]. Our study sample was relatively comparable with the subgroup of UKCAT candidates providing data in this earlier study. The present sample used was slightly smaller in that we only included those with complete sociodemographic information, as opposed to just non-missing A level data. In practice this meant that it was mainly those under 21 years that lacked information on socioeconomic background that were excluded from the final sub-sample for analysis, as this was the principal WP variable missing, aside from A level attainment. It should be noted, however, that many older individuals would have already been excluded on the basis of missing A level data. Nevertheless, as in the present sample, the sub-group of 2006 candidates included in the previous study tended to be younger, more likely to be of White ethnicity have attended an independent/grammar school (EASL status was not available in the 2006 cohort). As outlined earlier, our findings were largely consistent with the observations reported in this earlier study. However, it is important to consider the apparent inconsistencies in the findings between these two studies. Firstly, our lack of any observed sex difference in A level achievement, in contrast to the findings of James et al. [10], can be explained by the differing ways that the metric of A level performance was constructed. In the present study we created a tariff score by summing the UCAS for the three best exam grades (excluding general studies and critical thinking) irrespective of whether the A levels were ‘pure’ science (i.e. chemistry, biology, physics and mathematics). Indeed, James et al. report only slightly higher average tariff scores for pure science subjects in males and no sex difference in overall average tariff scores. Thus, our observations are largely consistent with those reported by James et al., though we are unable to rule out the effects of a recognised recent secular trend towards males obtaining more top grades in science A levels compared to females [15]. Secondly, in the present study, males did not score significantly higher than females on the abstract reasoning and decision making scales. This appears to contrast somewhat with the findings of James et al. where decision making was the only subtest where a sex difference was not apparent. However, these inconsistencies may be relatively trivial once the differences in analysis approach are accounted for. Firstly, the males in our cohort performed more poorly than females on the decision making items, although the magnitude of this difference was slight and the p value for significant testing (p = .027) could be considered modest given the number of observations in the analysis. Thus, it may have been the case that by using a dichotomous outcome for UKCAT scores the James study may have been underpowered to detect a slight sex difference in performance on this subscale. Similarly, in the case of the abstract reasoning subtest this previous study reported a slight (but statistically significant) tendency for males to perform, more poorly on this element of the UKCAT; males had a 16% lower odds of scoring above the 30^th centile on this particular subtest. In contrast we observed no significant sex difference. However, we utilised UKCAT scores as a continuous, rather than as a dichotomous metric. Although these findings are not included in our results section, as with the report by James et al., we noted a slight excess of females scoring above the 30^th centile on abstract reasoning (1,397 females compared to 1,186 males). Thus, our results are largely consistent with those reported by the earlier study once the differences in methodologies are accounted for.

A previous study demonstrated that use of the UKCAT as a threshold score in the admissions process appears to ameliorate the disadvantage faced by lower socio-economic groups when applying to medical schools [5]. In addition, use of the UKCAT scores as a threshold in the admissions process was associated with increased odds of entrants being male, from a low socioeconomic status background and a state (non-grammar) school (the latter trend not reaching statistical significance). In contrast, universities placing less emphasis on use of the test were more likely to admit entrants with relatively low academic attainment and with English as a second language. These observations are generally consistent with the properties of the two performance metrics as reported in this present study. Thus, the present findings imply that it is mainly the differences in sensitivity to sociodemographic factors (i.e. bias) between A levels and the UKCAT that are driving these differences. The obvious exception to this is that in the present study we found no evidence that the UKCAT was less biased than A levels against those from a non-professional socioeconomic background than were A levels. However, it is possible that if UKCAT performance is less sensitive to schooling than A level attainment then this difference may be at least partly mediating this previous observation [5]. Moreover, our present results do not explain why universities that place little emphasis on the test scores may be more likely to have entrants with below average educational performance.

Limitations

The primary limitation of this study was that analysis could only be conducted on a minority of applicants for 2009, due to missing data. This limits our confidence in the generalisability of these findings to the wider population of UKCAT candidates. In particular, the individuals with complete data were more likely to be of White ethnicity, have attended an independent/grammar school, be younger and to speak English as a first language. Thus, we must be extremely cautious in drawing any conclusions about the association between WP variables and UKCAT performance in sub-groups of candidates who belong to the opposite sociodemographic categories. Missing data modelling in conjunction with imputational approaches could have been used to inform sensitivity analysis (i.e. assess how strong the findings are under different assumptions). Such an approach has been previously employed with educational data to provide an indication of the extent to which data are missing at random as opposed to being non-ignorable [5]. However, it was felt that imputational approaches could have added a significant degree of uncertainty to the dataset, especially as more than one variable would have had to be imputed. In addition the final sample providing data for analysis would have been difficult to compare with the previous sub-group of candidates studied [10]. Inclusion of advanced qualifications other than A levels may have modestly addressed the missingness but also potentially added a degree of complexity and possible confounding; it is uncertain to what extent other tests of educational attainment are equivalent to each other (e.g. Scottish Highers vs A levels). Thus, on balance, it was felt that restricting the analysis to those with complete data would enhance the internal validity of the findings, accepting that this would be at the expense of generalisablity of the results observed.

Whilst some descriptive analysis was conducted using more broadly defined ethnic groups it would have been desirable to have detailed modelling in relation to ethnicity. Certain ethnic groups (e.g. those describing themselves as ‘Black’) are relatively under-represented in UK medical and dental education whilst others are over-represented (e.g. Asians) in relation to the national population demographics [12].

A further limitation, as stated earlier, is that the use of Tobit regression, whilst necessary with censored data, still leads to some informational loss compared to linear regression, and the coefficients and confidence intervals produced by the two approaches (i.e. Tobit and linear regression) may not be easily comparable.

Implications for practice and directions for future research

The UKCAT is a high stakes test; the psychometric properties of the test, in conjunction with a widespread adoption as part of the admissions process could at least partly determine the nature of the UK’s future medical and dental workforce. Both the present and a previous study report evidence of a certain degree of sociodemographic bias in the UKCAT responses. Firstly, possibly the most consistently reported of these is the observation that males achieve higher scores on the UKCAT than females. In the UK females are currently disproportionately represented amongst medical and dental school entrants. This issue has, at times, stirred up controversial debate [16]. Currently female doctors are more likely to work part-time [17] and retire early [18] compared to their male counterparts. Nevertheless, it should be highlighted that a previous study reported that female doctors were at lower risk of professional misconduct after qualification, even after adjusting for a number of potential confounding factors [19]. Moreover, in the UK, women may outperform men in certain medical undergraduate [20] and post-graduate exams [21]. This study has highlighted some differences in the differential sensitivities of the UKCAT subtests to sex. Thus our findings may assist universities making informed decisions about how much weight to place on each element of the UKCAT when selecting entrants.

The potential insensitivity of the UKCAT to educational background is certainly a factor that could help address the issue of widening participation in the professions. However, it should be highlighted that the WP agenda is not purely focused on issues of social equity and fairness; there is evidence from North American research that students drawn from minority populations may be more likely to eventually practice in areas that have been traditionally underserved by health care provision [22]. In the US attempts to address racial imbalances within the professions, including medicine, via the ‘affirmative action’ approach have proved controversial and have been the subject of a series of Court cases [23]. Earlier North American researchers have suggested that the use of cognitively based aptitude tests (such as the UKCAT) will never address the under representation of racial minorities in medical education as such instruments tend to produce similar mean raw scores according to ethnicity [24]. Rather, it has been postulated that the most plausible way of achieving a medical school population with a similar ethnic profile to the population from which they are drawn is to have quotas for each group. It has been suggested that these quotas can be fulfilled without any appreciable lowering of average academic performance at medical school [25]. In the UK such affirmative action-style approaches have not been adopted and even their legality would have to be tested.

Our findings suggest that the UKCAT test that may penalise those who do not speak English as a first language more severely than science-based A levels do. Nevertheless, fluency in spoken English has been reported to correlate significantly with patient and examiner ratings of global communication, which is considered a key attribute of a clinician [26]. It could therefore be argued that it is reasonable for the UKCAT to evaluate elements of linguistic ability such as verbal reasoning.

Future research should focus on obtaining further evidence regarding whether or not the UKCAT has the ability to predict undergraduate and post-graduate performance and progression, over and above that possible via traditional measures of educational attainment. Moreover, it may be that the content and delivery of the test can be modified to further decrease the sensitivity to educational background. In the field of education there is some evidence that ‘dynamic testing’ may be better at predicting an individual’s academic and potential compared to traditional (‘static’) cognitive assessments. This may be especially true where a candidate’s education has been poor or disrupted [27]. Dynamic tests create a learning environment within the test structure by providing novel situations and then evaluate the nature and number of prompts, hints and clues the candidate requires in order to achieve a correct response. Such tests correlate highly with traditional ‘intelligence tests’ but provide additional information relating to cognitive flexibility and learning potential [28]; attributes obviously pertinent to medical or dental practitioners.