An assessment of the impact of formal preparation activities on performance in the University Clinical Aptitude Test (UCAT): a national study

Abstract

Background

Previous studies have shown performance in the University Clinical Aptitude Test (UCAT) to be associated with measures of candidate socio-economic advantage such as parental occupation and type of school attended. It is possible that access to preparation support and materials may in part explain these associations. In this paper we determine whether use of formal preparation resources is associated with higher UCAT scores and whether differences in use of preparation resources exist between socio-demographic groups.

Methods

After completing the 2017 UCAT UK school-leaver candidates (n = 14,332) were asked to answer a questionnaire regarding their use of official UCAT and commercial resources, school-based support, and time spent preparing. Multiple linear and logistic regression models were used to evaluate the associations between preparedness, demographic characteristics and UCAT performance.

Results

Five thousand, four hundred thirty-nine (38%) candidates responded to the questionnaire. Use of freely available UCAT official practice tests, paid commercial materials, attendance at school-based preparation courses and spending more time preparing were significantly associated with higher UCAT scores. Candidates who were from less deprived backgrounds and attending independent or grammar schools were significantly more likely to use paid commercial materials and spend longer preparing.

Conclusions

Reported use of preparation resources varies between candidates from different socio-demographic backgrounds and is associated independently with performance in the UCAT. Increasing the availability of freely available resources may mitigate some of these differences.

Introduction

There is growing use of psychometric aptitude tests by universities to assist in the selection of future medical students. Use of these tests enables medical schools to assess candidate attitudes and potential future professional behaviour, and also overcomes two existing challenges in assessing cognitive ability: (i) continued grade inflation in school-leaving examinations [1] (which historically have formed the basis for medical schools to assess candidate academic ability) which limits attempts to differentiate between applicants, and (ii) the socio-demographic patterning of performance in these examinations and which may hinder institutional efforts to widen access to medicine [2, 3].

The most widely used aptitude test in the UK is the University Clinical Aptitude Test (UCAT; previously the UK Clinical Aptitude Test (UKCAT)). Introduced in 2006 it is now used by the majority of UK medical schools as part of their student selection processes. The UCAT is a computer-based test and is delivered on behalf of the UCAT Consortium by Pearson VUE at designated test centres throughout the UK. Applicants to the medical schools using the UCAT sit the test before they make their application to medical school and can only make one attempt at the test in each annual university application cycle [4].

The UCAT comprises four cognitive subsections (verbal reasoning, quantitative reasoning, abstract reasoning, and decision making) and situational judgement test (SJT) [4]. Each cognitive section is scored separately on a scale ranging from a minimum of 300 to a maximum of 900, resulting in a total score ranging from 1200 to 3600. The SJT component is marked separately with students being placed into four Bands in which Band 1 is the best performance. Details of the distribution of total scores by decile and the proportion of students in each SJT Band for test cycles 2014–2017 are shown in Tables 1 and 2 [5].

Table 1 Deciles of total UKCAT score in 2014, 2015 and 2017 (NB. 2016 is excluded as the decision making section was not used operationally that year) [5]

Table 2 Proportion of candidates in each SJT Band in 2015, 2016 and 2017[5]

There is no consensus as to how medical schools should use the UCAT in selection and a variety of approaches exist whereby UCAT scores are used separately or integrated with other metrics to shortlist candidates for interview and/or to determine whether an offer is made [6].

Although the UCAT uses constructs which are designed to be less affected by socio-demographic factors than historical predictors of academic ability [4], studies have suggested that the inherent biases associated with the school examinations taken by UK school leavers (pupils from White and professional social-class backgrounds and attending independent or academically selective (‘grammar’) schools tend to do best) also exist for UCAT performance albeit the biases may be reduced [6]. Further, males have been shown to perform better than females on the cognitive components of the UCAT [2, 7,8,9,10,11] with the reverse noted for the SJT element [10, 11].

The reasons for this socio-demographic patterning of performance in the UCAT are not completely clear. We have previously suggested that access to preparation support and materials may in part explain the associations with measures of social advantage [6]. The UCAT Consortium provides free-to-use preparation resources on their website; these include practice tests, practice questions for each subsection, a candidate preparation toolkit and an official guide to the test [4]. Schools, aware that the UCAT is a key element in selection for medical degree programmes, may also include advice and practice for the UCAT as part of wider measures they have in place to support pupils to make successful applications [9]. UCAT candidates also have access to an increasing number of commercially-produced practice materials and courses which provide participants with test-taking strategies and access to a large, highly realistic bank of questions typically at a significant cost [8, 9]. Estimates concerning the use of commercial courses or materials vary with one study reporting that 20% of students attended a fee-paying course [9] whilst other studies in Australia, where a similar admissions test was used, suggest a prevalence of over 50% [12, 13].

In this paper we report a study in which we linked candidates’ responses to a questionnaire-based survey designed to elicit information on their preparation to their performance in the 2017 sitting of the UCAT. Specifically we sought to determine whether (i) formal preparation, including use of free and paid-for preparation resources, is associated with higher UCAT scores, and (ii) the demographic characteristics of those using preparation resources and whether use of preparation resources differs between socio-demographic subgroups. We limited our analyses to candidates aged 16–20 years because we wished to focus on the experience of school pupils and recent school leavers rather than graduates.

Methods

All UK-resident candidates who sat the UCAT in 2017 were invited to participate in the study. Candidates aged under 16 years or who were aged 21 years and older were subsequently excluded from the study population to leave a sub-population we termed ‘school leavers’ (aged 16–20 years). Upon completion of the test candidates were directed to a brief on-screen questionnaire which was preceded by a statement regarding the research and a tick-box in order to gain consent. The questionnaire was created by the research team, informed by previous literature [8, 9] and comprised five questions regarding candidates’ engagement with school/college-based support, use of the official UCAT preparation materials, use of free-to-access and of paid online commercial materials, and an estimate of total time spent preparing for the test (see Additional file 1: Appendix 1). Whilst we acknowledge that books are a commonly used preparation resource, we believed these were unlikely to be used as a candidate’s primary resource. As such, the survey focused on online materials which are typically more up-to-date and reflective of the current test content and layout. Candidates were able to decline to participate at any point and could exit the UCAT without prejudice.

Construction of dataset

When candidates complete their online registration form to secure a place to sit the UCAT they provide the following demographic information: date of birth, gender, ethnicity, parental occupation, nationality, and school/college details. Additionally, they are invited to consent to the use of this personal information and data relating to their UCAT performance by researchers to build further the evidence-based underpinning use of the test in selection. The demographic, test performance and preparation survey data of all candidates consenting to participate in research were linked to create a pseudo-anonymised dataset which was made available to the research team via the University of Dundee Health Informatics Centre’s safe haven.

Data collection

Socio-demographic characteristics

Candidate age was calculated as on September 1 2018 using reported date of birth. Ethnicity was self-defined using the full Census classification and then categorised before release to the research team as Asian, Black, White, Mixed or Other. Candidates selected their school type from a drop-down list as either: Sixth Form or Further Education College, Comprehensive, Independent or Fee Paying Private, Grammar or Other. Information on parental occupation was used to derive the National Statistics Socio-economic Classification (NS-SEC) for each student in accordance with the Government method of calculation [14]. The NS-SEC is a measure of socioeconomic status with a scale ranging from 1 to 5; 1 is the least deprivation whilst 5 is the most deprived. Additionally, information was available on whether a candidate had been awarded a bursary by UCAT (candidates in financial need who meet specific criteria are eligible for a bursary covering the full test fee).

UCAT performance

Individual sub-section and total UCAT scores were provided for each candidate.

Previous attempts at the UCAT

It is possible that some candidates choose to have a ‘practice run’ at the UCAT, sitting the test in the year prior to making their application for medical school, so as to familiarize themselves with the process. This approach may be considered a form of preparation and we recorded the number of times, if any, the candidate had taken the test previously.

Self-reported preparation methods and time

Using the responses to the questionnaire six binary categories of preparedness were created (see Additional file 1: Appendix 2 and Table 3). It was not deemed appropriate to combine these to create a single variable to indicate if students were prepared or unprepared.

Table 3 Preparedness categories and how they were derived

Statistical analyses

Using a conservative estimate of a standard deviation of 300 in total UCAT score (based on UCAT data [5]), to detect a difference in UCAT score between preparedness groups of only 40 points, 1463 participants are required in each group (2926 in total) for 95% power and 5% significance. For reference, a difference of 80 points equates to moving from the 5^th to 6^th decile rank in total UCAT scores. The data collected provides over 5000 participants meaning the study is adequately powered.

All statistical analysis was conducted using Stata 15 (StataCorp LP, College Station, TX, USA). Descriptive statistics of the candidate demographics and their UCAT results were reported in terms of responders and non-responders. Chi-squared tests were conducted for the categorical and binary variables whilst independent t-tests and Mann Whitney U tests were conducted for the continuous variables, as appropriate.

In order to determine (and confirm previously described) associations between the demographic variables and UCAT performance, a multiple linear regression model was constructed with UCAT score as the outcome and gender, ethnicity, NS-SEC, school type, bursary status and attempt number as covariates. Inclusion in this aspect of the analysis was not dependent on responding to the questionnaire as all data used were routinely collected by the UCAT Consortium. A binary logistic regression model was fitted with SJT Band 1 or 2 (compared to SJT Band 3 or 4) as the outcome and the same covariates as above.

A multiple linear regression model was then fitted to investigate the association between total UCAT score (outcome variable) and each of the preparation categories, using the same covariates as above. Logistic models were fitted to investigate the association between the preparedness categories and performing well or badly on the SJT component of the test (SJT Band 1 or 2 (better performance) compared to SJT Band 3 or 4 (poorer performance) with the same covariates as above.

Binary logistic regression models were constructed, using each of the preparedness categories as an outcome and participant characteristics as covariates. Based on UCAT guidance, preparation time was categorised into those who had reported spending more than 20 h and those who had reported spending less than this.

Whilst p-values < 0.05 are often considered statistically significant, we would urge the reader to exercise caution when interpreting the results of the significance tests given the number of tests conducted and the resulting increased risk of type 1 error; we also encourage readers to look at the magnitude of differences between groups with corresponding confidence intervals given the large sample size.

Research ethics statement

Ethical approval for this study was provided by the University of Birmingham Ethics Committee (ERN_17-0521).

Results

In 2017 14,332 UK-resident candidates aged 16–20 years sat the UCAT of whom 5,439 (38%) responded to the preparedness questionnaire (Fig. 1).

Fig. 1

Derivation of study sample

Respondent Characteristics and Preparation

Compared to non-responders, respondents were more likely to be younger, female, of White ethnicity, of higher socioeconomic status, have attended a sixth form/further education college or comprehensive school, and were less likely to have received a bursary for the test (Table 4). Although statistically significant, the absolute magnitude of differences in socio-demographic characteristics between responders and non-responders was small, due to the large sample size. For instance, the mean difference in age between responders and non-responders, whilst statistically significant, was 18 days. Responders also performed better than non-responders in the test overall, and in each subsection, with the exception of the abstract reasoning component (Table 5). A significantly greater proportion of respondents achieved an SJT Band 1 or 2 compared to Band 3 or 4 (p = 0.025) and were taking the test for the first time.

Table 4 Demographic characteristics of the study population in terms of responders and non-responders

Table 5 UCAT total and sub-section scores for responders and non-responders

Among responders (n = 5439), 877 (16.1%) reported attendance at a school-based preparation course, 4434 (81.5%) reported using the official UCAT practice tests and 3047 (56%) reported using paid commercial materials (Table 6). Only 34 (0.6%) respondents reported not spending any time preparing for the test whilst 1320 (24.3%) prepared for over 40 h (Table 7).

Table 6 Demographic characteristics by preparedness category. Below are binary logistic regression models with each of the preparedness categories (with the exception of retaking) as the outcome variables and demographic variables as covariates. The reference category for each variable has an odds ratio of 1. Significant results with negative direction of effect have been shaded red whilst significant results with positive direction of effect have been shaded green

Table 7 Demographic characteristics by time spent preparing for the test. Binary logistic regression model with preparing > 20 h as the outcome variable and demographic variables as covariates as the final column. The reference category for each variable has an odds ratio of 1. Missing values not included in chi-squared tests or regression analysis. Significant results with negative direction of effect have been shaded red whilst significant results with positive direction of effect have been shaded green

Socio-demographic factors and UCAT performance

The results of the multiple linear regression model, constructed using all candidates who were UK residents and aged 16–20 years and with non-missing values (n = 13,016), indicate that candidates who are male, attend grammar or independent or private fee-paying schools, were of White ethnicity or of higher socioeconomic background perform significantly better on the test (Table 8). Further test subsection analyses can be found in Additional file 1: Appendix 3.

Table 8 Multiple linear regression model with total UCAT score as the outcome and preparedness categories and demographic variables as covariates. The third and fourth columns show the multiple linear regression model with total UCAT score as the outcome and demographic variables as covariates using all included candidates (i.e. responders and non-responders) and does not therefore adjust for preparedness

A binary logistic regression model (Table 9), with the Band 1 or 2 (better performance) in the SJT component as the outcome (compared to Band 3 or 4), showed that males were significantly less likely to achieve the top two Bands (Band 1 or 2) than females. Those from independent, grammar and comprehensive schools were significantly more likely to be placed in the top two Bands compared to those attending a sixth form or college. Candidates of Asian and Black ethnicities performed significantly worse than those of White ethnicity. Candidates from more deprived backgrounds performed significantly worse than those from least deprived backgrounds. Students who were retaking the test were significantly more likely to obtain a Band 1 or 2; further analysis with Band 4 as the outcome variable showed similar results (Additional file 1: Appendix 4).

Table 9 Binary logistic regression model with SJT Band 1 or 2 as the outcome using all included candidates and does not adjust for preparedness (n = 14,432). Binary logistic regression model with SJT Band 1 or 2 as the outcome and preparedness categories and demographic variables as covariates using all responder (n = 5439)

Impact of preparation on UCAT performance

Use of the freely available official UCAT timed practice tests was associated with greatest impact on test performance (MD = 67.77, 95% CI: 52.01 to 83.53; p < 0.001) (Table 8). Attendance at a school-based preparation course (MD = 36.66, 95% CI: 20.25 to 53.07) or using paid commercial materials (MD = 37.65, 95% CI: 24.81 to 50.49) also significantly improve test performance (p < 0.001 for both). Overall UCAT performance improved as the time spent preparing increased (see Fig. 2). When evaluating subsections of the UCAT, performance appeared to increase with greater preparation time categories for the abstract reasoning and quantitative reasoning subsections only; for other subsections performance seems to plateau at moderate levels of preparation. Differences in scores between those who retook the test, used paid commercial materials or spent longer preparing, compared to those who did not, were largely observed in the abstract reasoning and quantitative reasoning subsections (Additional file 1: Appendix 5). Attendance at a school-based preparation course and studying for over 20 h for the test were both found to be significantly associated with achieving a Band 1 or 2 in the SJT (Table 9). Moreover, the socio-demographic patterns in UCAT performance seen using all eligible candidates such as better performance by males or those of White ethnicity, remain despite adjustment for preparedness in our model.

Fig. 2

Graphs of time spent preparing compared to UCAT performance both overall and in each subsection. The data has been generated from the multiple logistic regression models outlined in the methods and results with error bars relating to the 95% confidence interval for each data point. The mean difference in UCAT score for each category is compared to the average score for the ‘Did not prepare category (i.e. 0 h)

Use of preparation materials and time taken to prepare, and candidate characteristics

Males were significantly less likely to report using the official UCAT practice tests, using free commercial materials or to prepare for more than 20 h for the test than females (see Tables 6 and 7). Compared to those at sixth form or college, students attending grammar and independent schools were significantly more likely to report attendance at a school-based preparation course with odds ratios of 1.50 (95% CI: 1.22 to 1.85, p < 0.001) and 3.02 (95% CI: 2.49 to 3.65, p < 0.001) respectively. Similarly, those attending independent or grammar schools were significantly more likely to use the official UCAT tests, paid commercial and to prepare for more than 20 h for the test compared to those attending sixth forms or colleges.

Students of Asian or Black ethnicity were significantly more likely to have reported attending a school-based preparation course, using free commercial materials or spending more than 20 h preparing for the test compared to those of White ethnicity. Compared to White students, those of Asian ethnicity were significantly more likely to use paid commercial resources.

Candidates in most socio-economically deprived quintile were significantly less likely to use the official tests, use paid commercial materials or prepare more than 20 h for the test compared to the least deprived (those with an NS-SEC of 1). Students in receipt of a bursary were significantly less likely to report using paid commercial materials or to prepare for more than 20 h for the test compared to those who had not received a bursary.

Discussion

Our analysis indicates that use of the official UCAT materials, use of paid commercial materials, attendance at school-based preparation courses and spending more time preparing are all associated with better performance on the cognitive components of the test. A dose–response relationship was observed for the increasing time spent preparing with no apparent ceiling of effect. Statistically significant differences in the use of these preparation resources were identified between different socio-demographic groups and those from differing school types.

With regards to the SJT component, spending more time preparing for the test was a significant predictor of better performance but the findings indicate no evidence of an association between use of specific preparation resources and performance with the exception of attendance at school-based preparation courses. One explanation for this finding may be that because SJT performance correlates with personality traits such as conscientiousness [15, 16], those spending longer preparing for the test are more likely to be conscientious candidates and will therefore inherently perform better on this component.

Our findings reiterate the socio-demographic patterning reported in UCAT and SJT performance by otherss [2, 6,7,8, 10, 11]. We note also that candidates in receipt of UCAT Consortium bursaries perform significantly worse on the test.

Our data suggest that candidates using preparation resources perform better in the test, an observation which concurs with the literature on the Undergraduate Medicine and Health Sciences Admissions Test (UMAT) used in Australia [12, 13]. Two questionnaire-based studies by Lambe et al. have previously investigated the impact of modes of preparation on self-reported UCAT performance [8, 9]. The first study found that increased preparation time correlated with higher test scores [8] whilst the second noted that those who used UCAT-provided practice tests performed 67 points better than those who did not (95% CI: 22.6 to 110.3, p < 0.001) [9]. Further interpretation of these observations is compromised by absence of information on socio-demographics of respondents and non-respondents [8, 9]. In contrast to our study, Lambe et al. did not identify a significant difference in UCAT performance between those who had attended school or commercial courses and those who had not. However the reported use of paid commercial coaching resources and school-based preparation courses by their participants was markedly lower than that reported in our study (9% commercial and 7% school-based versus 56% and 16% respectively) [8, 9].

Our analysis suggests that the differences in UCAT scores associated with preparedness can largely be attributed to better performance in the abstract reasoning and quantitative reasoning subsections which appears to echo the findings from the equivalent sections of the UMAT [12, 13]. In our subsection analysis, only performance in these sections demonstrated a dose–response relationship with time spent preparing. The researchers investigating the UMAT argued that these sections are more easily answered using learned, specific problem-solving skills whilst the verbal reasoning component is less amenable to using learned techniques from coaching [12].

An important limitation of this study is the likelihood of responder bias. Whilst the responders and non-responders were largely comparable in terms of whole numbers with regards to demographic characteristics, those who responded performed significantly better on the test. Although responders were unaware of their test score when completing the survey, it is possible that those who believed they had not performed well were less likely to complete our questionnaire. We also acknowledge the categories used for some of the variables, such as ethnicity and school type, are very broad and as a result there is likely to be a degree of variability within each group. An additional limitation is the lack of a measure of prior academic attainment for each candidate, which is a known predictor of UCAT performance [2] and may possibly moderate the impact of preparation [13]. In the future we plan to combine our data with the UKMED dataset to allow adjustment for further confounders such as academic attainment and to answer further research questions such as the impact of preparation on the predictive validity of the UCAT. Similarly, in the absence of a measure of pre-preparation UCAT score it is not possible to determine at an individual level whether preparation leads to an improvement in test performance.

We further acknowledge that there is likely a degree of correlation within the different demographic and preparedness variables, for instance between school type and socioeconomic status or preparation time and use of commercial materials. These inter-variable relationships may influence the independent effect of each variable and further research to delineate this would be useful (although was considered beyond the scope of this paper). In addition, qualitative research investigating candidate preparation in relation to socio-demographic characteristics and social norms may further our understanding of the issues.

The effectiveness of preparation for cognitive tests may be attributed to improving candidates’ test taking techniques and pattern recognition skills but perhaps also by increasing their familiarity with the test format and various question types. Medical schools should be cautious about presuming these types of selection tests only assess inherent intellectual ability. It is possible that access to preparation tools and support may partially explain the association between higher UCAT scores and attendance at a selective schools and higher socioeconomic family background. The key implication, therefore, is that addressing the differential in access to preparation resources may mitigate differences in performance between socio-demographic groups. Suggested means of achieving this include creating more freely available resources through the UCAT website and increased awareness of such resources.

Preparation was found to have little association with performance in the SJT component, yet differences of a large magnitude exist between ethnic and socioeconomic groups; language, cultural and social norms may be important determinants of SJT performance, rather than preparedness.

Conclusions

This study has demonstrated an association between use of preparatory materials and courses and scores in the 2017 UCAT and demonstrated differential use of preparation resources among candidates from different socio-demographic groups. We have also confirmed previous reports of associations between candidate performance and demographic characteristics including ethnicity and family socio-economic status.