This study rests on the co-incidence of three fortunate sets of circumstances. First, it was possible to follow an entire medical school entry cohort of students, the majority of whom had taken the UKCAT cognitive tests but the scores had not been used in the students’ selection (so there was no restriction of range within the study population) and then for research purposes these students took the same non-cognitive tests that were included within the subsequent years’ UKCAT. Secondly, the tutor assessments provide perhaps the closest approximation to all-round evaluation of individual professional performance during the early years of a medical course. Thirdly, the HYMS theme-based examination system requires allocation of examination marks into the three separate themes that depend mainly on scientific knowledge (Theme A), on clinical information gathering and inter-personal skills (Theme B) and on application of statistical and analytical skills to acquired knowledge (Theme C), thus allowing different aspects of students’ performance to be distinguished.
Principal findings
This study has found numerous significant relationships between students’ prior cognitive and non-cognitive test measures (all of which became components of the UKCAT), subsequent tutor assessments of individual and group-related behaviours, and academic and clinical examination results undertaken over the first two years of a medical school course.
The individual predictive ability of any of the measures (of cognitive skills, non-cognitive traits and of behaviour) appears weak, the majority of the statistically significant correlations being in the range 0.16 to 0.24 and only a few exceeded 0.30 (accounting for only 9% of the variance). However, in the field of organisational psychology it is well recognised that even quite weak correlations are useful predictors of workplace outcomes, especially if the ratio of applicants to selectees is large [12], as with applications to medical school. Such measures may therefore usefully predict medical student performance and later professional conduct. This position is supported by a recent meta-analysis of the predictive value of ability and personality test scores, which concluded that such scores are more successful in predicting educational, work and life outcomes than is often admitted by critics; in particular “In medical education, personality characteristics gain importance for later academic performance when applied practice (such as performance in practicums and clerkships) increasingly plays a part” [13].
A further possible criticism of the findings is that there are relatively few significant relationships, as revealed by the number of blank cells in Additional file 4: Table S1 and Additional file 5: Table S2 particularly. However, of the 380 possible relationships that could have been included in these tables (19 personality scales by 20 tutor ratings), 51 (or 13.4%) were significant at the 5% level or better; this becomes 18.2% of 280, if only the 14 personality scales actually appearing in the tables are considered. When compared with the chance proportion of 5% it is clear that the overall pattern of results is better than chance. Additional analyses were conducted to adduce further evidence for the significance of the overall pattern, including multiple regression to calculate the proportion of common variance shared by the personality scales and the tutor ratings, and application of a recently published resampling technique [14] for calculating the probability of relationships between personality and behaviour. However, neither provided stronger support than the simple number of correlations, so the details are not presented here.
In summary, although the statistically significant coefficients in the matrices could all be chance findings this seems unlikely because all of the significant results (and most of the non-significant ones, which are not shown) are in the direction that the particular personality traits and cognitive skills would be expected to affect the behavioural and examination outcomes.
The findings in more detail
Overall, it appears that the non-cognitive tests do predict normally unrecorded aspects of medical students’ performance. Greater narcissism, aloofness and irrational thinking predicted lower tutor ratings for group-related behaviours, and poorer overall examination performance including in the most knowledge-based Theme A, as well as in Theme B (person-centred care), though not in Theme C. Good teamwork skills therefore appear to be an important student attribute. The finding that RESILIENCE predicted good group functioning in year 1, while conscientiousness and SELF-CONTROL predict this in year 2, may reflect students’ increasing ease and familiarity with medical school. The early years of the HYMS course involves not only problem-based learning sessions but also a structured weekly programme of lectures, laboratory-based practical classes and tutored clinical experience, so the importance of group functioning to the examination outcomes cannot be explained by exclusive reliance on problem-based group learning.
While the UKCAT cognitive tests scores did not predict many of the behaviours subsequently rated by tutors, they did predict both year 1 and year 2 overall examination scores, and performance in Theme A and Theme C. Notably, but not unexpectedly, cognitive test scores did not predict performance in the communication skills or physical examination components of the clinical examinations (OSCEs) in either year, although they did predict overall Theme B scores at the end of year 2. This may reflect increasing emphasis in year 2 on clinical reasoning skills within this theme.
Problems with predicting traditional medical school results
Prediction of any outcome depends on selection and accurate measurement of both predictor (be that a cognitive skill or a personality trait) and an appropriate outcome (for example, academic marks, skills, or professional behaviours). In order to examine whether any of the cognitive and pilot non-cognitive components of the UKCAT are valid predictors it is necessary to have appropriate measures of the behaviours that the tests are expected to predict. Paradoxically, in-course examinations do not usually assess outcomes that are related to the qualities that non-cognitive selection tests are designed to measure. In general, medical schools have concentrated on traditional examinations (in part so that licensing requirements can be fulfilled unambiguously) that test recall of factual material and reasoning, which depends on memory and cognitive skills. Non-cognitive skills, such as communication and doctor-patient relationships, are often tested informally within medical training, and the results recorded in pass/fail format, usually with only a small number of failures (as with many academic medical examinations). Thus a problem for statistical comparisons of traditional medical school results is that pass/fail clinical outcomes are categorical and typically severely skewed while the predictor test results are normally distributed on a continuous scale. The present study benefits from finer gradation in both selection tests and the outcomes measured, and from better matching between predictor and criterion variables.
Other studies
Cognitive tests would be expected to predict success in examinations of knowledge recall. This has been shown in one recent study [15] about medical students, but not in others [5–7]. No reported studies have been found that tested the hypothesis that non-cognitive characteristics desired in a good doctor have any bearing on students’ examination performance, although a Conscientiousness Index [16] has been proposed as a measure of medical students' professionalism based on a variety of routinely made behavioural observations that can be recorded in a systematic and reliable way. The present study is the first to demonstrate that measurement of a range of different personal qualities can predict different aspects of medical students’ performance. The broad range of suitable measures of non-cognitive skills and professional behaviours used here provides possible tools for future research.
Relevance of this study: implications for clinicians and policymakers
All health professionals require good communication skills and the ability to work effectively as part of a team. Doctors have additional roles, not only in education and in research but also, principally, as decision-makers and leaders of teams working in situations of clinical complexity and uncertainty. All doctors therefore should be committed to reflective practice, monitoring their own contribution and working continually to improve their own and their team’s performance [17]. Selection tests need to encompass more than purely cognitive skills, and test for the other abilities that underpin the whole range of the doctor’s professional tasks.
In the present study, non-cognitive tests were found to predict behaviours likely to be important when working as a doctor, such as functioning well with others in groups, acknowledging weaknesses and accepting feedback (which underpins the ability to learn and change through experience), and identified other traits, such as narcissism, aloofness and irrational thinking, that are likely to diminish a doctor’s ability to fulfil these roles. Improvement in the predictive power of such non-cognitive tests depends not only on finding better tests, but on devising and using better measures of critical behaviours in medical school and professional practice.
Unanswered questions and future research
The findings raise three further questions with implications for medical education. First, in the selection of future medical students, should the predictive ability of cognitive and non-cognitive qualities be explored more fully, particularly in relation to the difficult judgments and high level of inter-personal skills required of a doctor? Secondly, should medical schools be making use of more standardised and repeated behavioural observations undertaken by tutors throughout the students’ training, in addition to current measures of clinical competence (such as mini-clinical evaluation exercise [18]) used in formative and summative assessments of medical students? Thirdly, are medical schools failing to assess all the appropriate outcomes from their courses by continuing to rely too heavily on formal examinations based mainly on recall of acquired knowledge and some reasoning? The latter approach has produced many doctors who are intellectually and academically prepared for their careers, but personal failings, such as poor communication skills, lack of empathy and concern for patients, motivation, and mental health issues, tend to impinge on their work effectiveness. Such failings are typically detected too late, when brought to light by examination failure or the need for disciplinary action. If it is accepted that doctors require more than just academic knowledge and technical skills then it makes sense to look for additional qualities at the outset and select those applicants who already have these qualities or seem capable of developing them.
