Structure of the tutor assessment scales

The structure of the tutor assessment scales was analysed for each session using reliability analysis. The internal consistency of the scales was high in all cases, the Cronbach alpha coefficients being 0.91 for May 2008 (17 items), 0.87 for January 2009 (14 items), and 0.90 for May 2009 (20 items), indicating that there was a strong tendency for the tutors to give similar ratings to individual students for all items in the scales.

Prediction of tutor assessments by non-cognitive tests

The significant relationships found between non-cognitive characteristics and tutor assessments in year 1 and year 2 are shown in Additional file 4: Table S1 and Additional file 5: Table S2 respectively.

From the SAI test, high neuroticism (p < .05), moodiness (p < .05), irrational thinking (p < .05 to p < .001), and low conscientiousness (p < .05) predicted lower ratings by tutors for some individuals’ behaviours in year 1. These traits combine into the SAI factor RESILIENCE, which itself predicts some positive behaviours of individuals in the first year PBL groups (p < .05). In year 2, the SAI factor SELF-CONTROL (incorporating + restraint,-permissiveness, -antisocial behaviour) (p < .05 to p < .001) and conscientiousness (p < .05 to p < .01) predict some positive behaviours of individuals, while the individual traits permissiveness and anti-social behaviour both correlate negatively (p < .05) with some year 2 behaviours.

All of the individual measures of the ITQ (empathy, confidence, narcissism and aloofness) correlate with some tutor assessed behaviours of individuals in year 1 and in year 2 (p < .05 to p <0.001) and, in general, the correlations are in the direction that is expected intuitively, for example narcissism and aloofness are negatively correlated with positive behaviours. The combined ITQ factor INVOLVEMENT (empathy, confidence, low narcissism, low aloofness) correlates positively with the overall tutor assessment in year 1 (p <0.05) but not in year 2. In year 2 only, social responsibility (IVQ test) predicts appropriate self-management (rather than group-orientated) behaviours (p < .05 to p < .01).

Prediction of examination results by tutor assessments

The tutor assessments focus on a range of desirable professional behaviours. Many, but not all, items predict aspects of examination performance in both year 1 and year 2, as shown in Additional file 6: Table S3 and Additional file 7: Table S4. For both years, those tutor assessment items that related to the students’ participation in the functioning of the group (for example, contribution of work to the group, and contribution to the positive atmosphere in the group) best predicted overall examination performance (including OSCE skills), and best predicted the differences between the best and worst performing students in their examinations (top 20% vs bottom 20% of marks range). Somewhat surprisingly, there are more significant relationships with Theme A scores (recall of scientific knowledge) than with Theme B (clinical techniques and skills, including OSCEs).

Prediction of examination outcomes by non-cognitive subtest scores

The non-cognitive traits that predicted examination outcomes are shown in Additional file 8: Table S5. Greater narcissism and aloofness (from ITQ), and irrational thinking (from SAI), predicted poorer overall examination scores (p < .05), and poorer performance in Theme A, Theme B overall and communication skills OSCEs. Conscientiousness (from SAI), confidence (from ITQ) and the INVOLVEMENT factor score all predicted better performance in the practical skills OSCEs (p <0.05 to p <0.001). Narcissism (p <0.05), aloofness (p < .001) and the INVOLVEMENT factor score (p < .05) each differentiated between the best and the worst performing students in the year 1 examinations. Non-cognitive test scores did not predict examination performance in Theme C.

Prediction of tutor assessments by UKCAT cognitive test scores

UKCAT cognitive tests scores did not predict many of the behaviours subsequently rated by tutors, as shown in Additional file 9: Table S6. In year 1, no behavioural items in the year 1 tutor assessments (May 2008) correlated with the overall UKCAT cognitive ability test score, and only one (negative) correlation was found with a single subtest score. Although the year 2 tutors undertook assessments twice, only eight significant correlations between the behavioural measures and cognitive test scores were found. The abstract reasoning test score predicted four behaviours in the May 2009 assessment by tutors. The UKCAT cognitive test overall score correlated with only one rated behaviour from the May 2009 tutor assessment and no behaviour from the May 2008 and January 2009 tutor assessments.

Prediction of examination outcomes by UKCAT cognitive test scores

The UKCAT cognitive test overall score and the decision analysis subtest score were significant predictors for both year 1 and year 2 overall exam scores (p < .001), and performance in Theme A (which tests knowledge acquisition) (p < .05 to p < .01), as shown in Additional file 10: Table S7. Both were also strong predictors in each year of the best versus worst examination performers. Performance in Theme C (which requires acquired knowledge and the application of numerical and analytical skills) was the result best predicted each year by both the overall cognitive ability score (p < .001) and by each subtest score. Cognitive test scores did not predict performance in the communication skills components of the clinical examinations (OSCEs) in either year; physical examination skills correlated positively only with decision analysis and only in year 1. Overall performance in theme B (patient centred-care, communication and interpersonal understanding) was predicted, but only in year 2, by three of the four cognitive test scores.

Reliability of tutor assessments

Additional file 11: Table S8 shows the re-test reliabilities of those behaviours assessed at least twice, by different tutors (year 1 tutor: May 2008; year 2 tutor: January 2009, May 2009). The reliabilities of the overall tutor ratings are high (p < .001), as are the reliabilities of the items that relate to group characteristics, such as integrates into group, demonstrates appropriate attitudes, takes responsibility for group learning, contributes work for the group, and willing to learn from others (p < .01 to p < .001).

Principal findings

This study has found numerous significant relationships between students’ prior cognitive and non-cognitive test measures (all of which became components of the UKCAT), subsequent tutor assessments of individual and group-related behaviours, and academic and clinical examination results undertaken over the first two years of a medical school course.

The individual predictive ability of any of the measures (of cognitive skills, non-cognitive traits and of behaviour) appears weak, the majority of the statistically significant correlations being in the range 0.16 to 0.24 and only a few exceeded 0.30 (accounting for only 9% of the variance). However, in the field of organisational psychology it is well recognised that even quite weak correlations are useful predictors of workplace outcomes, especially if the ratio of applicants to selectees is large [12], as with applications to medical school. Such measures may therefore usefully predict medical student performance and later professional conduct. This position is supported by a recent meta-analysis of the predictive value of ability and personality test scores, which concluded that such scores are more successful in predicting educational, work and life outcomes than is often admitted by critics; in particular “In medical education, personality characteristics gain importance for later academic performance when applied practice (such as performance in practicums and clerkships) increasingly plays a part” [13].

A further possible criticism of the findings is that there are relatively few significant relationships, as revealed by the number of blank cells in Additional file 4: Table S1 and Additional file 5: Table S2 particularly. However, of the 380 possible relationships that could have been included in these tables (19 personality scales by 20 tutor ratings), 51 (or 13.4%) were significant at the 5% level or better; this becomes 18.2% of 280, if only the 14 personality scales actually appearing in the tables are considered. When compared with the chance proportion of 5% it is clear that the overall pattern of results is better than chance. Additional analyses were conducted to adduce further evidence for the significance of the overall pattern, including multiple regression to calculate the proportion of common variance shared by the personality scales and the tutor ratings, and application of a recently published resampling technique [14] for calculating the probability of relationships between personality and behaviour. However, neither provided stronger support than the simple number of correlations, so the details are not presented here.

In summary, although the statistically significant coefficients in the matrices could all be chance findings this seems unlikely because all of the significant results (and most of the non-significant ones, which are not shown) are in the direction that the particular personality traits and cognitive skills would be expected to affect the behavioural and examination outcomes.

The findings in more detail

Overall, it appears that the non-cognitive tests do predict normally unrecorded aspects of medical students’ performance. Greater narcissism, aloofness and irrational thinking predicted lower tutor ratings for group-related behaviours, and poorer overall examination performance including in the most knowledge-based Theme A, as well as in Theme B (person-centred care), though not in Theme C. Good teamwork skills therefore appear to be an important student attribute. The finding that RESILIENCE predicted good group functioning in year 1, while conscientiousness and SELF-CONTROL predict this in year 2, may reflect students’ increasing ease and familiarity with medical school. The early years of the HYMS course involves not only problem-based learning sessions but also a structured weekly programme of lectures, laboratory-based practical classes and tutored clinical experience, so the importance of group functioning to the examination outcomes cannot be explained by exclusive reliance on problem-based group learning.

While the UKCAT cognitive tests scores did not predict many of the behaviours subsequently rated by tutors, they did predict both year 1 and year 2 overall examination scores, and performance in Theme A and Theme C. Notably, but not unexpectedly, cognitive test scores did not predict performance in the communication skills or physical examination components of the clinical examinations (OSCEs) in either year, although they did predict overall Theme B scores at the end of year 2. This may reflect increasing emphasis in year 2 on clinical reasoning skills within this theme.

Problems with predicting traditional medical school results

Prediction of any outcome depends on selection and accurate measurement of both predictor (be that a cognitive skill or a personality trait) and an appropriate outcome (for example, academic marks, skills, or professional behaviours). In order to examine whether any of the cognitive and pilot non-cognitive components of the UKCAT are valid predictors it is necessary to have appropriate measures of the behaviours that the tests are expected to predict. Paradoxically, in-course examinations do not usually assess outcomes that are related to the qualities that non-cognitive selection tests are designed to measure. In general, medical schools have concentrated on traditional examinations (in part so that licensing requirements can be fulfilled unambiguously) that test recall of factual material and reasoning, which depends on memory and cognitive skills. Non-cognitive skills, such as communication and doctor-patient relationships, are often tested informally within medical training, and the results recorded in pass/fail format, usually with only a small number of failures (as with many academic medical examinations). Thus a problem for statistical comparisons of traditional medical school results is that pass/fail clinical outcomes are categorical and typically severely skewed while the predictor test results are normally distributed on a continuous scale. The present study benefits from finer gradation in both selection tests and the outcomes measured, and from better matching between predictor and criterion variables.

Other studies

Cognitive tests would be expected to predict success in examinations of knowledge recall. This has been shown in one recent study [15] about medical students, but not in others [5–7]. No reported studies have been found that tested the hypothesis that non-cognitive characteristics desired in a good doctor have any bearing on students’ examination performance, although a Conscientiousness Index [16] has been proposed as a measure of medical students' professionalism based on a variety of routinely made behavioural observations that can be recorded in a systematic and reliable way. The present study is the first to demonstrate that measurement of a range of different personal qualities can predict different aspects of medical students’ performance. The broad range of suitable measures of non-cognitive skills and professional behaviours used here provides possible tools for future research.

Relevance of this study: implications for clinicians and policymakers

All health professionals require good communication skills and the ability to work effectively as part of a team. Doctors have additional roles, not only in education and in research but also, principally, as decision-makers and leaders of teams working in situations of clinical complexity and uncertainty. All doctors therefore should be committed to reflective practice, monitoring their own contribution and working continually to improve their own and their team’s performance [17]. Selection tests need to encompass more than purely cognitive skills, and test for the other abilities that underpin the whole range of the doctor’s professional tasks.

In the present study, non-cognitive tests were found to predict behaviours likely to be important when working as a doctor, such as functioning well with others in groups, acknowledging weaknesses and accepting feedback (which underpins the ability to learn and change through experience), and identified other traits, such as narcissism, aloofness and irrational thinking, that are likely to diminish a doctor’s ability to fulfil these roles. Improvement in the predictive power of such non-cognitive tests depends not only on finding better tests, but on devising and using better measures of critical behaviours in medical school and professional practice.

Unanswered questions and future research

The findings raise three further questions with implications for medical education. First, in the selection of future medical students, should the predictive ability of cognitive and non-cognitive qualities be explored more fully, particularly in relation to the difficult judgments and high level of inter-personal skills required of a doctor? Secondly, should medical schools be making use of more standardised and repeated behavioural observations undertaken by tutors throughout the students’ training, in addition to current measures of clinical competence (such as mini-clinical evaluation exercise [18]) used in formative and summative assessments of medical students? Thirdly, are medical schools failing to assess all the appropriate outcomes from their courses by continuing to rely too heavily on formal examinations based mainly on recall of acquired knowledge and some reasoning? The latter approach has produced many doctors who are intellectually and academically prepared for their careers, but personal failings, such as poor communication skills, lack of empathy and concern for patients, motivation, and mental health issues, tend to impinge on their work effectiveness. Such failings are typically detected too late, when brought to light by examination failure or the need for disciplinary action. If it is accepted that doctors require more than just academic knowledge and technical skills then it makes sense to look for additional qualities at the outset and select those applicants who already have these qualities or seem capable of developing them.