This article has Open Peer Review reports available.
Unhappiness and dissatisfaction in doctors cannot be predicted by selectors from medical school application forms: A prospective, longitudinal study
© McManus et al; licensee BioMed Central Ltd. 2005
Received: 14 August 2005
Accepted: 13 December 2005
Published: 13 December 2005
Personal statements and referees' reports are widely used on medical school application forms, particularly in the UK, to assess the suitability of candidates for a career in medicine. However there are few studies which assess the validity of such information for predicting unhappiness or dissatisfaction with a career in medicine. Here we combine data from a long-term prospective study of medical student selection and training, with an experimental approach in which a large number of assessors used a paired comparison technique to predict outcome.
Data from a large-scale prospective study of students applying to UK medical schools in 1990 were used to identify 40 pairs of doctors, matched by sex, for whom personal statements and referees' reports were available, and who in a 2002/3 follow-up study, one pair member was very satisfied and the other very dissatisfied with medicine as a career. In 2005, 96 assessors, who were experienced medical school selectors, doctors, medical students or psychology students, used information from the doctors' original applications to judge which member of each pair of doctors was the happier, more satisfied doctor.
None of the groups of assessors were significantly different from chance expectations in using applicants' personal statements and the referees' reports to predict actual future satisfaction or dissatisfaction, the distribution being similar to binomial expectations. However judgements of pairs of application forms from pairs of doctors showed a non-binomial distribution, indicating consensus among assessors as to which doctor would be the happy doctor (although the consensus was wrong in half the cases). Assessors taking longer to do the task concurred more. Consensus judgements seem mainly to be based on referees' predictions of academic achievement (even though academic achievement is not actually a valid predictor of happiness or satisfaction).
Although widely used in medical student selection to assess motivation, interest and commitment to a medical career, the personal statement and the referee's report cannot validly be used by assessors, including experienced medical school selectors, to identify doctors who will subsequently be dissatisfied with a medical career.
Many doctors in the UK, and probably elesewhere, are unhappy with their careers in medicine, with a fifth or more of junior doctors considering leaving medicine . Medical school selectors do not wish only to select academically able students who can cope academically with the medical course, but also want to assess motivation for studying medicine, becoming a doctor, and for a lifetime of practising medicine. An important question, therefore, is whether the information available during student selection can be used to predict which doctors will be happy or unhappy with a career in medicine.
Applicants to UK medical schools apply by means of the UCAS (previously UCCA) form, which contains standard demographic data, information on educational achievement and qualifications, a personal statement from the applicant, and a report from a referee . Empirically, educational achievement and intellectual ability have been found to be unrelated to stress, burnout and dissatisfaction with medicine , and therefore the most likely source for information which might be able to predict happiness with a medical career is the applicants' personal statements and their referees' reports.
Experimental research manipulating the informational content of personal statements, resumes and cover letters shows that applicants are judged as more competent and to have greater potential when the resumes contained relevant educational references , whereas trying to present oneself in a positive light was not always beneficial [4, 5]. Although informative, such studies do not address the actual predictive utility of personal statements, and whether selectors can identify those applicants who will eventually be successful or happy in their career, particularly when UCAS/UCCA personal statements are not structured to help selectors make their judgments.
In this study we used a simple experimental design to assess the ability of assessors to distinguish pairs of doctors who had been qualified for five or six years, one of whom who was very happy and satisfied with their medical career and the other was very unhappy and dissatisfied, using only the personal statement and referee's report submitted as a part of the doctors' UCCA application forms to medical school, twelve years previously.
This study was approved under the normal procedures of the Ethics Committee of the UCL Department of Psychology. The study was exempt from the need for specific permission by the UCL Research Ethics Committee (see http://www.grad.ucl.ac.uk/ethics/index.php; http://www.grad.ucl.ac.uk/ethics/exemptions.php).
The study used a paired comparison (two-alternative, forced-choice) design in which a group of assessors in 2005 decided which of two doctors, one very happy and one very unhappy, was the happier doctor (assessed in 2002/3), based on the personal statement and referee's report from the application forms the doctors had submitted to UCCA in 1990.
The doctors and their application forms
Responses of happy and unhappy doctors.
How often do the following statements describe the way you feel about working as a doctor?
A few times a week
Once a week
A few times a month
Once a month or less
A few times a year
a) I think of giving up medicine for another career
b) I reflect on the satisfaction that I get from being a doctor
c) I regret my decision to have become a doctor
The assessors made their judgements in questionnaire booklets which contained twenty pairs of UCCA forms, with each side of a double-page spread showing the personal statement and referee's report from a satisfied and a dissatisfied doctor, randomised to left (labelled A) or right (labelled B). Pairs were matched for sex, and were from applicants who were UK nationals aged under 21 when they applied to medical school. All forms were transcribed, checked against the originals, fully anonymised and printed in a standard format. For each pair of forms, there was a question at the bottom of the page: "Which is the satisfied, happy doctor? Definitely A/Probably A/Probably B/Definitely B". For the main statistical analysis, judgements of 'definitely' and 'probably' were combined; for correlations with the use of 'definite' judgements, see Additional File 1. Two separate booklets were assembled, Book 1 and Book 2, each containing 20 different pairs of doctors. Overall therefore, forms from 80 doctors were used, 40 being very satisfied and 40 very dissatisfied, but, in order not to overload assessors, each individual assessor saw only one booklet containing 20 pairs of forms from a total of 40 doctors. It should be emphasised that we made no attempt to differentiate 'happiness' and 'satisfaction', and for the purposes of this paper these words should be regarded as synonyms.
The pairs of forms were assessed by four different groups of assessors: 35 experienced medical school selectors (recruited with the assistance of the CHMS Admissions to Medical Schools Group), a convenience sample of 19 doctors known to SI, 22 medical students from UCL, and 20 psychology students from UCL (for details see Additional File 1). The student groups were included because of studies suggesting that students can sometimes be better assessors than experienced judges . After judging the pairs of forms, assessors provided brief data on age, sex, experience of medicine and student selection, and how long the task had taken, and they completed brief assessments of their own personality (Big Five, empathy, and communicative ability).
Note: There is much potential for confusion in understanding the results of this study. In particular we would like to emphasise that when we refer to assessors we are referring to the individuals making judgements of the forms in 2005, and when we refer to doctors (and to pairs of doctors) we are referring to the individuals who applied to medical school in 1990, and whose UCCA forms were used in this study, and to the questionnaire responses made by the same individuals as practising doctors in 2002/3. In particular it should be noted that both the doctors and the assessors each completed a Big Five personality assessment, and each is analysed below.
The pairs of application forms were judged by 96 assessors, 48 of whom looked at Book 1 and 48 looked at Book 2. A chance null hypothesis predicts that random guessing would result in an assessor making an average of 10 correct judgements out of 20 (50%). In fact overall, of the 1920 judgements made by the 96 judges, 963 (50.2%) were correct, which is not significantly different from 50% (χ2 = .018, 1df, P = .89).
Figure 3 confirms that assessors cannot be making random judgements, and that for many of the pairs of doctors there is a clear agreement between the assessors as to which doctor will be the happier (even though that judgement is erroneous for half of the pairs of doctors). A 'consensus score' was calculated to assess how often an individual assessor had concurred with the consensus judgement on the 21 pairs where on aggregate across all assessors there was agreement beyond chance expectations. Eighty-four of the assessors also reported how long they had taken to carry out the task (mean 80.8 minutes; SD 42.6 minutes; median = 60 minutes; range 30–240 minutes). Assessors who had taken longer to carry out the task were less likely to be correct overall (Pearson's r = -.289, p = .008) and more likely to concur with other judges (r = .274, p = .012), both correlations with time remaining significant after partialling out the other measure.
It is clear that assessors are unable to carry out the task beyond chance expectations, but they nevertheless concur with one another in their judgements. The question therefore arises as to the nature of the information from the forms which is acting as the basis for the assessors' (erroneous) consensus. We had background measures on the doctors in the pairs and we therefore looked to see whether those measures predicted the judgements of the assessors. It should be emphasised here that we are not looking here at correlations between the actual happiness of doctors and the background measures, but rather between those same background measures and which doctors are thought more likely to be the happier by the assessors. Although personality is known to be a strong correlate of happiness and satisfaction in doctors [3, 7], there was no correlation between the difference in Big Five personality scores of the two doctors in each pair and the consensus judgement of the assessors (be it right or wrong). Assessors are not therefore implicitly assessing aspects of personality. We also had measures of educational achievement in the pairs of doctors, including O-level/GCSE grades, A-level grades, and expected A-level grades in those applying pre-A-level. The difference in educational achievement between the doctors in a pair correlated significantly with the consensus judgement of the assessors, with the largest correlation being with expected A-level grades (r = .321, p = .011, N = 62). Since expected A-level grades, and other aspects of educational achievement, are often referred to in referees' reports, it seems likely that assessors are judging that the doctor with the higher, predicted educational achievement will be the happier, more satisfied doctor.
UK medical schools put extensive effort into reading the application forms submitted by applicants, using them as a basis for shortlisting for interview (and in some cases for making offers), and assessing both academic achievement and motivation from them. However the present study suggests that assessors, including experienced medical school selectors, are unable to use application forms to judge which applicants will become happy and satisfied doctors. That does not mean, though, that selectors are making random judgements on the basis of the forms. There is a consensus between assessors, which is greatest amongst those who have spent longest reading the forms. However that consensus seems principally to result from using information about the academic potential of applicants, despite there being good reason to believe that it is not a lack of academic ability which causes the dissatisfaction of unhappy doctors, or that academic ability correlates with happiness . In view of the implications of the present findings for current methods of student selection in the UK (and elsewhere), it would be reassuring if other studies, in other settings, and with different outcome criteria, were to find similar results, and we hope that such studies will be carried out.
The implicit use of academic ability as a criterion is consistent with other work , which has shown that when raters used a structured pro-forma to assess the presence of various attributes in a resume (e.g., academic ability, social exchanges), then those ratings are related to applicants personality and cognitive abilities . However, these results consider judgements based on pre-defined categories, whereas personal statements are often judged without such a standardized approach – as in UCCA and UCAS forms – and therefore the predictive validity of personal statements for identifying 'at risk' candidates is low. Even should structured personal statement coding be correlated with personality, it might still be more effective to use psychometrically assessed personality scores than personal statements, if it were desired to select on the basis of personality. Such conclusions are, of course, predicated on the basis that selectors use personal statements to identify qualities that mark whether a candidate will be a happy and good doctor. The analyses reported here suggest, though, that academic criteria are still relied on as the main differentiator, which may reflect social and cultural stereotypes of what makes a good doctor.
The overall outcome measure in our study, of satisfaction and happiness with medicine as a career, might be criticised on the grounds that what is really required is knowledgeable, effective, doctors who stay in post. If knowledgeable, academically well-qualified doctors alone is all that is required, then probably the best predictor of academic success is educational achievement [3, 10], but that is assessed most efficiently, effectively and objectively from measures of educational achievement, rather than indirectly from a referee's report. Academic ability and career satisfaction are not however correlated . There is however good evidence that our unhappy doctors are stressed and burned out (see Additional File 1), that stress and burnout result in less effective work in doctors , and that dissatisfied, unhappy doctors are the ones who are most likely to want to leave medicine . Satisfaction is therefore an important outcome variable.
Perhaps the most surprising thing about our study is that the task was completely impossible. That might reflect the relative poverty of the information provided on the application forms (for the personal statement the median length is 107 words (mean = 106, SD 22, quartiles 92–120, range = 54–162), and for referees' reports the median length is 398 words (mean = 391, SD 111; quartiles 325–465; range 114–734)). The information provided might also be inaccurate or misleading, either unintentionally, or as part of a process of 'impression management' . Referees have an obligation to the medical school, but also to the person on whose behalf they are making a statement, perhaps ignoring their weaknesses and over-emphasising their strengths, so that some geese appear as swans. It is therefore possible that information obtained differently might be more effective in prediction. The UCCA forms completed in 1990 are now somewhat different from those currently used, and it is possible that modern forms might provide more information which is more useful. However several selectors spontaneously commented that they were surprised by the honesty and the openness of the referees' reports made in 1990, and that more modern statements seemed bland, non-committal and uninformative in comparison. Our results are therefore likely to be valid for modern application forms.
We had wondered whether there may be further information about applicants concealed within the subtleties and the nuances of the language used (e.g. the use of negative versus positive emotion words). We therefore used a computerised text analysis program, LIWC [12, 13], to assess personal statements and referees' reports on about 70 semantic, syntactic and stylistic measures, but could find no systematic difference between UCCA forms from happy and unhappy doctors. Nevertheless it is possible that there is additional information within the specific content of the forms, which could be indirect indicators of personality, and in particular neuroticism, which is itself a major predictor of stress and burnout.
The Schwartz Report into admissions to universities, was critical of the use of selection methods for which there was not demonstrable evidence of reliability or validity . UCCA/UCAS forms, along with A-level grades, are the single most important piece of evidence used in medical school selection, each being carefully read and assessed, and often they are the sole basis for rejection. Applicants, their parents, and their schools therefore put much effort into every detail of the forms. Our study is an assessment of the long-term validity of judgements made from the information on UCCA/UCAS forms. Judgements made are to some extent reliable, as seen by the consensus between assessors (and shown elsewhere ). However although many claims are made for the utility of the personal and referee's information provided on the forms, we could find no evidence of long-term predictive validity for an important outcome variable – the judgement of whether or not an applicant will be a happy and satisfied doctor, or instead will be an unhappy, stressed, burned out, dissatisfied doctor who does not enjoy their job and thinks often of leaving for another career.
We are grateful to Dr Patricia Hughes, Chair of the CHMS Admissions to Medical Schools Group, for her assistance in distributing the questionnaire booklets to medical school selectors, and to the selectors and other assessors who took part in the study. The 2002 follow-up of the 1991 cohort, and subsequent analyses, were funded by the London Deanery.
- Moss PJ, Lambert TW, Goldacre MJ, Lee P: Reasons for considering leaving UK medicine: questionnaire study of junior doctors' comments. Brit Med J. 2004, 329: 1263-1265.View ArticleGoogle Scholar
- Universities and Colleges Admissions Service website. [http://www.ucas.ac.uk]
- McManus IC, Smithers E, Partridge P, Keeling A, Fleming PR: A levels and intelligence as predictors of medical careers in UK doctors: 20 year prospective study. Brit Med J. 2003, 327: 139-142.View ArticleGoogle Scholar
- Knouse SB: Impressions of the resume: the effects of applicant education, experience, and impression management. Journal of Business and Psychology. 1994, 9: 33-45. 10.1007/BF02230985.View ArticleGoogle Scholar
- Knouse SB: Impression management in the resume and its cover letter. Journal of Business and Psychology. 1994, 3: 242-249. 10.1007/BF01014492.View ArticleGoogle Scholar
- McManus IC, Richards P, Winder BC, Sproston KA, Styles V: Medical school applicants from ethnic minorities: identifying if and when they are disadvantaged. Brit Med J. 1995, 310: 496-500.View ArticleGoogle Scholar
- McManus IC, Keeling A, Paice E: Stress, burnout and doctors' attitudes to work are determined by personality and learning style: A twelve year longitudinal study of UK medical graduates. BMC Medicine. 2004, 2: 29-10.1186/1741-7015-2-29.View ArticleGoogle Scholar
- Kocsis R: Profiling the criminal mind: does it actually work?. Lancet. 2004, 364: s14-s15. 10.1016/S0140-6736(04)17623-X.View ArticleGoogle Scholar
- Cole MS, Field HS, Giles WF: Using recruiter assessments of apploicants' resume content to predict applicant mental ability and big five personality dimensions. ijsa. 2003, 11: 78-88.Google Scholar
- McManus IC, Powis D, Ferguson E, James D, Richards P: Selecting medical students: Why intellectual aptitude tests probably aren't useful for selecting UK school-leaver entrants, and how A-levels could be. Brit Med J. 2005.Google Scholar
- Winefield HR: Work stress and its effects in general practitioners. Occupational stress in the service professions. Edited by: Dollard MF, Winefield AH, Winefield HR. 2003, London: Taylor and Francis, 191-212.View ArticleGoogle Scholar
- Pennebaker JW, Francis ME, Booth RJ: Linguistic Inquiry and Word Count (LIWC): LIWC 2001. 2001, Mahwah, NJ: Lawrence ErlbaumGoogle Scholar
- Pennebaker JW, Mehl MR, Niederhoffer KG: Psychological aspects of natural language use: our words, our selves. ARP. 2003, 54: 547-577.Google Scholar
- Admissions to Higher Education Steering Group: Fair admissions to higher education: recommendations for good practice. 2004, Nottingham: Dept for Education and Skills Publications, [http://www.admissions-review.org.uk]Google Scholar
- McManus IC, Richards P: Reliability of short-listing in medical student selection. Med Educ. 1989, 23: 147-151.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/5/38/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.