Research article | Open | Open Peer Review | Published:
An evaluation of the performance in the UK Royal College of Anaesthetists primary examination by UK medical school and gender
BMC Medical Educationvolume 9, Article number: 38 (2009)
There has been comparatively little consideration of the impact that the changes to undergraduate curricula might have on postgraduate academic performance. This study compares the performance of graduates by UK medical school and gender in the Multiple Choice Question (MCQ) section of the first part of the Fellowship of the Royal College of Anaesthetists (FRCA) examination.
Data from each sitting of the MCQ section of the primary FRCA examination from June 1999 to May 2008 were analysed for performance by medical school and gender.
There were 4983 attempts at the MCQ part of the examination by 3303 graduates from the 19 United Kingdom medical schools. Using the standardised overall mark minus the pass mark graduates from five medical schools performed significantly better than the mean for the group and five schools performed significantly worse than the mean for the group. Males performed significantly better than females in all aspects of the MCQ – physiology, mean difference = 3.0% (95% CI 2.3, 3.7), p < 0.001; pharmacology, mean difference = 1.7% (95% CI 1.0, 2.3), p < 0.001; physics with clinical measurement, mean difference = 3.5% (95% CI 2.8, 4.1), p < 0.001; overall mark, mean difference = 2.7% (95% CI 2.1, 3.3), p < 0.001; and standardised overall mark minus the pass mark, mean difference = 2.5% (95% CI 1.9, 3.1), p < 0.001. Graduates from three medical schools that have undergone the change from Traditional to Problem Based Learning curricula did not show any change in performance in any aspects of the MCQ pre and post curriculum change.
Graduates from each of the medical schools in the UK do show differences in performance in the MCQ section of the primary FRCA, but significant curriculum change does not lead to deterioration in post graduate examination performance. Whilst females now outnumber males taking the MCQ, they are not performing as well as the males.
Undergraduate education has undergone radical reform at most United Kingdom (UK) medical schools in the last two decades[1, 2]. Undergraduate and postgraduate medical education are now viewed as a continuum in the training of a specialist consultant or General Practitioner (GP), yet there has been comparatively little consideration of the impact which changes in style of the learning curriculum content and objectives of undergraduate education might have on postgraduate performance[3, 4]. Whilst some studies have focused on comparing competencies of graduates from traditional and Problem Based learning (PBL) curricula in the first postgraduate year[3, 5, 6], others have examined whether it is possible to predict the career path a graduate is likely to take from a particular medical school  or compared the perceived preparedness for the first postgraduate year across all UK universities. However, there have been few studies comparing the outcomes of different medical curricula beyond the first year after graduation (Foundation 1 (F1) year). A small study in 1993 showed the pass rate per medical school for the Royal College of General Practitioners examination and a more recent publication for the examinations of the Royal College of Physicians have shown differences between the pass rates of graduates from UK medical schools.
The aims of this study were to compare the performance of graduates of UK medical schools in the postgraduate examinations in anaesthesia, which is the largest hospital based speciality in UK medicine. Performance in the written sections of the first part of the Fellowship of the Royal College of Anaesthetists (FRCA) examination were analysed in relation to undergraduate medical school as well as difference according to gender and type of undergraduate curriculum.
The written part of the primary FRCA examination consists of a multiple choice question (MCQ) paper, 90 questions each with five parts (true/false format) lasting three hours, equally divided between pharmacology, physiology and biochemistry, physics and clinical measurement. The MCQ uses a negative marking system such that candidates are awarded one mark for a correct answer, minus one mark for an incorrect answer and no marks if the question is not answered. If candidates are successful in the written section of this examination they progress to an oral section which examines the same clinical subjects in more depth as well as resuscitation, anatomy, history-taking, physical examination and communication skills. The final part of the FRCA, which is generally taken 2 years after the primary, assesses clinical aspects of anaesthesia and intensive care medicine.
Ethical approval for this study was sought and gained by the Royal College of Anaesthetists (RCA) and the University of Liverpool School of Medical Education Research Ethics Committee. The data for analysis were collated from RCA spreadsheets for each sitting of the primary FRCA examination from June 1999 to May 2008 and from the main trainee database of the RCA, using the RCA registration number as the unique identifier. All data were anonymous. Only those candidates who had qualified from one of the 19 UK universities who were awarding medical degrees during this period were included. Only the university awarding the degree was recorded; as the medical schools in London are awarded a University of London degree all are recorded as "London".
Data available for analysis included the percentage mark for each section of the examination for each candidate as well as their overall final percentage mark, their closed mark for each part of the examination, RCA reference number, year and university of primary medical degree, date of birth and gender. Up to June 2007 the pass mark for each sitting of the MCQ was determined by norm-referencing; it was set 1% below the mean score for that sitting but dependent on the cohort ability. The ability of the cohort was determined by the discriminator ratio (DR): the ratio of the sum of the scores on discriminator questions in the current examination to the sum of the scores on the discriminator questions the last time that discriminator was used. The DR should be between 0.95 and 1.05 and if it was outside these limits then the cohort was considered to be significantly less or more able than average respectively. When a significant difference existed, an adjustment was made to the banding for the pass mark by placing it either closer to or further from the mean. The pass marks ranged from 46.4% to 59.4%. Since June 2007 this system has been complemented by Angoff Referencing which has made little difference to the pass marks. To aid the standardisation of results over the timescale of the data the difference between each candidate's overall mark and the pass mark for that sitting of the MCQ was determined. A candidate whose overall mark equalled the pass mark would therefore have a score of zero and a positive or negative score indicated the degree to which the candidate had passed or failed the examination.
Data were analysed using SPSS 16.0. Comparison between groups was undertaken with independent sample t-tests or non parametric tests where appropriate, with p < 0.05 being considered significant. Data is presented as mean values with 95% confidence intervals unless otherwise stated.
From the June 1999 to the May 2008 sittings of the primary FRCA, there were 4983 attempts at the MCQ part of the examination by 3303 graduates from the 19 United Kingdom medical schools. Of the 4983 attempts at the primary MCQ, 1080 (21.7%) were poor fails and 3903 (78.3%) were able to progress to the oral section of the examination. There were 4056 attempts at the oral section of the examination with 2793 (68.9%) passing the examination (there is a discrepancy between this number of attempts and the 3903 who progressed from the MCQ; since June 2007 the MCQ has became a separate examination and candidates can make multiple attempts at the oral section without retaking the MCQ). The overall pass rate the examination was 56.1%. The median year of qualification was 1999 (range 1959 to 2006) with the median time from qualification to sitting the MCQ being 4.6 years (interquartile range 3.8 – 5.7). Overall 44.8% (medical school range 34 to 56%) of the trainees sitting the examination were female (see Additional file 1).
Medical school effects
The performance of each medical school for each section of the MCQ, including the overall final mark and the standardised percentage mark above or below the pass mark is shown in Additional file 1. The effect of medical school on the percentage mark above or below the pass mark is shown graphically in figure 1. Five medical schools performed significantly better than the mean for the group – Oxford, Cambridge, Edinburgh, Bristol and Newcastle-upon-Tyne. Five schools performed significantly worse than the mean for the group – Sheffield, Aberdeen, Leicester, Dundee and Belfast. The likelihood of passing the primary FRCA examination at the first attempt for graduates of each medical school is shown in Table 1, with a mean first time pass rate of 62.2% (95% CI 60.5, 63.8) for the whole group (range 22.2% to 92.6%). Five medical schools had first time pass rates significantly better than the mean for the group – Oxford, Cambridge, Nottingham, Birmingham and Edinburgh; two medical schools first time pass rates were significantly worse than the mean for the group – Belfast and Dundee.
Medical school curricula are constantly changing, and it has not been possible in this study to look for an effect of changes in balance of the Basic Science Curriculum. We did however look at the results in three schools where the curriculum changed to PBL. We selected this curriculum change not out of any belief that it is likely to be relevant but because this aspect of curriculum design has previously attracted attention [3, 5, 6]. Three medical schools have undertaken the change from traditional curricula to PBL curricula; Manchester in 1994, Liverpool and Glasgow both in 1996. Assuming that each course takes 5 years then the first graduates from these new curricula were awarded their degrees in 1999 and 2001 respectively. Graduates of these medical schools pre and post curricula change did not show any change in performance in any aspects of the MCQ. For the mean standardised percentage above and below pass mark (%): Manchester graduates pre and post 1999 = 4.1 (95% CI 2.9, 5.4) vs 2.8 (95% CI 1.4, 4.2), p = 0.17; Liverpool graduates pre and post 2001 = 3.5 (95% CI 0.03, 6.9) vs 3.8 (95% CI 2.3, 5.3), p = 0.86; Glasgow graduates pre and post 2001 = 2.2 (95% CI 0.9, 3.4) vs 0.6 (95%CI -1.8, 3.1), p = 0.23.
Males performed significantly better in all aspects of the MCQ. For physiology, mean difference = 3.0% (95% CI 2.3, 3.7), p < 0.001; pharmacology, mean difference = 1.7% (95% CI 1.0, 2.3), p < 0.001; physics with clinical measurement, mean difference = 3.5% (95% CI 2.8, 4.1), p < 0.001; overall mark, mean difference = 2.7% (95% CI 2.1, 3.3), p < 0.001; and standardised overall mark minus the pass mark, mean difference = 2.5% (95% CI 1.9, 3.1), p < 0.001. The overall pass rate for the examination was 53.5% for females and 58.2% for males (Chi squared = 11.2, p = 0.001). The variation in gender performance by medical school is shown in figure 2 with the difference in performance by males being significant in nine medical schools. The performance in the three parts of the MCQ, and hence overall performance, declined for both sexes from 1999 to 2008, with females showing a greater decline than males (figure 3). Overall females made 2288 attempts at the MCQ with 1710 (74.7%) progressing to the oral section of the examination; this is compared to 2193 of 2695 (81.4%) attempts for males (Chi squared = 31.7, p < 0.001). Of those female candidates who undertook the oral section of the examination 1224 of 1837 (66.6%) attempts passed; for males, 1569 of 2219 (70.7%) attempts led to a pass (Chi squared = 7.6, p = 0.006). The likelihood of each gender passing the primary examination at the first attempt is shown in Table 1, with the first time pass rate for males being 63.9% (95% CI 61.6, 66.1) compared to 60.0% (95% CI 57.4, 62.5) for females (Chi squared = 4.99, p = 0.025).
Correlations between parts of the MCQ
There was a highly significant correlation between the 3 parts of the MCQ; physiology with pharmacology, r = 0.69, n = 4893; pharmacology with physics and clinical measurement, r = 0.65, n = 4893; physiology and clinical measurement, r = 0.63, n = 4893.
The results do show that there are differences between medical schools in the UK on graduate performance in the FRCA examination. A graduate from Oxford, Cambridge, Birmingham, Nottingham and Edinburgh is more likely to pass the primary FRCA at the first attempt than a graduate from Belfast or Dundee. These results are in some agreement with the medical school graduate performance in the MRCP examination with Oxford, Cambridge, Edinburgh and Newcastle upon Tyne being in the top quartile and Belfast, Dundee, Aberdeen in the bottom quartile of both series.
A study at the University of Liverpool has shown that 5–6 years after graduation, trainees do feel their undergraduate education does impact on their competencies as doctors[10, 11] and are able to relate their experiences to their undergraduate education, suggesting that undergraduate education has some impact on post graduate performance. In addition there is some correlation with these results and the rankings in the 2008 Times survey grading medical schools with the top quartile medical schools (Oxford, Cambridge, Edinburgh, Bristol and Newcastle) being ranked 1,2,3 = 9th and 16th respectively and the bottom quartile medical schools (Sheffield, Aberdeen, Leicester, Dundee and Belfast) being ranked 19th, 5th, 18th, 22nd and 20th respectively. However, the performance in post graduate examination does not correlate with career choice as those medical school graduates who are most likely to pursue a career in anaesthesia are from Bristol, Edinburgh and Southampton whilst Oxford and Cambridge graduates are less likely to make that career choice.
Female graduates are underperforming in both the MCQ and oral parts of the primary FRCA examination compared to male graduates, and are less likely to pass the examination at the first attempt. The underperformance by females has also been found in Parts 1 and 2 of the Membership of the Royal College of Physicians examination (MRCP(UK)), both of which are MCQ type examinations, however they performed better than males in the clinical assessment section (PACES) of the examination. This is in contrast to performance at medical school where females tend to outperform males[15, 16] and are more likely to be awarded an honours degree. In school science examinations male students have historically outperformed female school students although this has recently been reversed. One reason suggested for this change is that the assessment system in schools now favours female students compared to male students. The revision strategy of male students of 'cramming-it-all-in-at-the-last-minute' is perhaps less beneficial for assessment throughout the year for modular examinations and coursework, but the "cramming" approach may be best suited for final summative assessments such as postgraduate medical examinations. In 1974 females comprised approximately 27% of those qualifying from medical school, but by 1991 50% of the medical student population were female and by 2005 constituted 61% of the undergraduate student population. So after years of underrepresentation, females now outnumber males in the medical workforce in the UK: Although overall 44.8% of candidates were female in this study their representation increased for each year of qualification, such that for those qualifying after 2002 58% were female; this is similar to the 58% of doctors who were female who responded to a BMA survey in 2006. The primary FRCA examination format has not changed during the period of the study, however other factors may affect gender performance such as part time training, which may lead to difficulties with career progression although this has been disputed, and the stresses involved in achieving a satisfactory work-life balance. However, there are concerns that negative marking, which is intended to correct for guessing, may discriminate between students on their risk taking behaviour and as female students are less likely to take risks this could lead to gender bias. The RCA is aware of these concerns and negative marking has been replaced by number-right scoring in all its examinations from September 2008.
The Universities of Manchester, Glasgow and Liverpool introduced integrated problem-based learning curricula replacing their traditional lecture based courses in 1994 and 1996 respectively. There have been concerns expressed about these recent reforms  in UK medical education and fears that moving to a PBL system may have a negative impact on the basic science knowledge of PBL graduates which may impact on the ability of PBL to pass science based postgraduate exams[28, 29]. These results suggest that reforming a medical curriculum from a traditional course to an integrated PBL in the UK does not impact on the ability of graduates to undertake science based post graduate examinations; this is confirmed by results from studies in North America[30–32] which have also shown there are no significant differences between PBL and traditional graduates on their licensing exams. Although there is variation in the content of medical school curricula in the UK, all medical schools have to incorporate the recommendations of Tomorrow's Doctorsinto their curricula but this data confirms that some elements of performance do not change.
There was a highly significant correlation between the 3 parts of the MCQ. It would be expected that all candidates would have prior knowledge of physiology and pharmacology, but it is unlikely that any of the candidates will have been taught and examined on physics with clinical measurement. This therefore tests their ability to attain and apply new knowledge and it would be expected that students who do well in this area would also be better able to apply previous physiology or pharmacology knowledge from their undergraduate education.
There are some limitations to the study. Only a small percentage of medical graduates will take the anaesthesia examinations, therefore performance in these examinations cannot be extrapolated to all post graduate examinations. Although the recent introduction of Modernising Medical Careers (MMC) for postgraduate medical education has resulted in some reform of medical training in the UK, this will have had little impact on the results of this study as the pre and post MMC trainees have experienced similar training pathways. However, the exposure of undergraduates to anaesthesia will vary between medical schools; for instance, with the advent of student selected components into undergraduate curricula students can voluntarily undertake an undergraduate attachment in anaesthesia. In addition MMC has allowed foundation doctors the opportunity to take a short rotation in anaesthesia, or a related specialty such as intensive care medicine; exposure that would more likely lead to a career in the speciality, but would not necessarily lead to an improvement in overall results. A confounding factor may be that even if graduates remain close to their medical school where they qualified the differences in performance may rather reflect the quality of postgraduate course available for anaesthesia trainees in that locality.
These results show that graduates from each of the medical schools in the UK do have differences in performance in the MCQ part of the primary FRCA. Whilst females now outnumber males taking the examination, it is concerning that females do not seem to be performing as well as males.
British Medical Association
Fellowship of the Royal College of Anaesthetists
General Medical Council
Multiple Choice Questions
Modernising Medical Careers
Membership of the Royal College of Physicians
Objectively Structured Clinical Examination
Problem Based learning
Royal College of Anaesthetists
Structured oral Examination
Jones R, Higgs R, de Angelis C, Prideaux D: Changing face of medical curricula. Lancet. 2001, 357 (9257): 699-703. 10.1016/S0140-6736(00)04134-9.
Bligh J, Lloyd-Jones G, Smith G: Early effects of a new problem-based clinically oriented curriculum on students' perceptions of teaching. Med Educ. 2000, 34 (6): 487-489. 10.1046/j.1365-2923.2000.00447.x.
Jones A, McArdle PJ, O'Neill PA: Perceptions of how well graduates are prepared for the role of pre-registration house officer: a comparison of outcomes from a traditional and an integrated PBL curriculum. Med Educ. 2002, 36 (1): 16-25. 10.1046/j.1365-2923.2002.01105.x.
McManus IC, Elder AT, de Champlain A, Dacre JE, Mollon J, Chis L: Graduates of different UK medical schools show substantial differences in performance on MRCP(UK) Part 1, Part 2 and PACES examinations. BMC Med. 2008, 6 (1): 5-10.1186/1741-7015-6-5.
Watmough S, Ryland I, Taylor DC: Preregistration house officer skill and competency assessment through questionnaires. Br J Hosp Med (Lond). 2006, 67 (9): 487-490.
Watmough S, Taylor DC, Garden A, Ryland I: Educational supervisors' views on the competencies of preregistration house officers. Br J Hosp Med (Lond). 2006, 67 (2): 92-95.
Goldacre MJ, Turner G, Lambert TW: Variation by medical school in career choices of UK graduates of 1999 and 2000. Med Educ. 2004, 38 (3): 249-258. 10.1046/j.1365-2923.2004.01763.x.
Cave J, Goldacre M, Lambert T, Woolf K, Jones A, Dacre J: Newly qualified doctors' views about whether their medical school had trained them well: questionnaire surveys. BMC Med Educ. 2007, 7: 38-10.1186/1472-6920-7-38.
Wakeford R, Foulkes J, McManus C, Southgate L: MRCGP pass rate by medical school and region of postgraduate training. Royal College of General Practitioners. BMJ. 1993, 307 (6903): 542-543. 10.1136/bmj.307.6903.542-a.
Watmough S, Taylor D, O'Sullivan H: PBL graduates assess their undergraduate medical education 6 years after graduation through interviews. AMEE 2008 Annual Conference. 2008, Prague: AMEE, 68.
Watmough S, Taylor D, Ryland I: Using questionnaires to determine whether medical graduates' career choice is determined by undergraduate or postgraduate experiences. Med Teach. 2007, 29 (8): 830-832. 10.1080/01421590701551755.
Good University guide 2008. accessed 6/8/08, [http://extras.timesonline.co.uk/gug/gooduniversityguide.php?sort=TOTAL&subject=MEDICINE]
Turner G, Goldacre MJ, Lambert T, Sear JW: Career choices for anaesthesia: national surveys of graduates of 1974–2002 from UK medical schools. Br J Anaesth. 2005, 95 (3): 332-338. 10.1093/bja/aei191.
Dewhurst NG, McManus C, Mollon J, Dacre JE, Vale AJ: Performance in the MRCP(UK) Examination 2003–4: analysis of pass rates of UK graduates in relation to self-declared ethnicity and gender. BMC Med. 2007, 5: 8-10.1186/1741-7015-5-8.
Haq I, Higham J, Morris R, Dacre J: Effect of ethnicity and gender on performance in undergraduate medical examinations. Med Educ. 2005, 39 (11): 1126-1128. 10.1111/j.1365-2929.2005.02319.x.
Cuddy MM, Swanson DB, Clauser BE: A Multilevel analysis of the relationships between examinee gender and United States Medical Licensing Exam (USMLE) step 2 CK content area performance. Acad Med. 2007, 82 (10 Suppl): S89-93. 10.1097/ACM.0b013e3181400379.
Ferguson E, James D, Madeley L: Factors associated with success in medical school: systematic review of the literature. BMJ. 2002, 324 (7343): 952-957. 10.1136/bmj.324.7343.952.
Lipsett A: Girls eclipse boys in tests at 14. Guardian. 2007, London: Guardian News and Media Limited
BBC: Are boys underachieving at school?. Accessed 11/8/08, [http://www.bbc.co.uk/schools/parents/life/sen/boys_and_girls/boys_underachievement_facts.shtml]
Allen I: Women doctors and their careers: what now?. BMJ. 2005, 331 (7516): 569-572. 10.1136/bmj.331.7516.569.
Federation of Royal Colleges of Physicians: Women in hospital medicine – career choices and opportunities. A report of a working party of the Federation of Royal Colleges of Physicians. London. 2001
British Medical Association: Cohort study 2006 medical graduates – first report. 2007, London: BMA
McKinstry B: Are there too many female medical graduates? Yes. BMJ. 2008, 336 (7647): 748-10.1136/bmj.39505.491065.94.
Dacre J: Are there too many female medical graduates? No. BMJ. 2008, 336 (7647): 749-10.1136/bmj.39505.566701.94.
Fowell S, Jolly B: Combining marks, scores and grades. Reviewing common practices reveals some bad habits. Med Educ. 2000, 34 (10): 785-786. 10.1046/j.1365-2923.2000.00796.x.
Eckel CC, Grossman PJ: Forecasting risk attitudes: An experimental study using actual and forecast gamble choices. Journal of Economic Behavior & Organization. 2008, 68 (1): 1-17. 10.1016/j.jebo.2008.04.006.
Williams G, Lau A: Reform of undergraduate medical teaching in the United Kingdom: a triumph of evangelism over common sense. BMJ. 2004, 329 (7457): 92-94. 10.1136/bmj.329.7457.92.
Koh GC, Khoo HE, Wong ML, Koh D: The effects of problem-based learning during medical school on physician competency: a systematic review. CMAJ. 2008, 178 (1): 34-41.
Watmough S, Taylor D, Garden A: Educational supervisors evaluate the preparedness of graduates from a reformed UK curriculum to work as pre-registration house officers (PRHOs): a qualitative study. Med Educ. 2006, 40 (10): 995-1001. 10.1111/j.1365-2929.2006.02563.x.
Kaufman DM, Mann KV: Basic sciences in problem-based learning and conventional curricula: students' attitudes. Med Educ. 1997, 31 (3): 177-180. 10.1111/j.1365-2923.1997.tb02562.x.
Kaufman DM, Mann KV: Comparing achievement on the Medical Council of Canada Qualifying Examination Part I of students in conventional and problem-based learning curricula. Acad Med. 1998, 73 (11): 1211-1213. 10.1097/00001888-199811000-00022.
Kaufman DM, Mann KV: Achievement of Students in a Conventional and Problem-Based Learning (PBL) Curriculum. Adv Health Sci Educ Theory Pract. 1999, 4 (3): 245-260. 10.1023/A:1009829831978.
General Medical Council: Tomorrow's Doctors. 2003, London: General Medical Council
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/9/38/prepub
The authors declare that they have no competing interests.
AB conceived and designed the study, analysed and interpreted the data, drafted the article and approved the final version to be published. SW conceived and designed the study, drafted the article and approved the final version to be published.