Study of performance on a novel paediatric examination to investigate disparity between what undergraduate medical students are taught and what is felt to be essential knowledge in the postgraduate paediatric domain?

It is recognised that newly qualied doctors feel unprepared in many areas of their daily practice and that there is a gap between what students learn during medical school and their clinical responsibilities early in their postgraduate career. This study aimed to assess if undergraduate students and junior paediatric doctors met a Minimum Accepted Competency (MAC) of knowledge. The knowledge of undergraduates and junior paediatric doctors was quantitatively assessed by their performance on a 30-item examination. The items within this examination were designed by non-academic consultants to test ‘must-know’ knowledge for starting work in paediatrics. The performance of the students was compared with their ocial university examination results and with the performance of the junior doctors.


Abstract
Background It is recognised that newly quali ed doctors feel unprepared in many areas of their daily practice and that there is a gap between what students learn during medical school and their clinical responsibilities early in their postgraduate career. This study aimed to assess if undergraduate students and junior paediatric doctors met a Minimum Accepted Competency (MAC) of knowledge.

Methods
The knowledge of undergraduates and junior paediatric doctors was quantitatively assessed by their performance on a 30-item examination. The items within this examination were designed by nonacademic consultants to test 'must-know' knowledge for starting work in paediatrics. The performance of the students was compared with their o cial university examination results and with the performance of the junior doctors.

Results
There were a total of 478 participants. Mean examination score was 45.9% for students and 64.2% for doctors (signi cantly higher [p<0.01]). A signi cantly reduced number of students passed the examination compared with their o cial university examinations (68% V 97%). A Spearman's rank coe cient showed a moderate but statistically signi cant positive correlation between students results in their o cial university examinations and their score in this investigative examination.

Conclusion
This work demonstrates a disparity between both student and junior doctor levels of knowledge with consultant expectations from an examination based on what front-line paediatricians determined as "must-know" standards. This study demonstrates the importance of involvement of end-users and future supervisors in undergraduate teaching.

Background
Every year a fresh group of medical graduates start work for the very rst time and become responsible for clinical decision-making and the treatment of patients. Senior medical staff provide guidance and supervision, but newly quali ed doctors are expected to carry out aspects of the job independently by virtue of their undergraduate medical training. Despite this, it is recognised that early-career junior doctors identify a number of gaps between what they were taught during these undergraduate years and their clinical work as a doctor (1). There is also a reported discrepancy between graduates' self-assessment and their educational supervisor's assessment of their practice, suggesting either a lack of clarity of expected standards or elements of inter-observer variability (2). A General Medical Council (GMC) report exploring the extent to which United Kingdom (UK) medical graduates are prepared for practice' recognised that newly quali ed doctors feel unprepared in many areas of their daily practice and recommended transition interventions, such as assistantships or work-shadowing, to address this (3).
Undergraduate curricula are generally designed to provide a broad base in paediatric knowledge for doctors in all elds, but even if an undergraduate curriculum is targeted towards later entry into clinical practice there are other hurdles to overcome. For specialities such as Paediatrics, where there is often a 12-month interval between quali cation and starting into clinical practice, knowledge retention is also important. Peter Sullivan, who questions how much of what is taught during the undergraduate years is actually retained, highlighted this potential de cit in medical education in the UK. With regards to the paediatric core curriculum, he reports that 1-year post original testing, student's mark decreased by 50% (4). A year's experience working in general clinical practice may consolidate clinical performance, but it would appear that undergraduate knowledge in specialities is not retained. This could have implications for both undergraduate content and methods of learning but also for postgraduate training programs and induction.
Many undergraduate curricula and assessment strategies are designed by academic doctors employed by universities, and there is little evidence of input from 'non-faculty' clinicians (5). However, after qualifying it is often the 'non-faculty' clinicians supervising them, who set the standard of what is expected in their clinical practice. While clinical knowledge and skills are not the only desired outcomes of an undergraduate program, they remain core to most courses. At undergraduate level, contributions from non-academic clinicians are often informal. This contrasts with the postgraduate exam approach that actively encourages and seeks out non-academic clinician input (6). There is a paucity of published literature on what level of knowledge is expected of new trainees by clinical consultants working in frontline paediatrics. While undergraduate and postgraduate training curricula are explicit, there is no clear roadmap or speci c clinical guidance documenting what is expected of the trainee as they start in paediatrics, other than an extrapolation from an undergraduate university assessment-which may re ect a more general graduate requirement.
This study aims to evaluate how undergraduate students perform against an examination of knowledge set solely by non-academic clinicians at a level that they deemed was "must -know", i.e. the basic level of knowledge they would expect from a junior doctor (Senior House O cer (SHO)) starting in their service and in paediatrics for the rst time.

Methods
This was a study of performance in a novel examination in paediatrics by undergraduate students from two large medical schools in Ireland and paediatric junior doctors. Ethical approval for the project was  Devising a Minimum Accepted Competency (MAC) paper Clinicians registered with the RCPI (Paediatric division) were contacted by e-mail on 01/08/2015 with a request to provide questions for use in this examination. They were asked to generate questions based on "must know" information that, in their opinion, was necessary for every junior doctor starting their rst post in paediatrics. Each clinician was asked to submit examination questions in 'multiple choice' (MCQ) or 'true/false' format. One follow up e-mail was delivered on 01/02/2016 and no further submission were permitted beyond 01/05/2016. Submissions from clinicians who held an academic position at a university were excluded.
An academic trained in assessment and question writing, who was not directly involved in the study, reviewed the questions for clarity and language, however neither content nor di culty level were changed.
A bank of questions was created, and a random number generator was then used to choose 30 questions to form the research examination (MAC) paper (01/06/2016). We limited the examination paper to 30 questions in order maximise participant recruitment.
Creating a passing score On 17/06/2016 the questions were standard set by the undergraduate academic paediatric faculty of the RCSI at a standard setting meeting for the university's paediatric written examination. Academic staff present included the professor of paediatrics, the associate professors of paediatrics and paediatric clinical lecturers. The academic team had experience of standard setting using the Angoff technique(4). In the Angoff technique, examiners decide what proportion of the 'minimally competent' or 'borderline' candidates they would expect to correctly answer a given question. The mean average of each examiner's decision is calculated to give the standard for that question. Then, the mean average of all of the questions is used to set a standard for the paper and generate a 'passing score'. Academic staff participating in the standard setting were blinded to whether questions formed part of the o cial university written examination or comprised part of this research study.

Delivery of the MAC examination Participants
Undergraduate students were recruited from two universities; RCSI (Dublin) and QUB (Belfast). In June 2016 all of the RCSI students from Senior Cycle 1 (SC1 -the penultimate year of university during which they complete their paediatric teaching) were invited to attend for a mock examination in one of the lecture theatres at RCSI, one week before sitting their university written examination. This mock examination was the 'MAC' examination. The following year, between October 2016 and May 2017, all RCSI students from SC1 were invited to sit the MAC examination at the end of their 6-week paediatric clinical attachment, in the lecture theatre at the hospital within which they were completing their clinical attachment (Temple Street Children's Hospital (Dublin) and Our Lady's Children's Hospital Crumlin (Dublin)). Between March 2018 and May 2018, QUB students from year 3 (the year in which they complete their paediatric teaching) were invited to sit the MAC examination at the end of their 6-week paediatric clinical attachment in one of the lecture theatres at the Royal Victoria Hospital, Belfast. Due to a delay in receiving ethical approval only 2 out of 5 of the QUB paediatric attachments could be included.
All SHO's currently enrolled in the Irish Basic Specialist Training (BST) (7) [1] scheme were approached to sit the MAC examination during the rst paediatric training day of the new academic year in October 2016, at which point they had been working in paediatrics for 3 months. This meeting and the subsequent examination took place in one of the lecture theatres at RCPI.

Data collection
Each of the examinations took place under standard examination conditions and was invigilated by the study investigator. Consent was obtained and paper examination sheets were distributed to each participant. These were collected and marked at the end of the examination by the study investigator.
MAC examination papers were destroyed once the mark had been transferred to the research database.

Data analysis
The results of the MAC examination were analysed to determine if participants had reached a cliniciandetermined minimum accepted competency. Examination results are reported as the mean with standard deviation and median with interquartile range (boxplots). The proportion of students achieving the standard set passing score is described. Normally distributed data were analysed using student t-tests with a p-value <0.05 representing statistical signi cance.
Undergraduate results on the MAC examination were compared with the same student's university nal results in paediatrics to determine how research examination scores compared using Spearman's Rank correlation. For RCSI undergraduates, MAC examination results were compared with numerical student' scores from their o cial university end of year paediatric written examination. For all undergraduate participants (RCSI and QUB), class rank in the MAC was compared with class rank in the nal paediatric exams. Institutional ethical approval did not allow for a direct comparison of the individual results of QUB and RCSI students. However, for the purpose of investigating consistency in the performance of students across two different institutions, we calculated a correlation between QUB students rank in the MAC examination with their rank in their o cial QUB paediatric examination and compared this with how RCSI students correlated between rank in MAC examination and rank in their o cial RCSI paediatric examination.
For the SHO participants, results were analysed to determine if there was a difference in the performance between the paediatric junior doctors and the undergraduate students.
Page 6/16 [1] The Basic Specialist Training (BST) scheme is a two-year programme completed in Senior House O cer (SHO) posts. Completion of BST in paediatrics is the rst step towards becoming a paediatrician in Ireland.

Results
The email request for questions was delivered to 238 out of 247 (96%) members of RCPI. A total of 76 questions (5 duplicates) were contributed by 15 consultants. The rst reply arrived on 17/08/2015 and the nal reply was received on 27/04/2016. The questions on the MAC examination were from a diverse selection of sub-specialty and general paediatricians. The question therefore tested a wide range of common and clinically important areas within paediatrics including; seizures, lower respiratory tract infections, growth and emergencies.
The questions were reformatted to a 'single best answer from 5 options' MCQ structure to match the question format in use at the time for paediatric undergraduates at RCSI. A random number generator was used (randomly selecting numbers between 1 and 71) and the rst 30 selected were used as the MAC examination.
Using a modi ed Angoff technique in a blinded setting, 9 members of the RCSI faculty calculated a passing score of 41.2%, equating to a passing score of 13/30 on the MAC examination. Reasons for declining the invitation to sit the MAC examination were not recorded as this was a voluntary extra assessment and not part of the mandatory curriculum.

Discussion
The literature includes examples of 'minimal essential requirement' MCQ examinations designed to assess competence (8). However, there does not appear to be a comparable study to our own with regards the generation of the MCQ items solely from 'non-academic' clinicians and so from this regard, this investigation introduces a novel concept.
With such a focus on the content of examination questions, there is a risk that one could lose sight of the bigger picture, which is to ensure that our medical institutions produce good quality physicians. A paper by Christensen et al in 2007 highlights the importance of an 'outcome-based' approach in medical education, compared to a process/content orientation. However, they do hold some reservations, as worry is expressed about the taxonomy of learning in pure outcome-based medical education, in which student assessment can be a major determinant for the learning process, leaving the control of the medical curriculum to medical examiners (9). In development of the MAC examination, we have designed an examination which is outcomes based but also designed by a wide range of clinicians as opposed to a minority of faculty members.
Another study with a similar methodology to our own showed that when students (as opposed to clinicians) wrote the MCQ questions, when sat by their peers, the results correlated well with results in their o cial pediatrics examination but were overall of a tougher standard (10). A recent study of surgical students showed that the scores from a peer-written examination correlated well with other independent measures of knowledge such as United States Medical Licensing Examination (USMLE) and National Medical Board Examination (NMBE) examinations and also with the universities' surgery clerkship examination (11). This is comparable with results from the MAC examination which have been shown to correlate well with the same students' marks on the o cial RCSI nal paediatric examinations.
Potts et al wrote a paper with a similar concept to our own MAC examination (12). They designed a summative assessment based on six core paediatric objectives. Passing all items was a requirement and failure required remedial oral examination of any missed items. When 'pre-warned' of the curriculum change and emphasis on these aspects of the curriculum, student's grades on this examination signi cantly improved compared with the previous year (control group). However, this same cohort of students performed worse on the NBME paediatric subject examination. In the Potts study, the students' poor performance in the NBME is likely because their attentions were drawn towards passing the new 'inhouse' summative examination, the consequence of which was missing many key components of the curriculum as set out by the NBME. The study did not answer whether or not these students were poorly prepared to be paediatricians, but simply highlights the distinct difference in the two curriculums. This raises a few worthy points; assessment drives what students learn and therefore needs to be re ected in the curriculum, also, students are very capable of meeting an agreed learning objective when prepared for it. However, if the curriculum they are taught is not re ected in all of their summative testing it can prove detrimental. This is comparable with our study in that the students performed poorly in the MAC compared with the o cial RCSI examination, the curriculum for which they were familiar with and were speci cally prepared for. When set a different test, albeit, on the same subject, the scores signi cantly reduced.

Standard setting MAC examination
The result of the standard-setting process was that the MAC examination was given a 'passing score' of 41.2% (13/30). This is relatively low for any ' nals' high-stake examination but particularly so when considered that the initial intention was to design an examination with questions which were deemed 'must know', 'basic knowledge.' One could argue that the method by which test questions were gathered (i.e. requesting for 'minimal required competency') had 'pre-standard set' the MAC examination at close to 100% [it cannot be exactly 100% due to the standard error of measurement (13)]. When considered like this, 41.2% becomes even more remarkable.
Despite the fact that in many institutions cut scores are often between 50-70 % (14), there is an argument that cut scores should be high. The higher the cut score, the smaller the chance of false positives (i.e. candidates able to pass the examination by guessing the answers). This is of particular importance when the licensure will be in a task, failing which will cause serious effect on the individual or society using the service (15), such as in nal medical examinations.
Our vision of the MAC examination was one in which the questions would be of a relatively easy level, appropriately standard set with a high passing score (for example, 80-90%) and therefore candidates would need to have a complete understanding a more limited syllabus upon which the questions are based in order to pass the test.
The low standard set passing score of 41.2% can be explained by the fact that either the providers of the questions (i.e. non-academic paediatricians) had a much higher expectation than that of the faculty, or that the faculty greatly underestimated the knowledge level of the students. Either way, a passing score of 41.2% re ected a faculty opinion that this was a di cult test in which a candidate need only get 13 out of a possible 30 questions correct to satisfy the criteria worthy of passing.

Analysis of results
We must consider why the students found the MAC questions so di cult and why so many did not achieve the passing score. Was the examination standard set at too high a level? This is unlikely as the passing score is already below industry standards. Alternatively, do the students simply not have the targeted knowledge required to pass the MAC? To put it another way, their level of paediatric knowledge re ects the RCSI curriculum, indicated by the high passing rates (96-97%) of the same students in the RCSI examination. Their relatively poor results in the MAC examination have therefore highlighted a signi cant gap between the RCSI curriculum and the knowledge required for the MAC examination ( i.e what the non-academic consultants expect them to know).
The poor results in the MAC examination do not indicate that these students will necessarily make poor paediatric doctors, but it highlights a potential difference between the RCSI curriculum and the 'hidden' curriculum as determined by non-academic clinicians.
Neither the students nor the junior doctors appeared to know the "must know" questions. Is this poor preparation or unrealistic expectations? Frontline non-academic paediatric clinicians were asked to provide clinical questions based around essential knowledge for practice. Despite the instruction to the question providers that MAC questions should re ect 'must know', 'basic knowledge' and a 'minimum accepted competency', this exercise yielded a relatively low passing score and re ected the di culty of the standard of questions being asked. Each individual submitting questions would have described the set standard for their own questions as 100% (i.e. "must know") but it is possible that this was an unrealistic expectation for undergraduate students. Contributors were not asked to assess the standard of other submitted questions. This would have been a useful exercise, but it was beyond the scope of this study to recruit and train non-academic clinicians in standard setting. When considered in the context of "must know" information, the average score of 45-46% for undergraduate students was considerably less than would be expected by clinicians working at the 'frontline' of general paediatrics. This may re ect unrealistic expectations, or a curricular emphasis on alternative content.
Reassuringly, the paediatric SHOs' about to embark on their paediatric career performed signi cantly better than the medical students. This is an important nding as the MAC examination was designed as a test of knowledge required for 'on the ground' clinical practice. In Ireland, paediatric training can commence at postgraduate year 2 (graduates complete a one year 'internship' in general medicine/surgery during which time they apply for subspecialty training to commence the following year). The majority, but not all, of the participating SHOs' would therefore have had 2 more years of clinical experience (1 in their nal year of undergraduate study and a 1-year internship). These participants appeared to have bene tted from the extra clinical experience, albeit not in paediatrics.
However, their results still did not match the "must know" standard initially expected by the clinical paediatricians setting the questions.
Why was the MAC examination result standard set so low if the questions were meant to be 'must know' 'basic' knowledge? This re ects a difference in opinion of expected standards between faculty for undergraduate students and that of non-academic clinicians for junior doctors in paediatrics. With the latter seemingly expecting a higher level of knowledge. However, perhaps rather than a 'higher level' of expected knowledge, non-academic clinicians expected a different type of knowledge. It is possible that an undergraduate focus on traditional 'textbook' facts did not align with the clinicians' focus on practical aspects of the job, which are particularly relevant to everyday clinical practice. This potential difference in knowledge or focus warrants further investigation at undergraduate level and possibly intervention at early postgraduate level for those planning to practice in paediatrics. There is a move in some third level institutions to revisit the structure of their undergraduate teaching to increase focus on clinical practice and the broader non-clinical skills required by the physicians (16).
All of the universities within the island of Ireland have recently collaborated to develop a national undergraduate paediatric curriculum. This will go some way to standardising the knowledge acquired by graduates working in Ireland and is a great opportunity to revisit how undergraduate programs are taught. This process should incorporate the views of a wide range of 'non-academic' paediatric clinicians to ensure that it can bridge the gap between what is taught and assessed at undergraduate level and what is practically important in the workplace. This study highlights the di culty in attempting to deliver an undergraduate course that both establishes a core of basic paediatric knowledge and prepares a student for the postgraduate clinical environment. However undergraduate medical education is not merely about transferring knowledge to future medical practitioners. It is also about developing transferrable general clinical and non-clinical skills required for good medical practice, including Human Factors, and engendering the skills for lifelong self-directed learning. It may be that bridging this 'gap' is not necessarily the responsibility of the university that is preparing graduates to work as general physicians rather than subspecialists, but rather the postgraduate training bodies should possibly be identifying ways in which this type of knowledge is provided and assessed prior to entering the training scheme. This could be delivered in a short induction course and the transitional period of assistantship that many universities now have in place would seem a suitable time to do this. It is anticipated that the results of this study can inform the content of transition interventions to better prepare them for practice.
Did the students from year to year perform differently? RCSI students have two paediatric examinations that contribute to their nal marks. The rst is a clinical examination done immediately after the six-week paediatric rotation, when students are fresh from their paediatric clinical experience. The second is a written MCQ given to all students at the end of the academic year, when students have been focusing on knowledge acquisition. There was no signi cant difference between the results obtained in the MAC examination between either year of RCSI students, despite the fact that one year had the assessment at the end of their paediatric rotation and the other at the end of the academic year. In addition, the fact that two large groups of students obtained such similar results in the exam suggests that this examination is reproducible from year to year.
The SHOs', with their increased clinical experience, performed signi cantly better than the students. This may re ect the clinical emphasis of the questions or possibly that junior doctors specialising in paediatrics were likely to be more interested in the subject and so would be expected to do better, irrespective of when they were assessed.

Quality of university examinations
Concerns have been raised that the quality of university examinations may not always be su cient for high-stakes decision-making (17) in clinical practice. Studies have shown that undergraduate medical examinations can be of relatively low quality (18) and that the quality of written examination questions can be signi cantly improved by providing question writers with formal training (19). It may be an unrealistic target to expect a large group of 'non-academic' clinicians to undertake extra training in examination writing. A potential solution to this problem would be to encourage our 'non-academic' colleagues to provide the question content, in any format they feel most comfortable with, and then to deploy a team of trained academics to revise these questions into a more suitable format and improve their psychometric properties. In fact, this is how the Royal College of Pediatrics and Child Health (RCPCH) generate their examination questions. They set up question setting groups throughout the country, headed by a member of faculty but attended by non-academic consultant pediatricians and senior registrars. These questions are then reviewed by the theory examiner team at 'board meetings' which occur twice a year, at which point the questions are either excluded or revised to be included in a potential bank of 'live' questions for use in subsequently written examinations.

Study limitations
There were 15 consultant clinicians providing 71 questions for the MAC examination. It is possible that there would have been even greater breadth and diversity to the questions if there had been a greater number of paediatricians contributing questions. The results of this study may have been in uenced by the fact that it relied on volunteers to provide questions. Therefore, these consultants have self-selected to a certain degree, and our sample may not accurately re ect the opinion of the 'average' paediatric clinician. However, their contribution is extremely valuable, as these individuals were su ciently motivated to contribute to this work.
The o cial RCSI written examination has 150 test items and therefore the MAC examination, with only 30, is testing a smaller sample of knowledge. We appreciate that this has limited our results. However, the questions used covered a range of topics within paediatrics and represent a nite amount of 'basic, must know knowledge.' Both the undergraduate students and SHOs' who sat the exam did so voluntarily, and so the results may re ect a more motivated population than the cohort overall. In the SHO cohort, the 93% response rate makes it unlikely that this would have an important effect. In the undergraduate cohort, the proportion of possible candidates volunteering for the exam was lower, so the chances of selection bias are greater. However, there was a signi cant positive correlation between their MAC results and their o cial university results. As these rankings did not merely cluster at the top of the class, it is clear that it was not just the highest achieving students who had volunteered to do the exam.

Conclusion
This study suggests there is a knowledge disparity between what is taught and assessed in the undergraduate domain and what is expected as essential knowledge in the postgraduate domain.
Increasing co-operation between academic and experienced non-academic clinicians should help to bridge this gap. Transition interventions such as assistantships and work shadowing would seem to provide a platform for this. It is anticipated that studies such as this will help inform the content of such interventions to ensure that future junior paediatric doctors are optimally prepared for practice.