Student evaluation of an OSCE in paediatrics at the University of the West Indies, Jamaica
- Russell B Pierre†1Email author,
- Andrea Wierenga†2,
- Michelle Barton1,
- J Michael Branday2 and
- Celia DC Christie1
© Pierre et al; licensee BioMed Central Ltd. 2004
Received: 08 April 2004
Accepted: 16 October 2004
Published: 16 October 2004
The Faculty of Medical Sciences, University of the West Indies first implemented the Objective Structured Clinical Examination (OSCE) in the final MB Examination in Medicine and Therapeutics during the 2000–2001 academic year. Simultaneously, the Child Health Department initiated faculty and student training, and instituted the OSCE as an assessment instrument during the Child Health (Paediatric) clerkship in year 5. The study set out to explore student acceptance of the OSCE as part of an evaluation of the Child Health clerkship.
A self-administered questionnaire was completed by successive groups of students immediately after the OSCE at the end of each clerkship rotation. Main outcome measures were student perception of examination attributes, which included the quality of instructions and organisation, the quality of performance, authenticity and transparency of the process, and usefulness of the OSCE as an assessment instrument compared to other formats.
There was overwhelming acceptance of the OSCE in Child Health with respect to the comprehensiveness (90%), transparency (87%), fairness (70%) and authenticity of the required tasks (58–78%). However, students felt that it was a strong anxiety-producing experience. And concerns were expressed regarding the ambiguity of some questions and inadequacy of time for expected tasks.
Student feedback was invaluable in influencing faculty teaching, curriculum direction and appreciation of student opinion. Further psychometric evaluation will strengthen the development of the OSCE.
The assessment of student's clinical competence is of paramount importance, and there are several means of evaluating student performance in medical examinations [1, 2]. The Objective Structured Clinical Examination (OSCE) is an approach to student assessment in which aspects of clinical competence are evaluated in a comprehensive, consistent and structured manner, with close attention to the objectivity of the process . The OSCE was introduced by Harden in 1975 , and first described as an assessment format in Paediatrics (Child Health) by Waterson and colleagues . Since its inception, the OSCE has been increasingly used to provide formative and summative assessment in various medical disciplines worldwide , including non-clinical disciplines .
The University of the West Indies was established in 1948 as a medical college of the University of London, which granted external degrees to those who successfully completed the course . The Faculty of Medical Sciences located on four campuses, on the islands of Jamaica, Bahamas, Barbados and Trinidad and Tobago, conducts bi-annual final examinations at the end of year 5. The 'traditional' format of examination that included long case, short cases and oral examination, was preserved until recent changes in the curriculum. In response to recommendations to improve the validity and fairness of the examination through adoption of proven methods and approaches in assessment and evaluation in medical education, the Faculty of Medical Sciences (FMS), University of the West Indies (UWI) initiated the OSCE as a formal method of assessment for the final examination in Medicine and Therapeutics, Child Health, Community Health and Psychiatry, in November 2000. Students and faculty were exposed for the first time to a relatively new assessment instrument in which aspects of competence (communication, history-taking and technical skills) were assessed in a structured, formal manner.
The Section of Child Health, Mona, Jamaica, implemented the OSCE examination as an end-of clerkship assessment for students in their 5th year, during the 1999–2000 academic year. It was felt timely in order to (a) direct and motivate student learning in areas not previously assessed in the 'traditional' curriculum, (b) verify students' competence in fundamental paediatric clinical skills, and (c) provide a forum for feedback to students on their strengths and weaknesses in clinical skills. It was thought that it would enhance faculty and student acceptance of this new assessment tool and promote faculty training for the newly introduced final OSCE examination.
In the absence of any previous information from this institution, the study was designed to evaluate student overall perception of the end-of-clerkship OSCE, determine student acceptability of the process and provide feedback to enhance further development of the assessment.
A standardised technique of marking was used and student performance was assessed by criterion reference for each station. Criterion-based scoring was used, with each checklist item scored as 0 (omitted, incorrect or inadequate), or 1–2 (correct or adequate).
Face and content validity of each checklist was established by review and consensus by a core group of senior paediatricians. Stations were first selected to represent the curricular goals and objectives and to reflect authentic clinical situations. Checklists were designed to include the features thought to be most important by the development committee. Through discussions, consensus was achieved on the checklist items and structure.
The study was conducted during the period July 2001 to December 2002. Five groups of students participated in the process, during their respective clerkship rotations. Student groups had at least two briefing sessions before the OSCE, and included an orientation about the examination process (both end-of-clerkship and final MB) and a review of commonly assessed competences. They were also apprised of the valuable contribution they could make towards improving the assessment and encouraged to participate in the evaluation.
A cross-sectional survey using a 32-item self-administered questionnaire was completed at the end of each OSCE . Students were asked to evaluate the content, structure, and organization of the OSCE, rate the quality of performance and objectivity of the OSCE process, and to give their opinion about the usefulness of the OSCE as an assessment instrument compared to other forms which they had experienced (essays, multiple choice questions, long and short cases, general clerkship rating).
Participation was on a voluntary basis and students were assured that those who declined involvement in the survey would not be penalised. The Curricular Affairs Section handled the administration and analysis of the questionnaires. Ethical approval was received from the University Hospital of the West Indies/University of the West Indies Faculty of Medical Sciences Ethics Committee. Following completion of the questionnaire, an OSCE review session was conducted with the students for feedback and teaching purposes, at the end of the clerkship. Students were given the opportunity to review their individual performances at the respective stations. Examiner evaluations were also used in the feedback process.
Data were collated and descriptive and non-parametric tests applied using Stata version7 . Basic statistical analysis of the Likert items was conducted by calculating frequencies, means and standard deviations. Qualitative analysis was done through a form of content analysis by identifying themes in student responses and grouping responses according to thematic content. Two of the authors individually conducted this content analysis and identified themes and final grouping of responses were developed by consensus.
Eighty-one students responded to the questionnaire, representing 92 % (81/88) of those who completed the Clerkship.
No comment %
Exam was fair
Wide knowledge area covered
Needed more time at stations
Exams well administered
Exams very stressful
Exams well structured & sequenced
Exam minimized chance of failing
OSCE less stressful than other exams
Allowed student to compensate in some areas
Highlighted areas of weakness
Student aware of level of information needed
Wide range of clinical skills covered
Most (73–82%) felt that the exam was well administered, and that the stations were arranged in an organised and well-sequenced order.
Students believed that the assessment was fair (68%). Fifty-three percent were aware of the level of information required at each station, yet 28% felt that the examination process minimized their chances of failing.
Students found the OSCE to be intimidating (48%) and more stressful (35%) than other assessment formats to which they were previously exposed. And most (70%) felt that they needed more time to complete the stations.
Quality of performance testing
Not at all %
To great extent %
Fully aware of nature of exam
Tasks reflected those taught
Time at each station was adequate
Setting and context at each station felt authentic
Instructions were clear and unambiguous
Tasks asked to perform were fair
Sequence of stations logical and appropriate
Exam provided opportunities to learn
Most saw the OSCE as a useful learning experience and that the content reflected real life situations in Child Health. More than half of the students were satisfied with the conduct, organisation and administration of the OSCE.
Perception of validity and reliability
Student perception of validity and reliability
Not at all %
To great extent %
OSCE exam scores provide true measure of essential clinical skills in paediatrics
OSCE scores are standardized
OSCE practical and useful experience
Personality, ethnicity and gender will not affect OSCE scores
Comparing assessment formats
Student rating of assessment formats
Which of the following formats is easiest?
Which of the following formats is fairest?
Learn very little %
Learn a lot %
From which of the following formats do you learn most?
Used much less %
Used much more %
Which of the following formats should be used more often in the clinical years of the programme?
Thirty-two percent of students felt that the clerkship rating was the easiest, while 48% rated MCQ as a more difficult form of assessment. The OSCE was overwhelmingly considered the fairest assessment format (80%), and essays (68%) to a lesser extent. OSCE (60%) and clerkship ratings (62%) were considered the most useful learning experiences. Compared to the other assessment formats, 52% considered that the OSCE should be used most in the clinical years.
Students were asked follow-up questions related to positive and negative aspects of the OSCE and suggestions for improvement. The open-ended responses were grouped by thematic content.
Among the positive attributes of the OSCE, students re-affirmed that the assessment was comprehensive (44 comments) and that it was an objective and fair process (43 comments). Some indicated that the opportunity for feedback helped to motivate them and drive the learning process (21 comments).
Students felt that the time allocated to perform expected tasks was insufficient (36 comments), and that the procedure was stressful (18 comments) and tiring (13 comments). Technical problems (28 comments) included unclear instructions, inadequate time provision and instructions between stations and detention of some candidates at stations by examiners.
Suggestions for improvement included increasing the duration of stations (29 comments), ensuring clear instructions (8 comments) and having more realistic expectations of students for the expected tasks. A few students wished to have more training with the OSCE and suggested that the examination should be videotaped to increase objectivity and permit review.
Students overwhelmingly perceived that the OSCE in Child Health had good construct validity. This was demonstrated by the favourable responses concerning transparency and fairness of the examination process, and the authenticity of the required tasks per station. Excellent levels of acceptance of the OSCE by students have been previously described in the literature [11–14]. They however expressed concerns and uncertainty about whether the process would minimize their chances of failing or that the results were a true reflection of their clinical skills. This was understandable, since it was their first encounter with this type of assessment.
Several felt that the examination was stressful and intimidating, yet paradoxically some students perceived it as an enjoyable, practical experience. Studies surveying student attitudes during the OSCE have documented that the OSCE can be a strong anxiety-producing experience, and that the level of anxiety changes little as students progress through the examination .
It is well recognised that assessment is a catalyst for both curriculum change and student learning. The students recognised the value of the instrument for formative evaluation. In addition, as many medical schools have adopted a student-centred approach to medical education, greater student participation in quality assurance exercises must be encouraged. Students perceived the OSCE to be fairer than any other assessment format to which they were exposed. The findings were somewhat similar to the views of students at Newcastle medical school . Although student views on fairness may not be consistent with published literature, the impact and influence on acceptability of the instrument should be noted.
They offered constructive criticism of the structure and organisation of the process. At some stations they felt that the instructions were ambiguous and that the time allocation was inadequate for the expected tasks. The feedback was invaluable and facilitated a critical review and modification of the station content and conduct of the examination over time. Faculty perceived that the concerns about time allocation per station and the degree of stress expressed by the students were due to inadequate preparation for the examination, particularly in competences not previously assessed in the 'traditional' examination.
The high student response rate has helped to ensure that the findings presented are a valid representation of student opinion. Students have traditionally viewed the end-of-clerkship assessment as a 'high-stake' examination and also perceive it as predictive of their performance at their final MB examination. Student perception of the OSCE however, may have been influenced by anxiety and lack of confidence associated with a new assessment. The responses may also have been affected by the timing of the inquiry (immediately after the examination); hence student stress and fatigue should be taken into consideration. Whereas the high response rate ensured that the views were reasonable representative of the students, differences in assessors could have influenced the interpretation of the results of open-ended responses.
Implementing the OSCE in Child Health at the University of the West Indies, Jamaica has been challenging, however student participation in the evaluation and their overall acceptance of the instrument have been encouraging. Feedback from students and faculty has been useful in effecting improvements to the process and greater emphasis has been placed on the teaching and evaluation of history taking, communication and technical competencies. It is also sending a clear message to students that the achievement of overall competence is imperative to clinical practice in the current environment. Ultimately, these provide the loop necessary to drive the continuum of curriculum development. This has been timely considering that the Faculty of Medical Sciences, Jamaica is undergoing significant reform . Further developments involving psychometric evaluation will strengthen the process.
In summary, the findings highlight the need for student participation in the development of new assessment tools in medical curricula. Student acceptance will be more favourable for assessment formats that they perceive to be transparent, authentic and valid. 'Traditional' medical curricula must be responsive to global paradigm shifts in undergraduate medical education.
We wish to thank Dr. Jerome De Lisle of the Centre for Medical Science Education (CMSE), EWMSC, Trinidad, for professional advice and permission to use the questionnaire in this study. We also express our gratitude to the participating students and lecturers in Child Health who contributed to the implementation of the OSCE in the department.
- Harden RM: How to assess clinical competence – an overview. Med Teach. 1979, 1: 289-296.View ArticleGoogle Scholar
- Fowell SL, Bligh JG: Recent developments in assessing medical students. Postgrad Med J. 1998, 74: 18-24.View ArticleGoogle Scholar
- Harden RM: What is an OSCE?. Med Teach. 1988, 10: 19-22.View ArticleGoogle Scholar
- Harden RM, Stevenson M, Downie WW, Wilson GM: Assessment of clinical competence using objective structured examination. Br Med J. 1975, 1: 447-451.View ArticleGoogle Scholar
- Waterson T, Cater JI, Mitchell RG: An objective undergraduate clinical examination in child health. Arch Dis Child. 1980, 55: 917-922.View ArticleGoogle Scholar
- Carraccio C, Englander R: The objective structured clinical examination, a step in the direction of competency-based evaluation. Arch Pediatr Adolesc Med. 2000, 154: 736-741.View ArticleGoogle Scholar
- Harden RM, Caincross RG: The assessment of practical skills: the Objective Structured Practical Examination (OSPE). Stud High Educ. 1980, 5: 187-196.View ArticleGoogle Scholar
- Sherlock P, Nettleford R: The University of the West Indies: a Caribbean response to the challenge of change. 1990, Hong Kong: Macmillan CaribbeanGoogle Scholar
- De Lisle J: 2001 Phase 2, OSCE student evaluation form. 2001, Mount Hope, Trinidad The Centre for Medical Science Education, Faculty of Medical SciencesGoogle Scholar
- StataCorp: Stata Statistical Software: Release 7.0. 2001, College Station, TX: StataCorp LPGoogle Scholar
- Newble DI: Eight years experience with a structured clinical examination. Med Educ. 1988, 22: 200-204.View ArticleGoogle Scholar
- Duerson MC, Romrell LJ, Stevens CB: Impacting faculty teaching and student performance: nine years' experience with the objective structured clinical examination. Teach Learn Med. 2000, 12: 176-182. 10.1207/S15328015TLM1204_3.View ArticleGoogle Scholar
- Kowlowitz V, Hoole AJ, Sloane PD: Implementation of the Objective Structured Clinical Examination in a traditional medical school. Acad Med. 1991, 66: 345-347.View ArticleGoogle Scholar
- Woodburn J, Sutcliffe N: The reliability, validity and evaluation of the objective structured clinical examination in podiatry. Assessment Evaluation Higher Educ. 1996, 21: 131-147.View ArticleGoogle Scholar
- Allen R, Heard J, Savidge M, Bittengle J, Cantrell M, Huffmaster T: Surveying students' attitudes during the OSCE. Adv Health Sci Educ. 1998, 3: 197-206. 10.1023/A:1009796201104.View ArticleGoogle Scholar
- Duffield KE, Spencer JA: A survey of medical students' views about the purposes and fairness of assessment. Med Educ. 2002, 36: 879-886. 10.1046/j.1365-2923.2002.01291.x.View ArticleGoogle Scholar
- Faculty of Medical Sciences, University of the West Indies, Mona. Report – MB, BS Undergraduate Programme, First Annual Report on Curriculum Development. 2003, FMS, UWI, Mona, JamaicaGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/4/22/prepub
This article is published under license to BioMed Central Ltd. This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.