Assessment of clinical competencies using clinical images and videos“CIVA”

Abstract

Background

This paper describes an assessment approach of clinical competencies whichwidens the number of problems and tasks evaluated using videos andimages.

Method

Clinical Image and Video Assessment (CIVA) was used to assess clinicalreasoning and decision making of final year medical students. Forty to fiftyclinical videos and images supported by rich text vignette and reviewed bysubject matter experts were selected based on examination blueprints foranalysis. CIVA scores were correlated with OSCE, Direct Observation ClinicalEncounter Exam (DOCEE) and written exam scores, using the 2-sided Pearsoncorrelation analysis, and their reliability was analyzed usingCronbach’s Alpha Coefficient. Furthermore, students personallyevaluated the CIVA using a 5- point Likert scale.

Results

CIVA and OSCE scores showed a high correlation (r = 0.83) incontrast with the correlation scores of the written examination(r = .36) and the DOCEE (r = 0.35). Cronbach’sAlpha for the OSCE and CIVA for the first batch was 0.71 and 0.78. As forthe second batch it was 0.91 and 0.91 respectively. Eighty-two percent ofstudents were very satisfied or satisfied with the CIVA process, contentsand quality.

Conclusions

A well constructed CIVA type assessment with a rich authentic vignette andgood quality videos and images could be used to assess clinical reasoningand decision making of final year medical students. CIVA is an assessmenttool which correlates well with OSCE, compliments the written and DOCEE andis easier to conduct at a possibly reduced cost.

Background

Clinical competency is an outcome which entails several skills directly related tothe holistic approach to patient care. Clinical reasoning, as a construct, guideshypothesis driven history taking, physical examination, diagnosis and management.Assessing clinical reasoning as a reflection of medical students’ ability inpatient care is key to medical education. Researchers have shown that physicians useboth analytical and non-analytical “pattern recognition” approaches,alternating between both in varying degrees [1]. As a result, developing, teaching and assessing clinical reasoning hasalways been a challenge in medical education [2–4].

Multiple tools and instruments have been described and used in assessment of clinicalcompetencies, consequently Van der Vleuten [5] used Millers Pyramid to hierarchically place them in accordance to theirassessment outcomes, those being - Knows, Knows how, Shows and Does. Moving upwardstowards the “does” level increases the assessment of theinstrument’s authenticity, bringing it closer to reality and the workplace.The balance between instrument validity, authenticity, reliability, practicality,educational impact and cost is what all assessment systems aim to achieve [6], in order to have an appropriate impact on student learning [7].

The most ideal way to judge students’ or physicians’ clinicalcompetencies and clinical reasoning, would be through direct observation of a largenumber of real patient encounters in the normal work place environment. This isusually difficult to achieve because of the lack of suitable patients and patientsafety regulations. All commonly described clinical assessment instruments –‘A’ type Multiple Choice Questions (MCQs), Extending Matching Questions(EMQs), Key Feature Examinations, Objective Structured Clinical Examinations(OSCEs), Mini C-Ex [8], Direct Observation Clinical Encounter Examination (DOCEE) [9] have their limitations. Most assessment systems use these instruments incombination in order to achieve the criteria set by Van Der Vleuten, as mentionedearlier. The College of Medicine at the University of Sharjah, follows anintegrated, outcome based curriculum, that is conveyed to the student in a problembased learning (PBL) manner. Clinical skills are introduced in Year One andconstitute one of the several crucial vertical themes within the curriculum.

In assessing students’ clinical competencies, test blue prints, written exams,OSCEs and DOCEEs are used during the clerkship phase and exit examinations to assessa medical student’s knowledge, skills and attitudes. The OSCEs, particularlyin the clerkship phase comprise around 15 interactive “7 minute”stations using simulated patients in contrast to the DOCEEs, where the medicalstudent on average, examines four real patients. The DOCEE is designed to observeand assess the student’s full encounter with real patients. Each DOCEE stationspans over a period of 30 minutes where the examiner goes through a standardizedcheck list to assess the student’s performance. As for the student’sdepth of clinical knowledge, it is tested using 100 A-type Multiple Choice Questions(MCQs), in addition to 100 Extended Matching Questions (EMQs). All final exit examsare conducted over a period of one week.

In order to increase the validity and reliability of assessment of clinicalcompetencies and ensure wider sampling from the broad world of clinical context, wehave introduced an additional instrument known as “Clinical Images and VideoAssessment” (CIVA). It’s rationale is based on assessing students’pattern recognition competencies in generating diagnostic hypotheses and decisionmaking in patient management.

CIVA is a computer-projected, class administered test, comprising 40–50scenario rich electronic stations (slides) that demonstrate clinical images orvideos, followed by a number of short questions that relate to clinical reasoningand decision making regarding the patient that had been described in the scenario.By increasing the number of problems and practice context, the number of clinicaltasks tested increases.

The aim of this study is to evaluate the validity, reliability and thestudents’ perception of CIVA as an additional examination tool used for theevaluation of clinical reasoning and decision making.

Context

The College of Medicine, established in 2004, adopted an outcome -basedcurriculum, with its main approach to teaching being through Problem-basedLearning (PBL). The pre-clerkship phase starts in the foundation year, followingwhich students during the years after, those being year 1 and 2 are subjected toclinical work that is accompanied by community training in year 3 and continuedthrough in years 4 and 5. In addition to the acquisition of knowledge, thecurriculum focuses on introducing clinical skills training as early as yearone.

Starting from Year 2, in addition to OSCEs, CIVAs are also included to assessclinical reasoning including pattern recognition of clinical signs, decisionmaking as well as medical writing skills, such as prescriptions and referralletters. To ensure content validity, an assessment blueprint is designedbeforehand based on outcome objectives, problems and clinical tasks(Table 1) [10].

Table 1 Final MBBS exam

The Clinical Skills team (CST) is comprised of six clinical tutors; a lecturer, asenior faculty (NS) as well as clinicians from the University of SharjahHospital. The team is responsible for the assessment of clinical skills. Theteam director is a member of the assessment committee, which is responsible forformulating each exam’s master blueprint and monitoring its quality. TheCST is responsible for developing and implementing OSCEs and CIVAs in accordancewith the master test blueprint, which includes all domains assessed by differentassessment tools, written A-type, Extended Matching, OSCE, CIVA and DOCEE. EachOSCE and CIVA is reviewed by the assessment committee for validity and mappingbased on the master exam blueprint. The CST has developed a large database ofclinical videos and images which are revised frequently for face and contentvalidity. Videos and images are selected to assess outcomes of clinicalreasoning and decision making (diagnosis, management, follow-up). Thevideos/images are supplemented with a rich but short clinical vignette which isrevised for authenticity by subject matter experts (consultant clinicians). Inthe final exit examination at the end of the clerkship phase, CIVA is comprisedof 50 stations (Table 2). For each image, thequestions as well as the model answers are reviewed meticulously to preventoverlap. The CIVA and OSCE are offered on the same examination day. Students aredivided into two groups which alternate in taking the OSCE and CIVA. A few daysprior to the test, the CIVA is pilot tested to ensure good quality of images andsounds and to estimate the time required. The CIVA is offered to a group of25–30 students. Students write their answers in a structured answer books,which are then marked by members of the CST according to the model answers andcross checked by second assessors. The weight of each station differs accordingto the number of questions related to that station. The time required forpreparing the CIVA station varied according to the complexity of the station.Once developed and verified, the CIVA is saved in a CIVA bank for future use.The average time for displaying/answering each station is two minutes and thetime needed for marking ranges between 30 to 60 seconds.

Table 2 Examples of CIVA stations, University of Sharjah, College ofMedicine / Clinical Skills Program

Methods

Two final year students batches (n = 52 and n = 95) sittingfor the final exit examination were studied. Students' perception of the CIVA wasobtained though a questionnaire. Quantitative responses using a 5-point Likert scale(strongly disagree, agree, undecided, agree and strongly agree) were recorded. Thequestion items were related to the clinical problem, quality of the image and videoand time allowed for each station. The overall response rate was 72%.

Qualitative data was collected from the free response of the students and analyzed toidentify common emerging themes.

Statistical analysis

Students’ final scores, for the two batches, in CIVA, OSCE, DOCEE and MCQwere subsequently compiled in one file for further analysis. Pearson CorrelationCoefficient (r) was used to measure the Correlation of CIVA with the OSCE, DOCEEand MCQ. The level of significance was set at 5%. The Cronbach’s AlphaCoefficient was calculated as a measure of exam reliability. Because questionitems in the CIVA and OSCE exams differed in number and scoring among the twobatches, a separate reliability measurement was conducted for each.

Results

Table 3 and Figure 1 show thecorrelation (r) between CIVA and OSCE, the DOCEE and written exams grades. Thestrongest correlation was found to be between CIVA and OSCE (r = 0.83,p < 0.001). However, CIVA grades correlate less with written forms(r = 0.36, p < 0.001) and DOCEE grades(r = 0.35, p < 0.001). Cronbach’s Alpha for theOSCE and CIVA for the first batch of students was 0.71 and 0.78 and for the secondbatch was 0.91 and 0.91 respectively, indicating good reliability.

Table 3 Correlations coefficient (r) between CIVA and other modes of assessment(N of students = 147)

Figure 1

Scatter plots for correlations showing Pearson’s correlation(r).

Qualitative feedback following CIVA from the majority of the students was positivefor both educational and technical quality. The majority of students agreed/stronglyagreed that: the overall knowledge tested was fair (86%), was relevant andcorrelated well to the curriculum (89%) and reflected common case scenarios (84%).Technically CIVA ran smoothly (93%), sound and video effects were very good (96%)and time allocated was fair (80%). Suggested areas for improvement were“time allocated was too long, I could have done it in 30minutes”, “few stations have a lot of time, others is no time,but it was good overall”, “ better quality ofpictures”, “too many stations I felt sleepy”, “somex-rays were not clear”, “one slide for the eye was notobvious”.

Discussion

The assessment method of Clinical Images and Videos Assessment (CIVA) at theUniversity of Sharjah can be considered as a valid and reliable additional method inthe assessment toolbox. It provides an opportunity to test a large sample ofclinical skills in a short time with little cost and resources. The specific skillsof clinical diagnosis for CIVA include pattern recognition of signs, interpretationand decision making as well as medical writing skills such as referral letters andprescription writing. The topics chosen were based upon the defined curriculumoutcome competencies, in line with GMC learning outcomes [11] and the assessment of these outcomes [7], uses the principles of the Millers Pyramid [12] in moving assessments higher up the hierarchical scale towards therealism of the clinical situation [5]. This model is so far useful with regard to assessment using differentassessment instruments that are appropriate at each level of the pyramid. The CIVAlike the OSCE is more appropriate to the second and third level of the pyramid.

CIVAs and OSCEs assess similar constructs of clinical reasoning and decision making,but the value of CIVA is the increased size and number of tasks in differentsimulation contexts and presentations such as “emergencies” which aredifficult and more expensive to simulate in OSCEs, such as seizures, severe asthma,acute cardiac conditions and trauma. However, CIVA is not an alternative tool toOSCE, as it is not ideal for assessing communication skills, history taking,physical examination or procedural skills. The main advantage of CIVA is thestandardized mean of assessing a large number of students in a short time.

The high correlation with OSCE (r = 0.83, p <0.001) indicatessimilarity of constructs assessed including clinical reasoning and decision making.It could also be due to a priming effect because both tests were administered on thesame day. However, the low correlations with written (r = 0.36, p<0.001) and DOCEE grades (r = 0.35, p <0.001) reflect thedifferences of the constructs measured. The written exams are an effective mean ofassessing knowledge and other domains, and the DOCEE evaluates the holistic approachto patient’s care.

Although CIVA is cheaper to administer, it does require significant time for gettingit right before adding the station on the database and for marking the answerbooklets.

Conclusion

CIVA provides an excellent reliable and valid tool to be added to the assessment toolbox in order to enhance the overall assessment of how competent medical students areat various phases of the curriculum. It assesses a large number of clinical signsbased on real patients. When well designed and enhanced with rich text and a realpatient’s scenario, it can replace several OSCE stations and overcome somelogistical issues encountered during OSCEs and DOCEEs. It is cheaper and requiresmuch less personnel than the OSCE. However, it needs significant time to review inorder to improve content validity and to mark the answer books.