New virtual case-based assessment method for decision making in undergraduate students: a scale development and validation

Abstract

Background

There are many Internet forums where patients can ask medical question and get an answer from doctors. The aim of this study was to develop and validate the rating scale for the assessment of decision-making skills in undergraduate medical students based on such Internet questions.

Methods

This cross-sectional observational study carried out in Medical School of University of Maribor in Slovenia during the family medicine teaching course in the fourth study year. The sample consisted of 159 students. The source of data were the scoring sheets of the students’ reports, assesses by two independent researchers. The assessment tool consisted of 10 items on a five-point Likert scale.

Results

Our final sample consisted of 147 (92.5%) students’ reports. The ICC for matching of the final total scores on assessment tool of both assessors was 0.742. Cronbach’s alpha of the assessment scale was 0.848. Factor analysis revealed four factors: initial assessment, physical examination planning, planning patient management and patient education/involvement.

Conclusions

This assessment tool can be used for assessing undergraduate students’ decision-making based on medical questions asked by real patients in a virtual setting.

Background

Family practice demands continuous decision-making also in patients with early presentation of undifferentiated and low intensity of symptoms. Traditional medical teaching prepares students for their medical carriers by studying diseases in their full and typical clinical presentations as these represent the majority of hospitalised patients. Family medicine teaching, on the other hand, can offer complementary knowledge, skills and attitudes regarding comprehensive management of patients’ complaints and management of diseases in their early stages often combined with somatoform and medically unexplained symptoms [1, 2].

Case-based discussion is a teaching method, which can also stimulate learning of comprehensive patients’ management, patient involvement and information provision [3] and is very useful in family medicine teaching. Because often there are not enough live family practice patients appropriate for covering all educational aspects of medical education, simulated patients or teaching models are not easy available and affordable [4] and it is also difficult to get a comprehensive verbatim transcript of what was asked and said during everyday office consultations, we have to look for alternatives for case-based discussions [3].

Nowadays, more and more teachers use virtual patients or virtual learning modules to stimulate learning of clinical decision-making skills in undergraduate and postgraduate students [5–7]. Namely, virtual patients or e-modules can be easier controlled by teachers and are especially appropriate to enhance their educational value [8, 9].

Case-based discussion evolved as a tool in formative assessment in specialty training [10]. Furthermore, virtual patients and e-modules emerged as tools in assessment as well [11]. Previous studies have shown high acceptance of such educational and assessment methods by students and teachers [5, 7, 12] and have been acknowledged for a good match with classic teaching and assessment methods [6, 11]. However, the assessment of decision-making based on virtual patients is poorly studied and the validity and reliability of methods may vary considerably [13, 14].

In the era of modern technology, patients use more and more new sources of information, especially Internet [15, 16], often in a form of an always open “walk-in” office. Although the quality of such information may be questionable and is difficult to judge its scientific value, it still can be regarded as a valuable source of patient problems which are presented early and in an undefined way which is typical for family medicine. Answering these kinds of virtual patients’ problems presents an additional challenge not only for doctors but also for students, because an important part of consultation, a non-verbal communication, is missing. Also, the students are not able to ask additional questions neither can perform any physical examination. So, one must rely solely on decision-making skills based on history provided by patients in the introductory lines.

Such virtual patients’ problems were the source of patient material for our study.

At the Maribor University, undergraduate study of medicine lasts for six years. Family medicine is taught in the fourth and sixth year of study (seventh and 11^th semester) and they are mandatory courses. The curriculum in the seventh semester for family medicine consists of lectures, seminar work and individual student assignments. One of the assignments is also a problem-based learning with virtual clinical case scenarios (see below) [17].

The aim of this study was to develop and test the rating scale for the assessment of decision-making skills in undergraduate medical students.

Methods

Study design and settings

This was a cross-sectional observational study carried out in Medical School of University of Maribor in Slovenia during the teaching of family medicine in the fourth study year. The study was approved by the ethics committee of the Department of Family Medicine, Maribor Medical School.

The study followed principles in the Declaration of Helsinki.

Sample

A sample consisted of all students who attended classes in family medicine in the study years 2009/2010 and 2010/2011 (N = 159). The inclusion criteria were: a student of the fourth-year regularly attending classes in family medicine in the study year 2009/2010 and 2010/2011, regular attendance to problem-based learning with virtual cases exercises and attendance to the final assessment of problem-based learning with virtual cases.

Problem-based learning with virtual clinical cases

This assignment takes place at the end of the seventh semester and takes three hours. Students get various virtual clinical cases and are asked to write their decisions, counselling, referrals, interventions, etc. The following teaching methods are used: lecture, small group work, one-to-one teaching and discussion. At the end of this exercise, each student is given its own virtual clinical case scenario and has to solve it and write a short report in a predefined format. The general instruction for students is that they have to solve these problems as if a patient would come to their practice with the same question as stated in the virtual case scenario. This report is than assessed by one teacher who gives the student a mark. The aims of these assignments are: teaching of practical aspect of patients’ management in family medicine, teaching of primary care approach to patients, holistic management, comprehensive management, patient involvement strategies and teaching of the approach to patients without clinical examination.

Virtual cases are selected by teachers on the following criteria: new presentation of a problem/symptom, relatively detailed description of the problem and at least some information on the patient, not a clear cut problem, but including some vague or psychological symptomatology, which makes possible a broad spectrum of diagnoses, and demands holistic and comprehensive management of the patient problem(s). Positive side of virtual cases is, that are all already written, written in patient words and do not damage any ethical perspective of breaking confidentiality.

The virtual clinical cases are taken from the freely available e-forum http://med.over.net/forum5/list.php?4. This is a forum on family medicine where questions can be asked by registered users and is moderated by a specialist in family medicine. The teachers themselves choose the appropriate virtual clinical cases and provide them to students.

Data collection

Source of data for this study were the scoring sheets of the reports of all students in the sample. A tool for scoring (assessment tool) was developed by teachers themselves and based on the two theories of consultation between doctor and patient. The first one was developed by Stott and Davis in 1979 [18] and consists of four parts: management of presenting problems, modification of help-seeking behaviours, management of continuing problems and opportunistic health promotion. The second one was developed by Cohen-Cole and Bird in 1989 [19] and consists of three parts: gathering data to understand the patient’s problems, developing rapport and responding to patient’s emotions and patient education and motivation. We could not chose a single theory to base our assessment tool on due to the need to adapt to the virtual nature of clinical cases, due to the fact that the students were not able to talk to the patients who wrote the clinical questions, and to allow for primary care orientation, comprehensive and holistic management as defined in European definition of general practice.

Each student’s report was independently assessed by two teachers (ZKK and JK) by the assessment tool presented below. Then, the mean value of scores of each item was calculated. The final dataset for statistical analysis therefore consisted of the two teachers’ mean scores for each item.

Assessment tool

An assessment tool consisted of 10 items: a student 1) asked the appropriate questions regarding patient’s history; 2) proposed the appropriate differential diagnoses; 3) proposed the appropriate clinical examination; 4) proposed the appropriate investigations; 5) proposed the appropriate referrals; 6) proposed the appropriate management; 7) explained the planned investigations and referrals to patient; 8) explained the planned management to patient; 9) explained the probable diagnosis to patient; 10) gave the patient instructions on self-management at home. All items could be graded on a five-point Likert scale: from one (not acceptable) to five (excellent). Maximum total score of the assessment scale was 50 point and minimal total score point was 5 points.

Data analysis

The data were analysed with the SPSS 13.0 package (SPSS Inc., Chicago, IL). We calculated the descriptive data. For the purpose of determining the level of scores’ matching of both assessors, we calculated the intraclass correlation coefficient (ICC). We calculated the reliability of the assessment tool by Cronbach’s alpha coefficient. We performed factor analysis with principal component analysis extraction method and Quartimax with Keiser normalization rotation method.

Results

Out of 159 students, 12 did not complete the assignment. So, the final sample consisted of 147 (92.5%) students’ reports. There were 82 (55.8%) reports from the study year 2009/2010 and 65 (44.2%) from the study year 2010/2011. There were 95 (64.6%) female students as authors of the reports.

Matching of assessment

The ICC for matching of the final total scores on assessment tool of both assessors was 0.742 (Table 1).

Table 1 Matching of scores on assessment tool of both assessors

Reliability

Cronbach’s alpha of the assessment scale was 0.848 (Table 2).

Table 2 Assessment tool: item analysis

Factor analysis

Factor analysis revealed four factors: initial assessment, physical examination planning, planning patient management and patient education/involvement (Table 3). It explained 78.1% variance: factor 1 – planning patient management – explained 28.7%, factor 2 – patient education/involvement – 25.2%, factor 3 – initial assessment – 13.3% and factor 4 – physical examination planning – 10.9% variance.

Table 3 Assessment tool: factor analysis*

Discussion

The tool for the assessment of virtual case-based decision-making showed good psychometric characteristics by good interclass correlations and high Cronbach’s alpha. Factor analysis revealed four factors, which are essential to any consultation: initial assessment, physical examination planning, planning patient management and patient education/involvement.

When developing this assessment toll, we used two consultation theories [18, 19]. The factors that emerged from our analysis correlate well with the factors from these two theoretical frameworks and also with other consultation theories (Table 4) [18, 20–22]. Therefore, we can claim that our tool is comprehensive and cover all essential parts of consultation in family medicine: establishing of initial contact with patient and developing a range of possible early diagnoses [23], physical examination planning and also very important part of consultation – patient involving in treatment (planning, explaining of further procedures, giving advices on self-management at home, appropriate actions in case of health status deterioration and rational use of health services in future). These consultation parts are a basis of each consultation and also present one of the educational aims of family medicine course [24] and now can be assessed by using this newly developed tool.

Table 4 Comparison of different consultation models according to the factors found in the present study

Many tools have been developed for the assessment of consultation, for direct observation of clinical skills performance, mainly on the level of specialty training [25]. However, only few of them have been thoroughly evaluated and tested [25]. Also, many of them are focused mainly on the assessment of clinical skills and management of presenting problems. Only few of them included also the assessment of patient involvement in consultation, preventive activities provided by doctors, modification of health-care utilization practices and self-management advices given [25]. These shortcomings have been overcome by our newly developed tool which, in addition, also showed that students performed poorly in some of these tasks.

There have also been several tools for assessment of students developed based on virtual cases or virtual patients [6, 7, 9, 11, 14]. A study from USA [11] assessed a class of 155 second year medical students to evaluate problem-based learning using a virtual patient. The results showed good knowledge of students but the study did not evaluate any assessment tool. Another study from Sweden [6] compared the assessment results between virtual patient simulation and regular course exam and found superior results in the group of students with virtual patients. However, the study did not provide any evaluation of the assessment tools. Yet another study from Germany [14] used a case-based online assessment tool by simulating consultations with virtual patients. This tool proved to be both valid and reliable and highly correlated to the results of a written exam. It seems that virtual patients and virtual clinical cases are often used in medical teaching but lack the reliable and valid assessment tools.

This study put forward a new method of teaching and assessing undergraduate students – the medical question asked by real patients in virtual settings without the possibility of further exploration of patients. This medical question was the only peace of information given to students. Their answers were assessed by the newly developed assessment tool. We demonstrated that it can be used for assessing undergraduate students’ decision-making in virtual cases. Based on this tool, students can be reliably assessed by only one assessor. Although not included in our study, this tool could be used when assessing the students’ consultations and decision-making also in real patients as it covers all essential parts of consultation. It could also be used to assess written assignments where students have to report on patient management and/or decision making. In formative assessment, it could also be used for self-assessment of students [5].

The main strength of this study is good matching of the assessment scores of both assessors. Similar studies [26, 27] on portfolio assessment also showed good matching between independent assessors and their authors therefore suggested that there is a possibility of reducing the number of assessors while maintaining a sufficient level of reliability. This holds true also for our instrument which can result in an increased feasibility of this newly developed assessment tool for both students and assessors. Another strength is also the fact that the tool is based on consultation theories which gives it additional validity besides good validity and reliability showed in our study.

There are also several limitations of the study. The main limitation is that we did not evaluate the educational value of such virtual cases. The students were not assessed before and after the appropriate education took place. This was due to the inability of the researchers to do so because of the organizational problems in the course. However, the aim of this study was not to test the educational value of this tool but to validate it. Another limitation is also the fact that we did not compare the assessment based on this tool to the classical assessment used in our students.

Further studies should evaluate the educational value of this tool, test it also in real-life clinical cases and compare it to other assessment methods. The tool would be given additional value also by testing it in different learners’ populations (i.e. trainees, continuous medical education) and by testing it among the medical students when they face real patients in FPs’ offices.

Conclusions

Medical questions by patients on Internet health forums can be used for assessing clinical decision-making in undergraduate medical students. A tool for assessing clinical decision-making in undergraduate students using medical questions on Internet health forums, presented in this paper, is a valid and reliable tool. It also can be used by only one assessor.