Research article | Open | Open Peer Review | Published:
New virtual case-based assessment method for decision making in undergraduate students: a scale development and validation
BMC Medical Educationvolume 13, Article number: 160 (2013)
There are many Internet forums where patients can ask medical question and get an answer from doctors. The aim of this study was to develop and validate the rating scale for the assessment of decision-making skills in undergraduate medical students based on such Internet questions.
This cross-sectional observational study carried out in Medical School of University of Maribor in Slovenia during the family medicine teaching course in the fourth study year. The sample consisted of 159 students. The source of data were the scoring sheets of the students’ reports, assesses by two independent researchers. The assessment tool consisted of 10 items on a five-point Likert scale.
Our final sample consisted of 147 (92.5%) students’ reports. The ICC for matching of the final total scores on assessment tool of both assessors was 0.742. Cronbach’s alpha of the assessment scale was 0.848. Factor analysis revealed four factors: initial assessment, physical examination planning, planning patient management and patient education/involvement.
This assessment tool can be used for assessing undergraduate students’ decision-making based on medical questions asked by real patients in a virtual setting.
Family practice demands continuous decision-making also in patients with early presentation of undifferentiated and low intensity of symptoms. Traditional medical teaching prepares students for their medical carriers by studying diseases in their full and typical clinical presentations as these represent the majority of hospitalised patients. Family medicine teaching, on the other hand, can offer complementary knowledge, skills and attitudes regarding comprehensive management of patients’ complaints and management of diseases in their early stages often combined with somatoform and medically unexplained symptoms [1, 2].
Case-based discussion is a teaching method, which can also stimulate learning of comprehensive patients’ management, patient involvement and information provision  and is very useful in family medicine teaching. Because often there are not enough live family practice patients appropriate for covering all educational aspects of medical education, simulated patients or teaching models are not easy available and affordable  and it is also difficult to get a comprehensive verbatim transcript of what was asked and said during everyday office consultations, we have to look for alternatives for case-based discussions .
Nowadays, more and more teachers use virtual patients or virtual learning modules to stimulate learning of clinical decision-making skills in undergraduate and postgraduate students [5–7]. Namely, virtual patients or e-modules can be easier controlled by teachers and are especially appropriate to enhance their educational value [8, 9].
Case-based discussion evolved as a tool in formative assessment in specialty training . Furthermore, virtual patients and e-modules emerged as tools in assessment as well . Previous studies have shown high acceptance of such educational and assessment methods by students and teachers [5, 7, 12] and have been acknowledged for a good match with classic teaching and assessment methods [6, 11]. However, the assessment of decision-making based on virtual patients is poorly studied and the validity and reliability of methods may vary considerably [13, 14].
In the era of modern technology, patients use more and more new sources of information, especially Internet [15, 16], often in a form of an always open “walk-in” office. Although the quality of such information may be questionable and is difficult to judge its scientific value, it still can be regarded as a valuable source of patient problems which are presented early and in an undefined way which is typical for family medicine. Answering these kinds of virtual patients’ problems presents an additional challenge not only for doctors but also for students, because an important part of consultation, a non-verbal communication, is missing. Also, the students are not able to ask additional questions neither can perform any physical examination. So, one must rely solely on decision-making skills based on history provided by patients in the introductory lines.
Such virtual patients’ problems were the source of patient material for our study.
At the Maribor University, undergraduate study of medicine lasts for six years. Family medicine is taught in the fourth and sixth year of study (seventh and 11th semester) and they are mandatory courses. The curriculum in the seventh semester for family medicine consists of lectures, seminar work and individual student assignments. One of the assignments is also a problem-based learning with virtual clinical case scenarios (see below) .
The aim of this study was to develop and test the rating scale for the assessment of decision-making skills in undergraduate medical students.
Study design and settings
This was a cross-sectional observational study carried out in Medical School of University of Maribor in Slovenia during the teaching of family medicine in the fourth study year. The study was approved by the ethics committee of the Department of Family Medicine, Maribor Medical School.
The study followed principles in the Declaration of Helsinki.
A sample consisted of all students who attended classes in family medicine in the study years 2009/2010 and 2010/2011 (N = 159). The inclusion criteria were: a student of the fourth-year regularly attending classes in family medicine in the study year 2009/2010 and 2010/2011, regular attendance to problem-based learning with virtual cases exercises and attendance to the final assessment of problem-based learning with virtual cases.
Problem-based learning with virtual clinical cases
This assignment takes place at the end of the seventh semester and takes three hours. Students get various virtual clinical cases and are asked to write their decisions, counselling, referrals, interventions, etc. The following teaching methods are used: lecture, small group work, one-to-one teaching and discussion. At the end of this exercise, each student is given its own virtual clinical case scenario and has to solve it and write a short report in a predefined format. The general instruction for students is that they have to solve these problems as if a patient would come to their practice with the same question as stated in the virtual case scenario. This report is than assessed by one teacher who gives the student a mark. The aims of these assignments are: teaching of practical aspect of patients’ management in family medicine, teaching of primary care approach to patients, holistic management, comprehensive management, patient involvement strategies and teaching of the approach to patients without clinical examination.
Virtual cases are selected by teachers on the following criteria: new presentation of a problem/symptom, relatively detailed description of the problem and at least some information on the patient, not a clear cut problem, but including some vague or psychological symptomatology, which makes possible a broad spectrum of diagnoses, and demands holistic and comprehensive management of the patient problem(s). Positive side of virtual cases is, that are all already written, written in patient words and do not damage any ethical perspective of breaking confidentiality.
The virtual clinical cases are taken from the freely available e-forum http://med.over.net/forum5/list.php?4. This is a forum on family medicine where questions can be asked by registered users and is moderated by a specialist in family medicine. The teachers themselves choose the appropriate virtual clinical cases and provide them to students.
Source of data for this study were the scoring sheets of the reports of all students in the sample. A tool for scoring (assessment tool) was developed by teachers themselves and based on the two theories of consultation between doctor and patient. The first one was developed by Stott and Davis in 1979  and consists of four parts: management of presenting problems, modification of help-seeking behaviours, management of continuing problems and opportunistic health promotion. The second one was developed by Cohen-Cole and Bird in 1989  and consists of three parts: gathering data to understand the patient’s problems, developing rapport and responding to patient’s emotions and patient education and motivation. We could not chose a single theory to base our assessment tool on due to the need to adapt to the virtual nature of clinical cases, due to the fact that the students were not able to talk to the patients who wrote the clinical questions, and to allow for primary care orientation, comprehensive and holistic management as defined in European definition of general practice.
Each student’s report was independently assessed by two teachers (ZKK and JK) by the assessment tool presented below. Then, the mean value of scores of each item was calculated. The final dataset for statistical analysis therefore consisted of the two teachers’ mean scores for each item.
An assessment tool consisted of 10 items: a student 1) asked the appropriate questions regarding patient’s history; 2) proposed the appropriate differential diagnoses; 3) proposed the appropriate clinical examination; 4) proposed the appropriate investigations; 5) proposed the appropriate referrals; 6) proposed the appropriate management; 7) explained the planned investigations and referrals to patient; 8) explained the planned management to patient; 9) explained the probable diagnosis to patient; 10) gave the patient instructions on self-management at home. All items could be graded on a five-point Likert scale: from one (not acceptable) to five (excellent). Maximum total score of the assessment scale was 50 point and minimal total score point was 5 points.
The data were analysed with the SPSS 13.0 package (SPSS Inc., Chicago, IL). We calculated the descriptive data. For the purpose of determining the level of scores’ matching of both assessors, we calculated the intraclass correlation coefficient (ICC). We calculated the reliability of the assessment tool by Cronbach’s alpha coefficient. We performed factor analysis with principal component analysis extraction method and Quartimax with Keiser normalization rotation method.
Out of 159 students, 12 did not complete the assignment. So, the final sample consisted of 147 (92.5%) students’ reports. There were 82 (55.8%) reports from the study year 2009/2010 and 65 (44.2%) from the study year 2010/2011. There were 95 (64.6%) female students as authors of the reports.
Matching of assessment
The ICC for matching of the final total scores on assessment tool of both assessors was 0.742 (Table 1).
Cronbach’s alpha of the assessment scale was 0.848 (Table 2).
Factor analysis revealed four factors: initial assessment, physical examination planning, planning patient management and patient education/involvement (Table 3). It explained 78.1% variance: factor 1 – planning patient management – explained 28.7%, factor 2 – patient education/involvement – 25.2%, factor 3 – initial assessment – 13.3% and factor 4 – physical examination planning – 10.9% variance.
The tool for the assessment of virtual case-based decision-making showed good psychometric characteristics by good interclass correlations and high Cronbach’s alpha. Factor analysis revealed four factors, which are essential to any consultation: initial assessment, physical examination planning, planning patient management and patient education/involvement.
When developing this assessment toll, we used two consultation theories [18, 19]. The factors that emerged from our analysis correlate well with the factors from these two theoretical frameworks and also with other consultation theories (Table 4) [18, 20–22]. Therefore, we can claim that our tool is comprehensive and cover all essential parts of consultation in family medicine: establishing of initial contact with patient and developing a range of possible early diagnoses , physical examination planning and also very important part of consultation – patient involving in treatment (planning, explaining of further procedures, giving advices on self-management at home, appropriate actions in case of health status deterioration and rational use of health services in future). These consultation parts are a basis of each consultation and also present one of the educational aims of family medicine course  and now can be assessed by using this newly developed tool.
Many tools have been developed for the assessment of consultation, for direct observation of clinical skills performance, mainly on the level of specialty training . However, only few of them have been thoroughly evaluated and tested . Also, many of them are focused mainly on the assessment of clinical skills and management of presenting problems. Only few of them included also the assessment of patient involvement in consultation, preventive activities provided by doctors, modification of health-care utilization practices and self-management advices given . These shortcomings have been overcome by our newly developed tool which, in addition, also showed that students performed poorly in some of these tasks.
There have also been several tools for assessment of students developed based on virtual cases or virtual patients [6, 7, 9, 11, 14]. A study from USA  assessed a class of 155 second year medical students to evaluate problem-based learning using a virtual patient. The results showed good knowledge of students but the study did not evaluate any assessment tool. Another study from Sweden  compared the assessment results between virtual patient simulation and regular course exam and found superior results in the group of students with virtual patients. However, the study did not provide any evaluation of the assessment tools. Yet another study from Germany  used a case-based online assessment tool by simulating consultations with virtual patients. This tool proved to be both valid and reliable and highly correlated to the results of a written exam. It seems that virtual patients and virtual clinical cases are often used in medical teaching but lack the reliable and valid assessment tools.
This study put forward a new method of teaching and assessing undergraduate students – the medical question asked by real patients in virtual settings without the possibility of further exploration of patients. This medical question was the only peace of information given to students. Their answers were assessed by the newly developed assessment tool. We demonstrated that it can be used for assessing undergraduate students’ decision-making in virtual cases. Based on this tool, students can be reliably assessed by only one assessor. Although not included in our study, this tool could be used when assessing the students’ consultations and decision-making also in real patients as it covers all essential parts of consultation. It could also be used to assess written assignments where students have to report on patient management and/or decision making. In formative assessment, it could also be used for self-assessment of students .
The main strength of this study is good matching of the assessment scores of both assessors. Similar studies [26, 27] on portfolio assessment also showed good matching between independent assessors and their authors therefore suggested that there is a possibility of reducing the number of assessors while maintaining a sufficient level of reliability. This holds true also for our instrument which can result in an increased feasibility of this newly developed assessment tool for both students and assessors. Another strength is also the fact that the tool is based on consultation theories which gives it additional validity besides good validity and reliability showed in our study.
There are also several limitations of the study. The main limitation is that we did not evaluate the educational value of such virtual cases. The students were not assessed before and after the appropriate education took place. This was due to the inability of the researchers to do so because of the organizational problems in the course. However, the aim of this study was not to test the educational value of this tool but to validate it. Another limitation is also the fact that we did not compare the assessment based on this tool to the classical assessment used in our students.
Further studies should evaluate the educational value of this tool, test it also in real-life clinical cases and compare it to other assessment methods. The tool would be given additional value also by testing it in different learners’ populations (i.e. trainees, continuous medical education) and by testing it among the medical students when they face real patients in FPs’ offices.
Medical questions by patients on Internet health forums can be used for assessing clinical decision-making in undergraduate medical students. A tool for assessing clinical decision-making in undergraduate students using medical questions on Internet health forums, presented in this paper, is a valid and reliable tool. It also can be used by only one assessor.
Krzton-Królewiecka A, Svab I, Oleszczyk M, Seifert B, Smithson WH, Windak A: The development of academic family medicine in central and eastern Europe since 1990. BMC Fam Pract. 2013, 14: 37-10.1186/1471-2296-14-37.
Svab I: Education in medicine. Zdrav Var. 2012, 51: 225-226.
Fraser SW, Greenhalgh T: Coping with complexity: educating for capability. BMJ. 2001, 323: 799-803. 10.1136/bmj.323.7316.799.
Tran TQ, Scherpbier A, Van Dalen J, Wright PE: Teacher-made models: the answer for medical skills training in developing countries?. BMC Med Educ. 2012, 12: 98-10.1186/1472-6920-12-98.
Abendroth M, Harendza S, Riemer M: Clinical decision making: a pilot e-learning study. Clin Teach. 2013, 10: 51-55. 10.1111/j.1743-498X.2012.00629.x.
Botezatu M, Hult H, Tessma MK, Fors UG: Virtual patient simulation for learning and assessment: superior results in comparison with regular course exams. Med Teach. 2010, 32: 845-850. 10.3109/01421591003695287.
Gesundheit N, Brutlag P, Youngblood P, Gunning WT, Zary N, Fors U: The use of virtual patients to assess the clinical skills and reasoning of medical students: initial insights on student acceptance. Med Teach. 2009, 31: 739-742. 10.1080/01421590903126489.
Cook DA, Triola MM: Virtual patients: a critical literature review and proposed next steps. Med Educ. 2009, 43: 303-311. 10.1111/j.1365-2923.2008.03286.x.
Cook DA, Erwin PJ, Triola MM: Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010, 85: 1589-1602. 10.1097/ACM.0b013e3181edfe13.
Miller A, Archer J: Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2010, 341: c5064-10.1136/bmj.c5064.
Gunning WT, Fors UG: Virtual patients for assessment of medical student ability to integrate clinical and laboratory data to develop differential diagnoses: comparison of results of exams with/without time constraints. Med Teach. 2012, 34: e222-e228. 10.3109/0142159X.2012.642830.
Samuel SE, Olle OB, Peter PH, Daphne DV, Jonas JS, Lars Owe LOD, Uno UF, Nabil NZ: Integrating virtual patients into courses: follow-up seminars and perceived benefit. Med Educ. 2012, 46: 417-425. 10.1111/j.1365-2923.2012.04219.x.
Bateman J, Allen ME, Kidd J, Parsons N, Davies D: Virtual patients design and its effect on clinical reasoning and student experience: a protocol for a randomised factorial multi-centre study. BMC Med Educ. 2012, 12: 62-10.1186/1472-6920-12-62.
Waldmann UM, Gulich MS, Zeitler HP: Virtual patients for assessing medical students - important aspects when considering the introduction of a new assessment format. Med Teach. 2008, 30: 17-24. 10.1080/01421590701758616.
Klemenc-Ketis Z, Kersnik J: Health seeking behaviour in general population with psychological symptoms. Psychiatr Danub. 2013, [in print]
Klemenc-Ketis Z, Kersnik J: Seeking health advice on the internet in patients with health problems: a cross-sectional population study in Slovenia. Inform Health Soc Care. 2013, 38: 280-290. 10.3109/17538157.2013.764306.
Klemenc-Ketis Z, Kersnik J: The role of the European academy of teachers in general practice and family medicine in family medicine education in Europe - the experience of the university of Maribor. Acta Med Acad. 2012, 41: 80-87. 10.5644/ama2006-124.41.
Stott NC, Davis RH: The exceptional potential in each primary care consultation. J R Coll Gen Pract. 1979, 29: 201-205.
Cohen-Cole S, Bird J: The Medical Interview: the Three Function Approach. 2000, New York: Mosby
Byrne PS: Long BEL (Eds.): Doctors talking to patients. 1976, London: RCGP Publications
Pendleton D: The consultation. An approach to teaching and learning. 1984, Oxford: Oxford Medical Publications
Kurtz S, Silverman J, Benson J, Draper J: Marrying content and process in clinical method teaching: enhancing the Calgary-Cambridge guides. Acad Med. 2003, 78: 802-809. 10.1097/00001888-200308000-00011.
Kaplan RM, Frosch DL: Decision making in medicine and health care. Annu Rev Clin Psychol. 2005, 1: 525-556. 10.1146/annurev.clinpsy.1.102803.144118.
Allen J, Price E, Svab I, Windak A: EURACT: European academy of teachers in general practice and family medicine. Eur J Gen Pract. 2012, 18: 124-125. 10.3109/13814788.2012.681366.
Kogan JR, Holmboe ES, Hauer KE: Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009, 302: 1316-1326. 10.1001/jama.2009.1365.
Michels NR, Driessen EW, Muijtjens AM, Van Gaal LF, Bossaert LL, De Winter BY: Portfolio assessment during medical internships: how to obtain a reliable and feasible assessment procedure?. Educ Health (Abingdon). 2009, 22: 313.
Rees CE, Sheard CE: The reliability of assessment criteria for undergraduate medical students’ communication skills portfolios: the Nottingham experience. Med Educ. 2004, 38: 138-144. 10.1111/j.1365-2923.2004.01744.x.
The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/13/160/prepub
Authors declare they have no competing interests.
ZKK and JK conceived the study, participated in its design and coordinated it. ZKK performed the statistical analysis. ZKK and JK interpreted the results. ZKK drafted the manuscript and JK approved its final version.