Skip to content

Advertisement

BMC Medical Education

Open Access
Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

New virtual case-based assessment method for decision making in undergraduate students: a scale development and validation

BMC Medical Education201313:160

https://doi.org/10.1186/1472-6920-13-160

Received: 17 August 2013

Accepted: 28 November 2013

Published: 3 December 2013

Abstract

Background

There are many Internet forums where patients can ask medical question and get an answer from doctors. The aim of this study was to develop and validate the rating scale for the assessment of decision-making skills in undergraduate medical students based on such Internet questions.

Methods

This cross-sectional observational study carried out in Medical School of University of Maribor in Slovenia during the family medicine teaching course in the fourth study year. The sample consisted of 159 students. The source of data were the scoring sheets of the students’ reports, assesses by two independent researchers. The assessment tool consisted of 10 items on a five-point Likert scale.

Results

Our final sample consisted of 147 (92.5%) students’ reports. The ICC for matching of the final total scores on assessment tool of both assessors was 0.742. Cronbach’s alpha of the assessment scale was 0.848. Factor analysis revealed four factors: initial assessment, physical examination planning, planning patient management and patient education/involvement.

Conclusions

This assessment tool can be used for assessing undergraduate students’ decision-making based on medical questions asked by real patients in a virtual setting.

Keywords

Virtual PatientUndergraduate Medical StudentMedical QuestionVirtual SettingVirtual Case

Background

Family practice demands continuous decision-making also in patients with early presentation of undifferentiated and low intensity of symptoms. Traditional medical teaching prepares students for their medical carriers by studying diseases in their full and typical clinical presentations as these represent the majority of hospitalised patients. Family medicine teaching, on the other hand, can offer complementary knowledge, skills and attitudes regarding comprehensive management of patients’ complaints and management of diseases in their early stages often combined with somatoform and medically unexplained symptoms [1, 2].

Case-based discussion is a teaching method, which can also stimulate learning of comprehensive patients’ management, patient involvement and information provision [3] and is very useful in family medicine teaching. Because often there are not enough live family practice patients appropriate for covering all educational aspects of medical education, simulated patients or teaching models are not easy available and affordable [4] and it is also difficult to get a comprehensive verbatim transcript of what was asked and said during everyday office consultations, we have to look for alternatives for case-based discussions [3].

Nowadays, more and more teachers use virtual patients or virtual learning modules to stimulate learning of clinical decision-making skills in undergraduate and postgraduate students [57]. Namely, virtual patients or e-modules can be easier controlled by teachers and are especially appropriate to enhance their educational value [8, 9].

Case-based discussion evolved as a tool in formative assessment in specialty training [10]. Furthermore, virtual patients and e-modules emerged as tools in assessment as well [11]. Previous studies have shown high acceptance of such educational and assessment methods by students and teachers [5, 7, 12] and have been acknowledged for a good match with classic teaching and assessment methods [6, 11]. However, the assessment of decision-making based on virtual patients is poorly studied and the validity and reliability of methods may vary considerably [13, 14].

In the era of modern technology, patients use more and more new sources of information, especially Internet [15, 16], often in a form of an always open “walk-in” office. Although the quality of such information may be questionable and is difficult to judge its scientific value, it still can be regarded as a valuable source of patient problems which are presented early and in an undefined way which is typical for family medicine. Answering these kinds of virtual patients’ problems presents an additional challenge not only for doctors but also for students, because an important part of consultation, a non-verbal communication, is missing. Also, the students are not able to ask additional questions neither can perform any physical examination. So, one must rely solely on decision-making skills based on history provided by patients in the introductory lines.

Such virtual patients’ problems were the source of patient material for our study.

At the Maribor University, undergraduate study of medicine lasts for six years. Family medicine is taught in the fourth and sixth year of study (seventh and 11th semester) and they are mandatory courses. The curriculum in the seventh semester for family medicine consists of lectures, seminar work and individual student assignments. One of the assignments is also a problem-based learning with virtual clinical case scenarios (see below) [17].

The aim of this study was to develop and test the rating scale for the assessment of decision-making skills in undergraduate medical students.

Methods

Study design and settings

This was a cross-sectional observational study carried out in Medical School of University of Maribor in Slovenia during the teaching of family medicine in the fourth study year. The study was approved by the ethics committee of the Department of Family Medicine, Maribor Medical School.

The study followed principles in the Declaration of Helsinki.

Sample

A sample consisted of all students who attended classes in family medicine in the study years 2009/2010 and 2010/2011 (N = 159). The inclusion criteria were: a student of the fourth-year regularly attending classes in family medicine in the study year 2009/2010 and 2010/2011, regular attendance to problem-based learning with virtual cases exercises and attendance to the final assessment of problem-based learning with virtual cases.

Problem-based learning with virtual clinical cases

This assignment takes place at the end of the seventh semester and takes three hours. Students get various virtual clinical cases and are asked to write their decisions, counselling, referrals, interventions, etc. The following teaching methods are used: lecture, small group work, one-to-one teaching and discussion. At the end of this exercise, each student is given its own virtual clinical case scenario and has to solve it and write a short report in a predefined format. The general instruction for students is that they have to solve these problems as if a patient would come to their practice with the same question as stated in the virtual case scenario. This report is than assessed by one teacher who gives the student a mark. The aims of these assignments are: teaching of practical aspect of patients’ management in family medicine, teaching of primary care approach to patients, holistic management, comprehensive management, patient involvement strategies and teaching of the approach to patients without clinical examination.

Virtual cases are selected by teachers on the following criteria: new presentation of a problem/symptom, relatively detailed description of the problem and at least some information on the patient, not a clear cut problem, but including some vague or psychological symptomatology, which makes possible a broad spectrum of diagnoses, and demands holistic and comprehensive management of the patient problem(s). Positive side of virtual cases is, that are all already written, written in patient words and do not damage any ethical perspective of breaking confidentiality.

The virtual clinical cases are taken from the freely available e-forum http://med.over.net/forum5/list.php?4. This is a forum on family medicine where questions can be asked by registered users and is moderated by a specialist in family medicine. The teachers themselves choose the appropriate virtual clinical cases and provide them to students.

Data collection

Source of data for this study were the scoring sheets of the reports of all students in the sample. A tool for scoring (assessment tool) was developed by teachers themselves and based on the two theories of consultation between doctor and patient. The first one was developed by Stott and Davis in 1979 [18] and consists of four parts: management of presenting problems, modification of help-seeking behaviours, management of continuing problems and opportunistic health promotion. The second one was developed by Cohen-Cole and Bird in 1989 [19] and consists of three parts: gathering data to understand the patient’s problems, developing rapport and responding to patient’s emotions and patient education and motivation. We could not chose a single theory to base our assessment tool on due to the need to adapt to the virtual nature of clinical cases, due to the fact that the students were not able to talk to the patients who wrote the clinical questions, and to allow for primary care orientation, comprehensive and holistic management as defined in European definition of general practice.

Each student’s report was independently assessed by two teachers (ZKK and JK) by the assessment tool presented below. Then, the mean value of scores of each item was calculated. The final dataset for statistical analysis therefore consisted of the two teachers’ mean scores for each item.

Assessment tool

An assessment tool consisted of 10 items: a student 1) asked the appropriate questions regarding patient’s history; 2) proposed the appropriate differential diagnoses; 3) proposed the appropriate clinical examination; 4) proposed the appropriate investigations; 5) proposed the appropriate referrals; 6) proposed the appropriate management; 7) explained the planned investigations and referrals to patient; 8) explained the planned management to patient; 9) explained the probable diagnosis to patient; 10) gave the patient instructions on self-management at home. All items could be graded on a five-point Likert scale: from one (not acceptable) to five (excellent). Maximum total score of the assessment scale was 50 point and minimal total score point was 5 points.

Data analysis

The data were analysed with the SPSS 13.0 package (SPSS Inc., Chicago, IL). We calculated the descriptive data. For the purpose of determining the level of scores’ matching of both assessors, we calculated the intraclass correlation coefficient (ICC). We calculated the reliability of the assessment tool by Cronbach’s alpha coefficient. We performed factor analysis with principal component analysis extraction method and Quartimax with Keiser normalization rotation method.

Results

Out of 159 students, 12 did not complete the assignment. So, the final sample consisted of 147 (92.5%) students’ reports. There were 82 (55.8%) reports from the study year 2009/2010 and 65 (44.2%) from the study year 2010/2011. There were 95 (64.6%) female students as authors of the reports.

Matching of assessment

The ICC for matching of the final total scores on assessment tool of both assessors was 0.742 (Table 1).
Table 1

Matching of scores on assessment tool of both assessors

Item

Mean score ± SD (assessor 1)

Mean score ± SD (assessor 2)

ICC

95% lower and higher C. I. for ICC

Student asked the appropriate questions regarding patient’s history

4.0 ± 0.9

4.2 ± 0.9

0.699

0.583, 0.782

Student proposed the appropriate differential diagnoses

4.0 ± 0.9

4.3 ± 0.8

0.782

0.698, 0.842

Student proposed the appropriate clinical examination

3.6 ± 1.2

3.2 ± 1.6

0.693

0.575, 0.778

Student proposed the appropriate investigations

3.8 ± 1.0

3.9 ± 1.2

0.658

0.526, 0.753

Student proposed the appropriate referrals

3.7 ± 1.1

3.7 ± 1.5

0.665

0.535, 0.758

Student proposed the appropriate management

3.9 ± 1.0

3.9 ± 1.2

0.654

0.521, 0.750

Student explained the planned investigations and referrals to patient

3.2 ± 1.3

3.2 ± 1.5

0.720

0.612, 0.798

Student explained the planned management to patient

3.1 ± 1.3

2.8 ± 1.5

0.729

0.624, 0.804

Student explained the probable diagnosis to patient

2.8 ± 1.2

2.5 ± 1.6

0.617

0.470, 0.724

Student gave the patient instructions on self-management at home

3.1 ± 1.2

3.2 ± 1.5

0.604

0.451, 0.714

Total

35.3 ± 7.9

34.8 ± 7.8

0.742

0.642, 0.814

Legend:

SD – standard deviation.

ICC – intraclass correlation coefficient.

C. I. – confidence interval.

Reliability

Cronbach’s alpha of the assessment scale was 0.848 (Table 2).
Table 2

Assessment tool: item analysis

Item

Corrected item-total correlation

Cronbach’s alpha if item deleted

Student asked the appropriate questions regarding patient’s history

0.590

0.833

Student proposed the appropriate differential diagnoses

0.585

0.833

Student proposed the appropriate clinical examination

0.396

0.850

Student proposed the appropriate investigations

0.525

0.850

Student proposed the appropriate referrals

0.642

0.824

Student proposed the appropriate management

0.590

0.830

Student explained the planned investigations and referrals to patient

0.487

0.840

Student explained the planned management to patient

0.610

0.827

Student explained the probable diagnosis to patient

0.605

0.828

Student gave the patient instructions on self-management at home

0.556

0.833

Factor analysis

Factor analysis revealed four factors: initial assessment, physical examination planning, planning patient management and patient education/involvement (Table 3). It explained 78.1% variance: factor 1 – planning patient management – explained 28.7%, factor 2 – patient education/involvement – 25.2%, factor 3 – initial assessment – 13.3% and factor 4 – physical examination planning – 10.9% variance.
Table 3

Assessment tool: factor analysis*

Item

Factor 1 – planning patient management

Factor 2 – patient education/involvement

Factor 3 – initial assessment

Factor 4 – physical examination planning

Student asked the appropriate questions regarding patient’s history

0.488

0.171

0.729

0.077

Student proposed the appropriate differential diagnoses

0.396

0.276

0.772

−0.051

Student proposed the appropriate clinical examination

0.372

0.198

0.006

0.813

Student proposed the appropriate investigations

0.760

−0.027

0.226

0.290

Student proposed the appropriate referrals

0.805

0.194

0.025

0.293

Student proposed the appropriate management

0.795

0.161

0.196

−0.153

Student explained the planned investigations and referrals to patient

−0.016

0.763

0.296

0.224

Student explained the planned management to patient

0.149

0.879

0.107

0.101

Student explained the probable diagnosis to patient

0.317

0.820

−0.058

−0.104

Student gave the patient instructions on self-management at home

0.498

0.528

−0.064

−0.400

*Rotated component matrix using Quartimax method with Kaiser Normalization.

Discussion

The tool for the assessment of virtual case-based decision-making showed good psychometric characteristics by good interclass correlations and high Cronbach’s alpha. Factor analysis revealed four factors, which are essential to any consultation: initial assessment, physical examination planning, planning patient management and patient education/involvement.

When developing this assessment toll, we used two consultation theories [18, 19]. The factors that emerged from our analysis correlate well with the factors from these two theoretical frameworks and also with other consultation theories (Table 4) [18, 2022]. Therefore, we can claim that our tool is comprehensive and cover all essential parts of consultation in family medicine: establishing of initial contact with patient and developing a range of possible early diagnoses [23], physical examination planning and also very important part of consultation – patient involving in treatment (planning, explaining of further procedures, giving advices on self-management at home, appropriate actions in case of health status deterioration and rational use of health services in future). These consultation parts are a basis of each consultation and also present one of the educational aims of family medicine course [24] and now can be assessed by using this newly developed tool.
Table 4

Comparison of different consultation models according to the factors found in the present study

Tool/model

Factor/step in consultation

Factor/step in consultation

Factor/step in consultation

Factor/step in consultation

Our new assessment tool

Initial assessment

Physical examination planning

Planning patient management

Patient education/involvement

Byrne & Long model [20]

The doctor establishes a relationship with the patient.

The doctor conducts a verbal or physical examination or both.

The doctor, or the doctor and the patient or the patient (in that order of probability) considers the condition.

The doctor, and the patient, detail treatment or further investigation.

The doctor either attempts to discover or actually discovers the reason for the patient’s attendance.

Stott & Davis model [18]

Management of Presenting problems.

  

Opportunistic Health Promotion.

Modification of Help Seeking Behaviour.

Pendleton model [21]

Reason for Attending

 

Doctor and patient choose an action for each problem.

Involve patient in management, sharing appropriate responsibility.

Sharing understanding.

Calgary-Cambridge model [22]

Initiating the session

Gathering information

Explanation and planning

Explanation and planning

Gathering information

Many tools have been developed for the assessment of consultation, for direct observation of clinical skills performance, mainly on the level of specialty training [25]. However, only few of them have been thoroughly evaluated and tested [25]. Also, many of them are focused mainly on the assessment of clinical skills and management of presenting problems. Only few of them included also the assessment of patient involvement in consultation, preventive activities provided by doctors, modification of health-care utilization practices and self-management advices given [25]. These shortcomings have been overcome by our newly developed tool which, in addition, also showed that students performed poorly in some of these tasks.

There have also been several tools for assessment of students developed based on virtual cases or virtual patients [6, 7, 9, 11, 14]. A study from USA [11] assessed a class of 155 second year medical students to evaluate problem-based learning using a virtual patient. The results showed good knowledge of students but the study did not evaluate any assessment tool. Another study from Sweden [6] compared the assessment results between virtual patient simulation and regular course exam and found superior results in the group of students with virtual patients. However, the study did not provide any evaluation of the assessment tools. Yet another study from Germany [14] used a case-based online assessment tool by simulating consultations with virtual patients. This tool proved to be both valid and reliable and highly correlated to the results of a written exam. It seems that virtual patients and virtual clinical cases are often used in medical teaching but lack the reliable and valid assessment tools.

This study put forward a new method of teaching and assessing undergraduate students – the medical question asked by real patients in virtual settings without the possibility of further exploration of patients. This medical question was the only peace of information given to students. Their answers were assessed by the newly developed assessment tool. We demonstrated that it can be used for assessing undergraduate students’ decision-making in virtual cases. Based on this tool, students can be reliably assessed by only one assessor. Although not included in our study, this tool could be used when assessing the students’ consultations and decision-making also in real patients as it covers all essential parts of consultation. It could also be used to assess written assignments where students have to report on patient management and/or decision making. In formative assessment, it could also be used for self-assessment of students [5].

The main strength of this study is good matching of the assessment scores of both assessors. Similar studies [26, 27] on portfolio assessment also showed good matching between independent assessors and their authors therefore suggested that there is a possibility of reducing the number of assessors while maintaining a sufficient level of reliability. This holds true also for our instrument which can result in an increased feasibility of this newly developed assessment tool for both students and assessors. Another strength is also the fact that the tool is based on consultation theories which gives it additional validity besides good validity and reliability showed in our study.

There are also several limitations of the study. The main limitation is that we did not evaluate the educational value of such virtual cases. The students were not assessed before and after the appropriate education took place. This was due to the inability of the researchers to do so because of the organizational problems in the course. However, the aim of this study was not to test the educational value of this tool but to validate it. Another limitation is also the fact that we did not compare the assessment based on this tool to the classical assessment used in our students.

Further studies should evaluate the educational value of this tool, test it also in real-life clinical cases and compare it to other assessment methods. The tool would be given additional value also by testing it in different learners’ populations (i.e. trainees, continuous medical education) and by testing it among the medical students when they face real patients in FPs’ offices.

Conclusions

Medical questions by patients on Internet health forums can be used for assessing clinical decision-making in undergraduate medical students. A tool for assessing clinical decision-making in undergraduate students using medical questions on Internet health forums, presented in this paper, is a valid and reliable tool. It also can be used by only one assessor.

Declarations

Authors’ Affiliations

(1)
Department of Family Medicine, Maribor Medical School, Maribor, Slovenia
(2)
Department of Family Medicine, Ljubljana Medical School, Ljubljana, Slovenia

References

  1. Krzton-Królewiecka A, Svab I, Oleszczyk M, Seifert B, Smithson WH, Windak A: The development of academic family medicine in central and eastern Europe since 1990. BMC Fam Pract. 2013, 14: 37-10.1186/1471-2296-14-37.View ArticleGoogle Scholar
  2. Svab I: Education in medicine. Zdrav Var. 2012, 51: 225-226.Google Scholar
  3. Fraser SW, Greenhalgh T: Coping with complexity: educating for capability. BMJ. 2001, 323: 799-803. 10.1136/bmj.323.7316.799.View ArticleGoogle Scholar
  4. Tran TQ, Scherpbier A, Van Dalen J, Wright PE: Teacher-made models: the answer for medical skills training in developing countries?. BMC Med Educ. 2012, 12: 98-10.1186/1472-6920-12-98.View ArticleGoogle Scholar
  5. Abendroth M, Harendza S, Riemer M: Clinical decision making: a pilot e-learning study. Clin Teach. 2013, 10: 51-55. 10.1111/j.1743-498X.2012.00629.x.View ArticleGoogle Scholar
  6. Botezatu M, Hult H, Tessma MK, Fors UG: Virtual patient simulation for learning and assessment: superior results in comparison with regular course exams. Med Teach. 2010, 32: 845-850. 10.3109/01421591003695287.View ArticleGoogle Scholar
  7. Gesundheit N, Brutlag P, Youngblood P, Gunning WT, Zary N, Fors U: The use of virtual patients to assess the clinical skills and reasoning of medical students: initial insights on student acceptance. Med Teach. 2009, 31: 739-742. 10.1080/01421590903126489.View ArticleGoogle Scholar
  8. Cook DA, Triola MM: Virtual patients: a critical literature review and proposed next steps. Med Educ. 2009, 43: 303-311. 10.1111/j.1365-2923.2008.03286.x.View ArticleGoogle Scholar
  9. Cook DA, Erwin PJ, Triola MM: Computerized virtual patients in health professions education: a systematic review and meta-analysis. Acad Med. 2010, 85: 1589-1602. 10.1097/ACM.0b013e3181edfe13.View ArticleGoogle Scholar
  10. Miller A, Archer J: Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2010, 341: c5064-10.1136/bmj.c5064.View ArticleGoogle Scholar
  11. Gunning WT, Fors UG: Virtual patients for assessment of medical student ability to integrate clinical and laboratory data to develop differential diagnoses: comparison of results of exams with/without time constraints. Med Teach. 2012, 34: e222-e228. 10.3109/0142159X.2012.642830.View ArticleGoogle Scholar
  12. Samuel SE, Olle OB, Peter PH, Daphne DV, Jonas JS, Lars Owe LOD, Uno UF, Nabil NZ: Integrating virtual patients into courses: follow-up seminars and perceived benefit. Med Educ. 2012, 46: 417-425. 10.1111/j.1365-2923.2012.04219.x.View ArticleGoogle Scholar
  13. Bateman J, Allen ME, Kidd J, Parsons N, Davies D: Virtual patients design and its effect on clinical reasoning and student experience: a protocol for a randomised factorial multi-centre study. BMC Med Educ. 2012, 12: 62-10.1186/1472-6920-12-62.View ArticleGoogle Scholar
  14. Waldmann UM, Gulich MS, Zeitler HP: Virtual patients for assessing medical students - important aspects when considering the introduction of a new assessment format. Med Teach. 2008, 30: 17-24. 10.1080/01421590701758616.View ArticleGoogle Scholar
  15. Klemenc-Ketis Z, Kersnik J: Health seeking behaviour in general population with psychological symptoms. Psychiatr Danub. 2013, [in print]Google Scholar
  16. Klemenc-Ketis Z, Kersnik J: Seeking health advice on the internet in patients with health problems: a cross-sectional population study in Slovenia. Inform Health Soc Care. 2013, 38: 280-290. 10.3109/17538157.2013.764306.View ArticleGoogle Scholar
  17. Klemenc-Ketis Z, Kersnik J: The role of the European academy of teachers in general practice and family medicine in family medicine education in Europe - the experience of the university of Maribor. Acta Med Acad. 2012, 41: 80-87. 10.5644/ama2006-124.41.View ArticleGoogle Scholar
  18. Stott NC, Davis RH: The exceptional potential in each primary care consultation. J R Coll Gen Pract. 1979, 29: 201-205.Google Scholar
  19. Cohen-Cole S, Bird J: The Medical Interview: the Three Function Approach. 2000, New York: MosbyGoogle Scholar
  20. Byrne PS: Long BEL (Eds.): Doctors talking to patients. 1976, London: RCGP PublicationsGoogle Scholar
  21. Pendleton D: The consultation. An approach to teaching and learning. 1984, Oxford: Oxford Medical PublicationsGoogle Scholar
  22. Kurtz S, Silverman J, Benson J, Draper J: Marrying content and process in clinical method teaching: enhancing the Calgary-Cambridge guides. Acad Med. 2003, 78: 802-809. 10.1097/00001888-200308000-00011.View ArticleGoogle Scholar
  23. Kaplan RM, Frosch DL: Decision making in medicine and health care. Annu Rev Clin Psychol. 2005, 1: 525-556. 10.1146/annurev.clinpsy.1.102803.144118.View ArticleGoogle Scholar
  24. Allen J, Price E, Svab I, Windak A: EURACT: European academy of teachers in general practice and family medicine. Eur J Gen Pract. 2012, 18: 124-125. 10.3109/13814788.2012.681366.View ArticleGoogle Scholar
  25. Kogan JR, Holmboe ES, Hauer KE: Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009, 302: 1316-1326. 10.1001/jama.2009.1365.View ArticleGoogle Scholar
  26. Michels NR, Driessen EW, Muijtjens AM, Van Gaal LF, Bossaert LL, De Winter BY: Portfolio assessment during medical internships: how to obtain a reliable and feasible assessment procedure?. Educ Health (Abingdon). 2009, 22: 313.Google Scholar
  27. Rees CE, Sheard CE: The reliability of assessment criteria for undergraduate medical students’ communication skills portfolios: the Nottingham experience. Med Educ. 2004, 38: 138-144. 10.1111/j.1365-2923.2004.01744.x.View ArticleGoogle Scholar
  28. Pre-publication history

    1. The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/13/160/prepub

Copyright

© Klemenc-Ketis and Kersnik; licensee BioMed Central Ltd. 2013

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Advertisement