Skip to main content
  • Research article
  • Open access
  • Published:

Multisource feedback analysis of pediatric outpatient teaching

Abstract

Background

This study aims to evaluate the outpatient communication skills of medical students via multisource feedback, which may be useful to map future directions in improving physician-patient communication.

Methods

Family respondents of patients, a nurse, a clinical teacher, and a research assistant evaluated video-recorded medical students’ interactions with outpatients by using multisource feedback questionnaires; students also assessed their own skills. The questionnaire was answered based on the video-recorded interactions between outpatients and the medical students.

Results

A total of 60 family respondents of the 60 patients completed the questionnaires, 58 (96.7%) of them agreed with the video recording. Two reasons for reluctance were “personal privacy” issues and “simply disagree” with the video recording. The average satisfaction score of the 58 students was 85.1 points, indicating students’ performance was in the category between satisfied and very satisfied. The family respondents were most satisfied with the “teacher”s attitude,“ followed by ”teaching quality”. In contrast, the family respondents were least satisfied with “being open to questions”. Among the 6 assessment domains of communication skills, the students scored highest on “explaining” and lowest on “giving recommendations”. In the detailed assessment by family respondents, the students scored lowest on “asking about life/school burden”. In the multisource analysis, the nurses’ mean score was much higher and the students’ mean self-assessment score was lower than the average scores on all domains.

Conclusion

The willingness and satisfaction of family respondents were high in this study. Students scored the lowest on giving recommendations to patients. Multisource feedback with video recording is useful in providing more accurate evaluation of students’ communication competence and in identifying the areas of communication that require enhancement.

Peer Review reports

Background

Communication is an important component of patient care. The physician-patient interview is the key component of all health care, particularly of primary medical care [1, 2]. Outpatient clinics offer trainees one of the most varied clinical experiences within the hospital setting, but they are often chaotic and over-stretched, with limited time for teaching [3]. When patients are informed and involved in decision making, they adhere to medical recommendations (e.g., vaccination and dietary modification) [1, 4]. Such joint decision making requires patients to be fully informed about alternatives and potential risks of treatment [1, 5]. Clinical practice within outpatient clinics can strengthen the collective knowledge of trainees [3]. This establishment of practice may also validate the role of trainees in the management of patients and facilitate social learning [3].

Assessment is an essential step in the curricular development process [6]. An evaluation method is important for the improvement of the quality of learning among medical students; however, such method is rare [6]. Concern about the inability of monitored examinations to assess the full spectrum of clinical competence, including humanistic quality, knowledge, and communication skills, stimulated the introduction of the “patient and peer assessment module” [7]. On the other hand, supervision features observation and sharing of clinical feedback, which can improve clinical performance [3]. Effective needs-assessment strategies include Multisource feedback (MSF) from educators and learners [8]. Thus, the patient and peer assessment, and self-evaluation modules are better methods to evaluate the communication and clinical performance of medical students [7].

We aimed to evaluate the outpatient communication skills of medical students by using MSF from family respondents of patients, nurses, a clinical teacher, and a research assistant. This may be an effective evaluation method in the future to improve physician-patient communication skills in the outpatient setting.

Methods

Settings: undergraduate students of Chang Gung Memorial Hospital (CGMH)

We employed a multi-respondent evaluation method using a structured paper questionnaire to investigate the communication skills of our students in the pediatric outpatient clinic. All seventh-year medical students trained at CGMH were enrolled in this study. There were 32 males and 28 females, with mean age of 25 years (24–27 y). MSF was obtained from a nurse, a clinical teacher, and a research assistant to assess the medical students’ communication competence; a self-assessment of skills was also administered by the students. The satisfaction score of the medical students was evaluated by 4 respondent groups which included the family, research assistant, nurse and students. The same teacher and the same research assistant were involved in the whole study. The patients, 36 males and 24 females, were one month to 16 years old (median: 3.3 y). The patients who were suitable for this teaching clinic were classified by nurses before the patients went into the outpatient clinic. The study was approved by the Ethics and Clinical Research Committee of CGMH (No. 98-2202B).

Instruments

Each outpatient interaction led by the medical students was video-recorded. The students, a nurse, a teacher, and a research assistant, watched the video together in a room at the same time and then filled out the corresponding paper questionnaire. The observers (students, nurse, teacher, research assistant) discussed the videotaped interactions after completing the assessments. The family respondents completed reasons for agreeing/disagreeing to the questionnaires before video recording and completed their assessments after the outpatient clinic at a room next to the clinic.

Assessment and evaluation

The completed questionnaires were validated by 3 professional teachers. The questionnaire included items such as family respondent’s reasons for agreeing/disagreeing with the video recording, multisource satisfaction with the outpatient interaction, and evaluation of skills in 6 domains (i.e., giving recommendations, listening, explaining, acknowledging, negotiation, and patient-centered communication). These skills were subdivided accordingly as shown in Table 1 and Figures 1, 2, 3[1, 9]. Every question can be answered with the following options and corresponding scores: very satisfied (100 points), satisfied (80 points), no opinion (60 points), dissatisfied (40 points), and very dissatisfied (20 points).

Table 1 Subdivided outpatient skills evaluation by the different groups
Figure 1
figure 1

Agree reasons. Patients agreed with the video recording to contribute to medical education, to get a more accurate diagnosis, and to cooperate on the study.

Figure 2
figure 2

Teaching satisfaction. Average satisfaction score was 85.1 points. Students scored highest on “teacher’s attitude,” followed by ”teaching quality.” Family respondents were most satisfied with “teacher’s attitude” followed by “teaching quality”; students were most satisfied with “teacher’s attitude” and “being open to questions.” The family respondents of patients were least satisfied with “being open to questions,” whereas students were least satisfied with “student’s attitude.” The results represented were mean ± SD. * P < 0.05 compared to the average scores.

Figure 3
figure 3

Outpatient skills in six assessment domains. Students scored highest on “explaining” and lowest on “giving recommendations” as in the average scores. The results represented were mean ± SD. * P < 0.05 compared to the average scores.

Our patients included children with respiratory tract infection, liver cirrhosis, and abdominal pain, and those requiring post-operative care. These patients were classified into difficult or common cases evaluated by students after the clinics. The liver cirrhosis patients included hepatitis or biliary atresia with routine follow-up and without complications.

The questionnaires in this study were tested with Cronbach’s α for reliability. Statistical methods employed were descriptive statistics and student’s t-test (one-sided) in comparison between difficult and common cases. Our hypothesis is that the student will have lower scores with difficult cases. Comparisons in teaching satisfaction and six domains between different respondent groups were analyzed by the ANOVA with Bonferroni’s correction, when multiple comparisons were evaluated. The correlation in the overall satisfaction with number of questions asked was also analyzed by Pearson correlation test. A p-value < 0.05 was considered statistically significant. The statistical analyses were performed using the Statistical Package for Social Science (SPSS, version 12) software package.

Results

Reliability

Cronbach’s α coefficient was used to assess reliability [9, 10]. The Cronbach’s α values for the questionnaires of the 4 respondent groups were as follows: 0.696 for the students, 0.974 for the nurses, 0.914 for the teacher and 0.918 for the research assistant. The Cronbachs’ α values for the students’ scores would be 0.857 had the item “asking about chief complaints” been deleted. The item “asking about chief complaints” is of low reliability for the students’ assessing themselves in this study. The overall Cronbachs’ α value in this study was 0.867.

Reasons for agreeing/disagreeing with video recording

A total of 60 family respondents of the 60 patients completed the questionnaires, 58 (96.7%) of them agreed with the video recording and is the source of data included for statistical analysis. Two reasons for reluctance were “personal privacy” issues and they “simply disagree” with video recording. Among the family respondents who agreed with the video recording, the following reasons were obtained: to contribute to medical education (41.8%), to get a more accurate diagnosis (13.9%), and to cooperate on the study (12.7%) (Figure 1).

Satisfaction

The average satisfaction score of the 58 medical students was 85.1 points (references 60 points), indicating students’ performance was in the category between satisfied and very satisfied, which was derived from the following group scores: 88.9 points (family), 85.6 (research assistant), 84.0 (nurses), 82.2 (students) (Figure 2). All participants were most satisfied with the “teacher’s attitude” (92.9 points) and “teaching quality” (88.4 points). The family respondents were likewise most satisfied with the" “teacher’s attitude” (90.7 points), followed by the “clinic quality” (90.7 points); students were most satisfied with the “teacher’s attitude” (97.5 points), followed by “being open to questions” (92.5 points). In contrast, the family respondents were least satisfied with “being open to questions” (86.3 points), while the students were least satisfied with the “student’s attitude” (74.7 points) (Figure 2).

We discarded the students’ self-ratings as input in the t-test for differences between difficult and common cases. There was no significant difference in the overall satisfaction by the other 3 groups’ (family respondents, nurses, research assistant) evaluation between difficult cases and common cases (82.4 ± 13.8 vs. 85.6 ± 13.3, t = 1.481, degrees of freedom (df) = 172, P = 0.070). Lower scores were obtained in the difficult cases than in the common cases with regard to “student’s attitude” (80.0 ± 14.8 vs. 87.2 ± 13.5, t = 3.289, df = 172, P = 0.001), and “being open to questions” (79.3 ± 15.2 vs. 86.8 ± 12.8, t = 3.451, df = 172, P = 0.001). The number of questions asked was about 10–15 in each case. There was no significant correlation in the overall satisfaction with the number of questions asked (r = 0.021, P = 0.898).

Evaluation of outpatient skill in 6 domains

Outpatient skills were assessed in 6 domains. Overall, students obtained the highest score on “explaining” (82.6 points) and the lowest score on “giving recommendations” (73.6 points).

The teacher assessed the students to have the highest score on the “explaining” outpatient skill (83.8 points), while the lowest was on “giving recommendations” (71.8 points). In the self-assessment of students, they had the highest score on the “explaining” skill (80.2 points) and the lowest on “providing recommendations” (72.5 points) (Figure 3). The teacher’s and the research assistant’s scores were comparable to the 4 groups’ average score. The nurses’ score was the highest and the students’ self-assessed score was lower than the average score of the 4 groups (Table 2).

Table 2 The total scores of outpatient skill in 6 domains between the different respondents

Subdivided outpatient skills evaluation by the different groups

In the subdivided outpatient teaching skills, the students performed best in “asking patients about their chief complaints” (90.5 points), while worst in “asking about school/life burden” (65.6 points). Teachers believed that the students were best in “consensus decision maker” (94.5 points), followed by “giving self-introduction” (90.0 points). In contrast, they believed that the students were worst in “asking about school/life burden” (52.3 points), as did the students (66.4 points), nurses (62.3 points), and the research assistant (74.0 points). The second worst skill of the students was “Get a consensus with patient in different views” (average 69.2 points) (Table 1).

Discussion

The agreement and satisfaction rate was high in this pediatric outpatient teaching evaluation. Different respondents will give much different scores in the evaluation. We noticed that the four respondents assessments could measure out more accurately the strengths and weaknesses of the students outpatient skills than the single one respondent assessment.

Why were their differences in satisfaction and skill scores between assessor groups? Were senior doctors more critical and strict in their assessments [1113]?. In our study, we found that students gave more strict scores to themselves than the rest of the groups. Did the nurses give more favorable assessments because they understand the trainees better and thus make allowances for weakness [11, 13]?. This favorable bias could be found in the nurses’ scores in our study, but not in the research assistant and the teacher. Alternatively, were their assessments more reliable because they know the students better [11]?. We found that the teacher’s and the research assistant’s scores were comparable to that of the 4 groups’ average score in Table 2. The nurses’ mean score was much higher and the students’ mean self-evaluation score was much lower than the average score. We believe that MSF containing four respondents assessments used in this study can give a more accurate assessment to distinguish students’ strong and weak points [14, 15].

The quality of the results may also be influenced by personal relationships, stakes and equivalence [7]. Students’ own scores tended to be the lowest in our study. This is common in our teaching environment, and is probably due to the named questionnaires. Physician-patient communication has frequently been judged to be inadequate and imperfect [16]. It is thus important that communication skills of physicians be assessed periodically with a confidential peer evaluation survey [7], as such surveys may provide unbiased and factual information from respondents. However, in our study, preserving the anonymity and confidentiality of students and patients was difficult to achieve. First, selecting patients who were willing to participate in the study was not easy; second, the students who participated all completed the peer assessment in one outpatient room. However, this may be improved in the future by using a one-way mirror room or by ensuring that students complete the questionnaires in separate rooms.

Published findings showed that the number of questions asked about a patient’s illness was inversely related to patient satisfaction [17]. However, this inverse relation was not obtained in our study. We found that students assigned to difficult cases scored lower on “student’s attitude” and “being open to questions.” Counseling about unhealthy or risky behaviors is an important communication skill that should be a vital part of health care visits [2]. Physicians’ attitudes towards the physician-patient relationship may contribute to the diagnostic value of the patient history [18]. To this end, we should further educate the students about the appropriate attitude and approach in answering questions of patients with diseases that are difficult to treat. This may be an important future direction of research on physician-patient communication.

A patient-centered medical interview is essential to create good interpersonal relationships and information exchange; it may also contribute to the diagnostic value of the patient history and facilitate informed decision making [18, 19]. Medical educators should focus on teaching and reinforcing behaviors that are known to enhance favorable patient outcomes and satisfaction [1]. Patient health outcomes can be improved with good physician-patient communication [19]. As Aspergren noted, communication skills can be taught in courses but are easily forgotten if not maintained by practice [20]. We believe that this pediatric-patient communication project will be helpful for the development and promotion of clinical skills of medical students in an outpatient practice. One limitation of this study was that this was a single site study using a specifically designed instrument and therefore might not be generalised. Therefore, these findings need to be tested in a large-scale study.

Conclusions

MSF with video-recorded is important in providing a more accurate evaluation of students’ communication competence and in identifying the areas of communication that require enhancement.

Authors’ information

M-MT, L-TH, Y-HH, K-ST are staff in Chang Gung Memorial Hospital. M-MT is a physician educator in Chang Gung Memorial Hospital.

Abbreviations

CGMH:

Chang Gung Memorial Hospital

MSF:

Multisource feedback.

References

  1. Beck RS, Daughtridge R, Sloane PD: Physician-patient communication in the primary care office: a systematic review. J Am Board Fam Pract. 2002, 15 (1): 25-38.

    Google Scholar 

  2. Teutsch C: Patient-doctor communication. Med Clin North Am. 2003, 87 (5): 1115-1145. 10.1016/S0025-7125(03)00066-X.

    Article  Google Scholar 

  3. Williamson J: Teaching and learning in out-patient clinics. Clin Teach. 2012, 9 (5): 304-307. 10.1111/j.1743-498X.2012.00545.x.

    Article  Google Scholar 

  4. Chervenak FA, McCullough LB: Clinical guides to preventing ethical conflicts between pregnant women and their physicians. Am J Obstet Gynecol. 1990, 162 (2): 303-307. 10.1016/0002-9378(90)90374-G.

    Article  Google Scholar 

  5. Mazur DJ, Hickam DH: Patients’ preferences for risk disclosure and role in decision making for invasive medical procedures. J Gen Intern Med. 1997, 12 (2): 114-117. 10.1007/s11606-006-5006-8.

    Article  Google Scholar 

  6. Burgess A, Black K, Chapman R, Clark T, Roberts C, Mellis C: Teaching skills for students: our future educators. Clin Teach. 2012, 9 (5): 312-316. 10.1111/j.1743-498X.2012.00554.x.

    Article  Google Scholar 

  7. Lipner RS, Blank LL, Leas BF, Fortna GS: The value of patient and peer ratings in recertification. Acad Med. 2002, 77 (10 Suppl): S64-66.

    Article  Google Scholar 

  8. Keister D, Grames H: Multi-method needs assessment optimises learning. Clin Teach. 2012, 9 (5): 295-298. 10.1111/j.1743-498X.2012.00549.x.

    Article  Google Scholar 

  9. Delavar M, Salmalian H, Faramarzi M, Pasha H, Bakhtiari A, Nikpour M, Ledari F: Using the objective structured clinical examinations in undergraduate midwifery students. J Med Life. 2013, 6 (1): 76-79.

    Google Scholar 

  10. Qu B, Zhao YH, Sun BZ: Assessment of resident physicians in professionalism, interpersonal and communication skills: a multisource feedback. Int J Med Sci. 2012, 9 (3): 228-236. 10.7150/ijms.3353.

    Article  Google Scholar 

  11. Ellul P, Moore A, Camilleri J, Grech V: An analysis of multisource feedback within the foundation programme. Clin Teach. 2012, 9 (5): 290-294. 10.1111/j.1743-498X.2012.00537.x.

    Article  Google Scholar 

  12. Davies JG, Ciantar J, Jubraj B, Bates IP: Use of a multisource feedback tool to develop pharmacists in a postgraduate training program. Am J Pharm Educ. 2013, 77 (3): 52-10.5688/ajpe77352.

    Article  Google Scholar 

  13. Roberts MJ, Campbell JL, Richards SH, Wright C: Self-other agreement in multisource feedback: the influence of doctor and rater group characteristics. J Contin Educ Health Prof. 2013, 33 (1): 14-23. 10.1002/chp.21162.

    Article  Google Scholar 

  14. Ten Cate O, Sargeant J: Multisource feedback for residents: how high must the stakes be?. J Grad Med Educ. 2011, 3 (4): 453-455. 10.4300/JGME-D-11-00220.1.

    Article  Google Scholar 

  15. Garra G, Wackett A, Thode H: Feasibility and reliability of a multisource feedback tool for emergency medicine residents. J Grad Med Educ. 2011, 3 (3): 356-360. 10.4300/JGME-D-10-00173.1.

    Article  Google Scholar 

  16. Higginbotham EJ, Bartley GB: Interview with eve j. Higginbotham, MD. JAMA Ophthalmol. 2013, 131 (1): 96-97. 10.1001/jamaophthalmol.2013.1640.

    Article  Google Scholar 

  17. Roter DL, Stewart M, Putnam SM, Lipkin M, Stiles W, Inui TS: Communication patterns of primary care physicians. JAMA. 1997, 277 (4): 350-356. 10.1001/jama.1997.03540280088045.

    Article  Google Scholar 

  18. Rich EC, Crowson TW, Harris IB: The diagnostic value of the medical history. Perceptions of internal medicine physicians. Arch Intern Med. 1987, 147 (11): 1957-1960.

    Article  Google Scholar 

  19. Stewart MA: Effective physician-patient communication and health outcomes: a review. CMAJ. 1995, 152 (9): 1423-1433.

    Google Scholar 

  20. Aspegren K: BEME Guide No. 2: Teaching and learning communication skills in medicine-a review with quality grading of articles. Med Teach. 1999, 21 (6): 563-570. 10.1080/01421599978979.

    Article  Google Scholar 

Pre-publication history

Download references

Acknowledgements

This work was supported by grants (CDRPG890023, CDRPG8C0021) from the Chang Gung Memorial Hospital, Taiwan.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Mao-Meng Tiao.

Additional information

Competing interests

None of the authors have financial disclosures or conflicts of interest to declare.

Authors’ contributions

M-MT and L-TH provided the idea and framework for the study and edited the manuscript. M-MT wrote the primary manuscript draft. Y-HH, K-ST, CJ Chen performed the data analysis, generated the figures. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Authors’ original file for figure 3

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Tiao, MM., Huang, LT., Huang, YH. et al. Multisource feedback analysis of pediatric outpatient teaching. BMC Med Educ 13, 145 (2013). https://doi.org/10.1186/1472-6920-13-145

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1472-6920-13-145

Keywords