- Research article
- Open Access
- Open Peer Review
Influence of feedback characteristics on perceived learning value of feedback in clerkships: does culture matter?
BMC Medical Educationvolume 17, Article number: 69 (2017)
Various feedback characteristics have been suggested to positively influence student learning. It is not clear how these feedback characteristics contribute to students’ perceived learning value of feedback in cultures classified low on the cultural dimension of individualism and high on power distance. This study was conducted to validate the influence of five feedback characteristics on students’ perceived learning value of feedback in an Indonesian clerkship context.
We asked clerks in Neurology (n = 169) and Internal Medicine (n = 132) to assess on a 5-point Likert scale the learning value of the feedback they received. We asked them to record whether the feedback provider (1) informed the student what went well, (2) mentioned which aspects of performance needed improvement, (3) compared the student’s performance to a standard, (4) further explained or demonstrated the correct performance, and (5) prepared an action plan with the student to improve performance. Data were analyzed using multilevel regression.
A total of 250 students participated in this study, 131 from Internal Medicine (response rate 99%) and 119 from Neurology (response rate 70%). Of these participants, 225 respondents (44% males, 56% females) completed the form and reported 889 feedback moments. Students perceived feedback as more valuable when the feedback provider mentioned their weaknesses (β = 0.153, p < 0.01), compared their performance to a standard (β = 0.159, p < 0.01), explained or demonstrated the correct performance (β = 0.324, p < 0.001) and prepared an action plan with the student (β =0.496, p < 0.001). Appraisal of good performance did not influence the perceived learning value of feedback. No gender differences were found for perceived learning value.
In Indonesia, we could validate four out of the five characteristics for effective feedback. We argue that our findings relate to culture, in particular to the levels of individualism and power distance. The recognized characteristics of what constitutes effective feedback should be validated across cultures.
To meet international quality standards for medical education, an increasing number of medical schools all over the world try to improve their curricula . One of these standards addresses feedback in the clinical setting, because this kind of feedback is one of the most important tools to encourage student learning [2–6]. Medical schools are advised to ensure that students receive timely, specific, constructive and fair feedback to enhance their learning . Several feedback characteristics have been described in literature as vital for the effectiveness of feedback [2–4, 6–15]. However, recent research indicates that the significance of feedback characteristics for the learning value of feedback may not hold for countries with different cultures . We wondered whether feedback characteristics that were perceived to provide great learning value in a certain culture are also perceived to have great learning value in other cultures, and thus, whether feedback procedures that were developed in a certain culture can be implemented straight in a different culture. Therefore, in this study, we investigated how feedback characteristics, suggested in literature as essential for the effectiveness of feedback, influence Indonesian students’ perceived learning value of feedback.
Cultural differences seem to influence feedback processes and, therefore, may have implications for the applicability and effectiveness of feedback characteristics. In literature, it is suggested that the instructiveness of feedback is influenced by the status of the supervisor [3, 17], observation of behaviour [2, 18]. In the replication of a Dutch feedback study  in Indonesia , we considered the influence of the feedback provider and the role of direct observation on learning value, two recognized characteristics of effective feedback. Students were asked to register every feedback moment and asses the learning value of the feedback during a 2 - weeks period, and to report whether feedback was provided by a specialist who was perceived as an expert and as a credible person, and whether the feedback was based on direct observation. Overall, no differences in perceived learning value of feedback were found between the two countries. However, the two feedback characteristics under study were found to influence the perceived learning value differently, which could be explained from cultural differences between the two countries on the dimensions individualism and power distance [16, 19]. In Indonesia, a country classified as high on power distance , feedback provided by a specialist was perceived to have more learning value than feedback provided by a resident. In the Netherlands, classified as low on power distance , feedback from specialists and residents was perceived as equally valuable. In Indonesia, it did not make a difference whether the feedback was based on observation of the candidate or not. In the Netherlands, feedback based on observation was valued higher than feedback that had not been based on observation. It appears that the feedback requirements that should be met to optimize the effectiveness of the feedback differ between cultures.
The Indonesian replication study only took two feedback characteristics into account that were suggested as important for the effectiveness of feedback, namely the feedback provider and direct observation. However, five further feedback aspects have been described as essential for the effectiveness of feedback [2–4, 6–15]. First, students should be told what they have done well, to know their strengths [4, 7, 9, 12, 13]. Appraisal of good performance enhances students’ self-confidence and reinforces good practice. Second, students should be made aware of aspects of their performance that need to be improved [7, 9, 12, 13]. Information about performance deficiencies helps students to set learning goals. Third, students’ performance should be compared against a standard, such as a professional judgment, a local standard or existing guidelines [2, 4, 6, 8, 9]. This helps students to become aware of their progress towards mastery. Fourth, an explanation of the correct performance – what, how and why a task should be performed– should be provided to students to equip them with sufficient information to correct their errors [10, 15]. It is even more effective for student learning if the feedback provider models the correct performance . Finally, the feedback provider should invite students to make a plan of action to improve their performance and discuss it with them [4, 6, 9, 12, 13]. Preparing plans for self-improvement helps students apply the feedback they received in practice and narrow the gap between actual and desired performance [4, 12].
Considering the differences between the two countries with regard to the importance of feedback provider and direct observation, the question arises what the influence is of the other five recognized feedback characteristics on Indonesian students’ perceived learning value of feedback, all the more since the importance of these aspects for the effectiveness of feedback has mainly been investigated and explored in countries classified as high on individualism (individualistic) and low on power distance. The aim of this study was to validate the importance of the other five characteristics for the effectiveness of feedback in a country low on individualism (collectivistic) and high on power distance. We investigated the influence of these characteristics on the perceived learning value of feedback in the context of the Mini Clinical Evaluation Exercise (mini-CEX). We formulated the following research questions:
How often do Indonesian students perceive the five internationally acknowledged characteristics of effective feedback during a mini-CEX?
How do these feedback characteristics influence Indonesian students’ perceived learning value of feedback?
This study was conducted at the Faculty of Medicine, Universitas Gadjah Mada, Indonesia. The clinical phase consists of 2 years of clerkships in a department-based system. There are four eleven-week clerkship rotations in major clinical disciplines: Internal Medicine, Surgery, Obstetrics-Gynaecology and Paediatrics. There are seven four-week clerkship rotations in Neurology, Psychiatry, Dermatology, Ophthalmology, Otorhinolaryngology, Anaesthesiology, Radiology and Medical Forensics. Students do their rotations in two main teaching hospitals and several affiliated hospitals. Because we included only the departments where the mini-CEX was implemented, this study focused on the Internal Medicine and Neurology clerkships.
We decided to analyse the effectiveness of feedback characteristics in the context of the mini-CEX, which allows for and facilitates the application of all mentioned characteristics of effective feedback [21–26]. The mini-CEX focuses on observable competencies, is based on direct observation and provides a structure within which strengths, weaknesses and action plans can be discussed. The structure of the mini-CEX can facilitate the feedback provider in the role of expert and encourage comparison between students’ performance levels and performance standards. Consequently, the mini-CEX encourages the feedback provider to explain – on the basis of standards – what constitutes correct performance. Because of its potential, the mini-CEX has been used as a method to improve feedback during clerkships in many countries [27–29].
Students are obliged to have themselves assessed with the mini-CEX at least four times during the 11-week Internal Medicine clerkship, and at least twice during the four-week Neurology clerkship. The assessment form contains eight clinical competencies: history taking, physical examination, diagnosis, patient management, communication and counselling, professionalism, organization/efficiency and overall clinical care. Judgments are given on a 4-point scale (1 – under expectation, 2 – as expected, 3 – above expectation, and 4 – outstanding). Immediately after being observed, students are expected to ask a specialist for feedback and, subsequently, discuss their action plans to improve their performance. The two highest scores on the mini-CEX in each rotation are part of the summative, final grade at the end of the rotation. All assessment forms are recorded in the students’ logbooks, which they have to bring along during clerkships in all departments. Guidelines for all procedures, for both students and examiners, are also provided in the logbooks.
At all times, the mini-CEX examiners, to whom we will refer to as the feedback providers, are required to be specialists and clinical teachers at the main teaching hospital or one of the affiliated hospitals. All feedback providers had been introduced to the basic concepts of the mini-CEX (criteria and assessment procedure) and trained in providing constructive feedback.
Participants and procedure
We asked students in Neurology (n = 169) and Internal Medicine (n = 132) to participate in the study. They were asked to assess the learning value of the mini-CEX feedback using a 5-point Likert scale ranging from 0 – not valuable to 4 – very valuable. Furthermore, we asked them to record for each mini-CEX whether the feedback provider:
informed the student what went well (‘strength’ variable);
mentioned which aspects of performance needed improvement (‘weakness’ variable);
compared the student’s performance to a standard, such as a protocol, guideline, standard of medical services, standard operating procedure or clinical skills’ book (‘comparison to standard’ variable);
explained the correct performance (‘correct performance’ variable);
prepared an action plan with the student to improve performance (‘action plan’ variable).
We have piloted the instrument and we have used these questions for our previous studies [14, 16]. Ethical approval for the study was obtained from the Medical and Health Research Ethics Committee (MHREC) at Universitas Gadjah Mada.
Because each participant reported several and a varying number of feedback moments, the feedback moments needed to be dealt with as measurements that were nested within students. Because of this hierarchical data structure, we conducted a multilevel analysis. The analysis process comprised three stages. First, we established the necessity for multilevel analysis by calculating the Infraclass Correlation Coefficient (ICC). An ICC near zero would indicate no variance at the student level, while a higher number would indicate that variance in instructiveness stemmed from both the feedback moment level and the student level. Second, the empty model was estimated, which describes the variation in learning value associated with feedback moments and students separately. Third, the main effects model was used to calculate the impact of the feedback characteristics ‘strength’, ‘weakness’, ‘comparison to standard’, ‘correct performance’ and ‘action plan’ on the perceived learning value. The impact of the feedback characteristics was adjusted for gender and department, which were also added to the model. Subsequently, we examined the difference in deviance between models – the deviance assesses the extent to which a model deviates from the data – to compare the fit of the models with the data. Differences between models were determined using chi-square tests with the degrees of freedom being equal to the number of parameters added. We analyzed the data using MLwiN (version 2.01), a program specifically designed for multilevel analysis.
A total of 250 students participated in this study, 131 from Internal Medicine (response rate = 99%) and 119 from Neurology (response rate = 70%). Of these participants, 25 were excluded for having completed their forms inaccurately or incompletely. The remaining 225 respondents (44% males, 56% females) reported 889 feedback moments.
During the mini-CEX, the feedback providers mentioned students’ strengths in 59% and weaknesses in 72% of the feedback moments. Furthermore, they compared students’ performance to a standard in 49%, explained the correct performance in 68%, and prepared an action plan with the student to improve performance in 55% of the feedback moments.
The ICC was 0.38, indicating that 38% of variance in instructiveness was explained at the student level, which indicates that a multilevel approach was necessary. The overall mean of the perceived learning value of feedback was 3.5 on a 4-point scale (SE = 0.03). Students perceived feedback as more valuable when the feedback provider mentioned their weaknesses (β = 0.183, p < 0.01) and compared their performance to a standard (β = 0.174, p < 0.01). Explaining or demonstrating the correct performance (β = 0.322, p < 0.001) and preparing an action plan with the student to improve performance (β = 0.471, p < 0.001) both influenced the perceived learning value of feedback positively. Mentioning strengths did not influence the perceived learning value of feedback (β = −0.01, p > 0.05). We found no significant gender differences in students’ perceived learning value of feedback (β = 0.052, p > 0.05). Feedback received in the Neurology department was perceived to have a higher learning value than feedback received in the Internal medicine department (β = 0.151, p < 0.01). The main effects model fitted the data significantly better than the empty model (χ2(7) = 220; p < 0.001) (Table 1).
The aim of our study was to validate the importance of five feedback characteristics for the perceived the learning value of feedback in Indonesia, a country that is classified as low on individualism and high on power distance. All five feedback characteristics – ‘mentioning students’ strengths’, ‘mentioning students’ weaknesses’, ‘comparing students’ performance to a standard’, ‘explaining or demonstrating the correct performance’, and ‘preparing an action plan with the student’ – were perceived by the students, but not in all feedback moments. We found a positive influence of the latter four feedback characteristics, but mentioning strengths did not influence Indonesian students’ perceived learning value of feedback.
Teachers’ goals and expectations are important factors affecting student motivation for learning in cultures low on individualism and high on power distance [20, 30, 31]. Because of the power distance, students expect their teachers to determine the learning paths. This might explain the positive influence of four of the five feedback characteristics on Indonesian students’ perceived learning value of feedback. Students probably interpreted feedback based on these characteristics as outlines that serve to improve their learning and, thus, help them live up to their teachers’ expectations. The fifth characteristic of effective feedback – mentioning strengths – did not positively influence the perceived learning value of feedback, whereas, in literature, mentioning strengths has been perceived as conditional for feedback to be effective [4, 7, 9, 12, 13]. An explanation for this outcome may be that mentioning strengths does not relate to teachers’ goals and expectations. Students probably interpreted the reported strengths as no more than an act of kindness from the part of the feedback provider. Furthermore, in Indonesian society, there is a tendency towards modesty, which implies that not standing out from but fitting in with others is valued . As such, compliments regarding personal performance may have little learning value for students.
We found that some feedback characteristics were more frequently used than others and we do believe that a cultural component is involved here as well. The characteristics ‘mentioning students’ weaknesses’ and ‘explaining or demonstrating the correct performance’ were used most frequently. Teachers can probably use these characteristics best to explain their expectations of the students. In countries classified as low on individualism and high on power distance, the goal of giving feedback is to correct errors and behaviour [32, 33]. Communicating clear expectations seems to be most effective to achieve this goal. The feedback characteristic ‘comparing performance to a standard’ was used least frequently in feedback. A possible explanation is that standards are perceived as just additional learning sources. Education in countries that are high on power distance, such as Indonesia, is teacher-centred [20, 30]. Teachers’ comments about students’ weaknesses and correct performance place them more in the centre of education than their comments about students’ performance in relation to the standards do. As such, teachers may feel that comparing performance to standards may be a less effective tool to communicate goals and expectations to the students. However, other factors may also influence the use of certain feedback characteristics. To get more insight into the feedback dynamics, we recommend qualitative research to explore why feedback providers attach more value to some than to other feedback characteristics and how this may be influenced by culture.
Seven feedback characteristics are described in literature as having a positive influence on student learning. In the current study, we examined five of these characteristics and in a former study, we investigated the other two . Based on both studies, we conclude that – in the Indonesian context – two out of the seven characteristics did not positively influence Indonesian students’ perceived learning value of feedback: ‘direct observation’ and ‘mentioning strengths’. Besides, whether a feedback provider can serve as a good role model seems to be dependent upon the status of the feedback provider. Whereas in the Dutch context, it did not matter whether students received feedback from residents or specialists, in the Indonesian context feedback from specialists – higher in status than residents – was rated more positively than feedback from residents was. The differences between the findings described in literature and our outcomes can be explained by cultural differences, especially by differences with regard to the cultural dimensions individualism and power distance. More research on different cultures is needed to strengthen our outcomes.
A strength of our study is that we used the context of the mini-CEX, with trained specialists as feedback providers , to investigate the relations between feedback characteristics and students’ perceived learning value of feedback. The mini-CEX provides a structure that allows for and facilitates application of all mentioned characteristics of effective feedback [21–26]. Another strength is that the current study was performed in two departments – Internal Medicine and Neurology – in which the mini-CEX had been systematically implemented in the clerkship programme 2 years before . We assumed that the specialists from these departments had become familiar with providing constructive feedback as they had been trained to do so and had already gained experience. However, further studies are needed to strengthen the generalizability of our results.
A limitation of our study is that only one Indonesian medical school was included. We are aware that our findings should be interpreted carefully because the cultural dimensions discussed could be a simplification of reality , since Indonesia is a very large country in which cultural conditions can differ from region to region . However, in general, subcultures reflect the overarching national culture in which they are embedded, with cultural differences between countries being larger than those between subcultures within countries . Therefore, our results may be generalizable to other medical schools in countries with a culture characterized by low individualism and high power distance. To strengthen the current outcomes, replication studies are needed.
Another limitation lies in the use of students’ reports and perceptions. We did not measure their actual learning. Even though perceived learning value is not the same as actual learning output, we feel perceptions of learning value provide a good indication of the quality of feedback, because positive perceptions by students are conducive to their learning processes . As a corollary, positive perceptions of the learning value of feedback can be expected to improve learning and future performance. However, more research is needed to unravel the impact of feedback characteristics on students’ actual learning.
In conclusion, some of the recognized feedback characteristics seem to be valid in different countries. However, the learning value of the characteristics ‘feedback provider’, ‘observation’ and ‘mentioning strengths’ seems to depend on the prevailing culture of countries, in particular on the levels of individualism and power distance. Therefore, we recommend that culture should be considered when implementing international standards for feedback in order to optimize student learning and development. We also suggest to further validate our findings in different cultural settings in order to improve our understanding of what constitutes effective feedback in different cultures and why.
Intraclass Correlation Coefficient
Mini Clinical Evaluation Exercise
World Federation of Medical Education (WFME). Basic Medical Education. WFME Global Standards for Quality Improvement. New Edition [Internet]. 2012. Available from: http://wfme.org/standards/bme/78-new-version-2012-quality-improvement-in-basic-medical-education-english/file. Accessed 15 May 2013.
Ende J. Feedback in clinical medical education. JAMA. 1983;250:777–81.
Daelmans HEM, Hoogenboom RJI, Donker AJM, Scherpbier AJJA, Stehouwer CDA, Van der Vleuten CPM. Effectiveness of clinical rotations as a learning environment for achieving competences. Med Teach. 2004;26:305–12.
Cantillon P, Sargeant J. Giving feedback in clinical settings. BMJ. 2008;337:a1961.
Ramani S, Leinster S. AMEE Guide no. 34: teaching in the clinical environment. Med Teach. 2008;30:347–64.
Van de Ridder JMM, Stokking KM, McGaghie WC, ten Cate OTJ. What is feedback in clinical education? Med Educ. 2008;42:189–97.
Gordon J. One to one teaching and feedback. BMJ. 2003;326:543–5.
Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide no. 7. Med Teach. 2006;28:117–28.
Nicholson S, Cook V, Naish J, Boursicot K. Feedback: its importance in developing medical student’s clinical practice. Clin Teach. 2008;5:163–6.
Archer J. State of the science in health professional education: effective feedback. Med Educ. 2010;44:101–8.
Norcini JJ. The power of feedback. Med Educ. 2010;44:16–7.
Ramani S, Krackov SK. Twelve tips for giving feedback effectively in the clinical environment. Med Teach. 2012;34:787–91.
Hewson MG, Little ML. Giving feedback in medical education: verification of recommended techniques. J Gen Intern Med. 1998;13:111–6.
Van Hell EA, Kuks JBM, Raat AN, van Lohuizen MT, Cohen-Schotanus J. Instructiveness of feedback during clerkships: influence of supervisor, observation and student initiative. Med Teach. 2009;31:45–50.
Shute VJ. Focus on formative feedback. Rev Educ Res. 2008;78:153–89.
Suhoyo Y, Van Hell EA, Prihatiningsih TS, Kuks JBM, Cohen-Schotanus J. Exploring cultural differences in feedback processes and perceived instructiveness during clerkships: replicating a Dutch study in Indonesia. Med Teach. 2014;36:223–9.
Daelmans HEM, Hoogenboom RJI, Scherpbier AJJA, Stehouwer CDA, van der Vleuten CPM. Effects of an in-training assessment programme on supervision of and feedback on competencies in an undergraduate internal medicine clerkship. Med Teach. 2005;27:158–63.
Smith CS, Irby DM. The roles of experience and reflection in ambulatory care education. Acad Med. 1997;72:32–5.
Hwang A, Francesco AM. The influence of individualism–collectivism and power distance on use of feedback channels and consequences for learning. AMLE. 2010;9:243–57.
Hofstede G. Culture’s consequences, comparing values, behaviors, institutions, and organizations across nations. Newbury Park: Sage Publications; 2001.
Kogan JR, Bellini LM, Shea JA. Feasibility, reliability and validity of the mini clinical evaluation Exercise (mini-CEX) in a medicine core clerkship. Acad Med. 2003;78:33–5.
Norcini JJ. The Mini Clinical Evaluation Exercise (mini-CEX). Clin Teach. 2005;2:25–30.
Hill F, Kendall K. Adopting and adapting the mini-CEX as an undergraduate assessment and learning tool. Clin Teach. 2007;4:244–8.
Norcini JJ, Burch V. Workplace-based assessment as an educational tool: AMEE Guide no. 31. Med Teach. 2007;29:855–71.
Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302:1316–26.
Boursicot K, Etheridge L, Setna Z, Sturrock A, Ker J, Smee S, Sambandam E. Performance in assessment: consensus statement and recommendations from the Ottawa conference. Med Teach. 2011;33:370–83.
Hauer KE. Enhancing feedback to students using the mini-CEX (clinical evaluation Exercise). Acad Med. 2000;75:524.
Dewi SP, Achmad TH. Optimising feedback using the mini-CEX during the final semester programme. Med Educ. 2010;44:509.
Suhoyo Y, Schönrock-Adema J, Rahayu GR, Kuks JBM, Cohen-Schotanus J. Meeting international standards: a cultural approach in implementing the mini-CEX effectively in Indonesian clerkships. Med Teach. 2014;36:894–902.
Hofstede G. Cultural difference in teaching and learning. Int J Intercult Rel. 1986;10:301–32.
Markus HR, Kitayama S. Culture and the self: implication for cognition, emotion, and motivation. Psychol Rev. 1991;98:224–53.
Pratt DD, Kelly M, Wong WSS. Chinese conception of “effective teaching” in Hongkong: towards culturally sensitive evaluation of teaching. Int J Lifelong Educ. 1999;18:241–58.
Wong AK. Culture in medical education: comparing a Thai and a Canadian residency programme. Med Educ. 2011;45:1209–19.
Bik OPG. The Behaviour of Assurance Professionals: A Cross-cultural Perspective. [PhD thesis University of Groningen]. Delft: Eburon Academic Publishers; 2010.
Hofstede G, de Hilal AVG, Malvezzi S, Tanure B, Vinken H. Comparing regional cultures within a country: lessons from Brazil. Journal of Cross-Cultural Psychology (JCCP). 2010;41:336–52.
Kirkpatrick D. Great ideas revisited: revisiting Kirkpatrick’s four-level model. Train Dev. 1996;50:54–9.
We would like to thank the students who participated in this study and Tineke Bouwkamp-Timmer for her constructive comments on several drafts of the manuscript and editorial help.
There is no funding to report.
Availability of data and materials
Data used in this study are available on request.
YS was responsible for the concept and design of the study, data collection and analysis, and for several drafts of the paper. EVH, WK and OE contributed to the concept and design of the study, data analysis and interpretation, and writing the paper. JSA and JBMK were responsible for overseeing the design of the study and contributed to data analysis and interpretation, and writing the paper. JCS was responsible for overseeing the concept and design of the study and contributed to data analysis and interpretation, and writing the paper. All authors read and approved the final manuscript for publication.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent to participate
Ethical approval for the study was obtained from the Medical and Health Research Ethics Committee (MHREC) at Universitas Gadjah Mada. We asked students in Neurology (n = 169) and Internal Medicine (n = 132) to participate in the study. They were asked to assess the learning value of the mini-CEX feedback using a 5-point Likert scale ranging from 0 – not valuable to 4 – very valuable. Furthermore, we asked them to record for each mini-CEX whether the feedback provider: (1) informed the student what went well; (2) mentioned which aspects of performance needed improvement; (3) compared the student’s performance to a standard, such as a protocol, guideline, standard of medical services, standard operating procedure or clinical skills’ book; (4) explained the correct performance; (5) prepared an action plan with the student to improve performance. Each questionnaire was accompanied by a letter explaining the purpose of the research.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.