- Research article
- Open Access
- Open Peer Review
Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: a systematic review
© Ferguson et al.; licensee BioMed Central Ltd. 2014
- Received: 11 December 2013
- Accepted: 3 April 2014
- Published: 11 April 2014
Multisource feedback (MSF) is currently being introduced in the UK as part of a cycle of performance review for doctors. However, although it is suggested that the provision of feedback can lead to a positive change in performance and learning for medical professionals, the evidence supporting these assumptions is unclear. The aim of this review, therefore, was to identify the key factors that influence the effectiveness of multisource feedback in improving the professional practice of medical doctors.
Relevant electronic bibliographic databases were searched for studies that aimed to assess the impact of MSF on professional practice. Two reviewers independently selected and quality assessed the studies and abstracted data regarding study design, setting, MSF instrument, behaviour changes identified and influencing factors using a standard data extraction form.
A total of 16 studies met the inclusion criteria and quality assessment criteria. While seven studies reported only a general change in professional practice, a further seven studies identified specific changes in behaviour. The main professional behaviours that were found to be influenced by the feedback were communication, both with colleagues and patients and an improvement in clinical competence/skills. The main factors found to influence the acceptance and use of MSF were the format of the feedback, specifically in terms of whether it was facilitated, or if narrative comments were included in the review, and if the feedback was from sources that the physician believed to be knowledgeable and credible.
While there is limited evidence suggesting that MSF can influence professional performance, the quality of this evidence is variable. Further research is necessary to establish how this type of feedback actually influences behaviours and what factors have greatest influence.
- Systematic review
- Multisource Feedback
Medical practitioners are now professionally accountable via periodic revalidation for the standard of patient services they provide . The assessment of professional performance is one way to demonstrate fitness to practise medicine and provide evidence of accountability. Medical regulators and educational bodies are endeavouring to identify reliable and robust methods to routinely assess the competence of trainee and qualified doctors to contribute supporting evidence of ongoing performance [2, 3]. However, while there is agreement on the need for regular performance assessment of doctors, how best to do this is unclear. Assessment methods involving simulated patients, video observation, audits of clinical records, critical incident analysis and multisource feedback are all suggested, but these approaches involve subjective judgements and lack adequate psychometric evaluation [2, 4–7]. An earlier systematic review of a diverse range of assessment methods highlighted the lack of rigour, effectiveness and overall utility of most approaches . However, in terms of costs and time considerations, feedback from colleagues (also known as multisource feedback) was deemed to be the most appropriate and practical method. Moreover, whilst MSF may be seen as a collection of subjective assessments, studies of assessment methods have concluded that subjective methods can be as reliable as objective methods provided the sample size of assessors is adequate [9, 10].
As such, multisource feedback (MSF) is being introduced by the UK medical regulator, the General Medical Council (GMC),  as part of a cycle of performance review for all medical doctors. The GMC asserts that this feedback needs to be from more than a single source (e.g. self, medical colleagues and non-medical co-workers) and cover the whole of the individual’s professional practice .
MSF is a questionnaire-based process, which allows for the feedback to be collected in a systematic way . The intent of the feedback is to guide behaviour change and improve performance . The participant usually receives a copy of their own data (presented as aggregate scores for each element of perceived professional performance being assessed by colleagues) along with comparison data for their group. Graphical as well as numerical data may be provided to the participant. Obtaining more than one person’s view of colleague performance means that a “more complete and accurate assessment of the employees’ competencies”  is generated. Many MSF tools also require participants to self-assess, by completing the questionnaire themselves as a means of comparing self and others’ perceptions of professional performance .
Although it is suggested that the provision of feedback can lead to learning and a positive change in performance for medical professionals, the evidence supporting these assumptions is unclear . Even with studies which have reported performance improvement over time, it is uncertain if this is attributable to the MSF process or some other mechanism. For example, a prospective longitudinal study which collected MSF data from 250 family physicians on two occasions, five years apart, did find improvements in behaviour, but due to the large gap in time between the two assessments, whether this improvement was attributable to behaviour change as a result of MSF participation was not clear .
Therefore, while participation in MSF is now compulsory for all medical doctors, an exploration of the factors that influence the effectiveness of MSF is a timely one, as it would ensure that MSF is delivered in a way that facilitates acceptance and behaviour change. The purpose of this literature review, therefore, was to identify the key factors that influence the effectiveness of multisource feedback in improving the professional practice of medical doctors. To achieve this we conducted a systematic review of the published evidence to address the following questions:
What impact does MSF have on the professional practice of medical doctors?
What are the main factors of MSF that influence its impact on the professional practice of medical doctors?
This study was conducted and reported according to PRISMA guidelines .
The following electronic bibliographic databases were searched from inception to November 2012 for primary studies: MEDLINE; EMBASE; CINAHL; PsycINFO; Psychology and Behavioral Sciences Collection; Cochrane CENTRAL Database; Applied Social Sciences Index and Abstracts (ASSIA). The search strategies employed included the following terms: peer review, colleague feedback, multisource feedback, psychological feedback, 360 degree feedback and health professional. These words were searched as a combination of key words, and MeSH subject heading terms. The full Medline search strategy is in below:
Exp Peer Review, Health Care/
(feedback adj3 (review$ or colleague or “360” or peer)).mp
(multisource or multi-source or “multisource) adj3 feedback.
((colleague or peer) adj3 assessment).mp
“360 degree evaluation”.mp
1 or 2 or 3 or 4 or 5 or 6
Exp Health Personnel/
7 and 8
To augment the database searches the reference lists of previous literature reviews and of all retrieved eligible studies were reviewed. The journals, Medical Education, Academic Medicine, British Medical Journal, Education for Primary Care, Journal of the American Medical Association and Journal of Continuing Education in the Health Professions were also hand searched from November 2002-November 2012.
Inclusion and exclusion criteria
Published empirical studies with a focus on the development and application of MSF methods by all healthcare professional groups were initially to be included (except literature reviews, as the primary studies reviewed would already have been included). However, a preliminary scoping review identified potentially eligible studies which involved medical professionals only. The focus of this review, therefore, is on published work of relevance to the medical professions.
Barr’s adaptation of Kirkpatrick’s four level evaluation model
Level 1: Le arners’ reaction
Relate to participants’ views of their learning experience and satisfaction with the programme.
Level 2: Learning outcomes
Level 2a: Modification of attitudes/ perceptions
Changes in reciprocal attitudes or perceptions between participant groups, towards patients/clients and their condition, circumstances, care and treatment.
Level 2b : Acquisition of knowledge /skills
Acquisition of concepts, procedures and principles of interprofessional collaboration or the acquisition of thinking/problem-solving, psychomotor and social skills linked to collaboration
Level 3: Change in behaviour
Behavioural change transferred from the learning environment to the workplace prompted by modifications in attitudes or perceptions, or the application of newly acquired knowledge/skills in practice. Overeem et al. (2010) identify that this level can be further separated into:
Level 3a: Self reported change in behaviour
Level 3b : Measured change in performance
Level 4 : Patient /Organisational outcomes
Level 4a : Change in organisational practice
This relates to wider changes in the organisation/delivery of care, attributable to an education programme.
Level 4b : Benefits to patients/ clients
Covers any improvements in the health and well being of patients/clients as a direct result of an education programme.
Study eligibility criteria
Is the aim clearly stated?
Relevant outcome measures?
All studies which met the criteria for inclusion were retrieved. Both reviewers independently read all of the retrieved studies and recommended which studies should be included in the review. Disagreements that arose were reconciled by discussion.
Study quality assessment
Assessment of study quality
Is the research question(s) or hypothesis clearly stated?
Is the subject group appropriate for the study being carried out (number, characteristics, selection, and homogeneity)?
‘Data’ collection methods
Are the methods used (qualitative or quantitative) reliable and valid for the research question and context?
Completeness of ‘data’
Have subjects dropped out? Is the attrition rate less than 50%? For questionnaire based studies, is the response rate acceptable (60% or above)?
Control for confounding
Have multiple factors/variables been removed or accounted for where possible?
Analysis of results
Are the statistical or other methods of results analysis used appropriate?
Is it clear that the data justify the conclusions drawn?
Could the study be repeated by other researchers?
Does the study look forwards in time (prospective) rather than backwards (retrospective)?
Were all relevant ethical issues addressed?
Were results supported by data from more than one source?
Two reviewers (JF and JW) independently undertook data extraction of all studies using a pre-designed proforma. The list below outlines the data retrieved from each article.
Aim/objectives of study
Type of study and Study design
- 3.Setting and population
Number of doctors included in study
- 4.Information regarding MSF instrument
Format of feedback
Kirkpatrick outcomes measured/reported
- 6.Key findings
Influences on change
Summary of characteristics of included studies
First author and Date
Type of study
Brinkman (2007) 
Yes: by a coach
Feedback report about baseline parent and nurse evaluations, and a tailored coaching session
Improved communication with patients & families. Improved demonstration of responsibility & accountability.
Burford (2010) 
Quantitative: Cohort Study employing questionnaires
Mini Peer Assessment Tool (Mini-PAT), Team Assessment of Behaviour (TAB)
Highlighted the need for a facilitator Perceived validity of raters
Intention to change behaviour (no specific examples given)
Fidler (1999) 
Quantitative Questionnaire survey & focus group
Physician Achievement Review (PAR)
Negative mean feedback ratings
Improved communication with patients, better follow-up of patients. Improved written & verbal communication with health professionals
Hall (1999) 
Quantitative Before & after study
Identified need for facilitated feedback. Age of physician. Gap between peer rating and self rating
Improved communication with patients
Lipner (2002) 
Mixed methods – focus groups & questionnaire
Intention to make changes by improving communication with patients (e.g. discuss treatment options more fully), improving communication with peers, and also participate in self-reflection
Lockyer (2003) 
Quantitative Before & after survey
Developed for study
Age of physician. Gap between peer and self ratings
Making printed material available, maintaining medical records, managing stress & improving telephone access for patients.
Overeem (2009) 
Qualitative – grounded theory interview study
PAR, American Board of Internal Medicine (AIM)
Yes: by a "mentor" or "coach"
Facilitated feedback. Reflection on feedback. Self efficacy. Goal setting.
Performance improvement – e.g. improved communication with colleagues.
Overeem (2010) 
Quantitative cross-sectional survey study
PAR, ABIM, Dutch Appraisal and Assessment Instrument (AAI)
Yes: a trained “facilitator”
feedback from colleagues, coworkers and/or patients summarized in a feedback report.
Facilitation Narrative comments
Intention to change professional performance & development of a personal development plan incorporating proposed changes.
Overeem (2012) 
Quantitative observational and questionnaire evaluation study
Yes: by a "mentor"
Report consisting of the collation of MSF ratings from colleagues, coworkers and patients.
Perceived quality of mentoring. Negative scores.
Intention to change one or more aspects of professional performance.
Owens (2010) 
Qualitative focus group and interview study
General Practitioners (trainees and doctors)
No: Doctors. Yes: Trainees-a supervisor.
Report – however format of report varied.
Receiving several comments about the same behaviour
GPs improved communication with staff. Trainees improved their professional behaviour with staff & patients
Sargeant (2003) 
Quantitative pilot study. Questionnaire evaluation survey
Familiarity. Patient feedback Highlighted need for facilitated feedback
Intention to make or had made practice changes – mainly involving communication with patients (esp. written communication, phone communication, waiting times & accessibility)
Sargeant (2005) 
Qualitative Focus groups
No: contact provided if needed
Mailed confidential report
Unbiased yet informed raters. Agreeing with the feedback. Perceived usefulness of feedback. Negative influence – disagreeing with feedback
Examples of changes included improved communication with consultants & patients, improving information provided to patients following diagnostic tests
Sargeant (2007) 
No: contact provided if needed
Mailed confidential report
Familiarity with/credibility of rater. Facilitation. Emotional response. Negative feedback. Patient Feedback. Clear and specific feedback.
Improved communication with patients (e.g. providing fuller explanation) & co-workers. (e.g. improved written/verbal communication with pharmacists)
Sargeant (2008) 
No: contact provided if needed
Mailed confidential report
Negative feedback. Feedback inconsistent with their own self perceptions
Non-specific behaviour changes reported
Sargeant (2009) 
Qualitative – grounded theory. Interview study
No: contact provided if needed
Mailed confidential report
Reflection. Emotional response. Facilitation. Feedback inconsistent with their own self perceptions.
General behaviour changes
Shepherd (2010) 
Mixed methods - questionnaire and interview study
MSF developed for study
Yes: by appraiser
Confidential report – downloaded from a website.
Honesty on part of raters, appraisers and appraisees
Examples given included: improving systems used for communication, changing behaviour in interactions with colleagues, improving delegation
Impact of MSF on medical professionals’ professional practice
Of the 16 studies, only one, an RCT, identified a measured change in behaviour (Kirkpatrick level 3b) . Alongside the MSF report, the intervention group also filled in a self assessment form and took part in a tailored coaching session to assist them in identifying their strengths and weaknesses and in setting specific behavioural goals. The 18 physicians who engaged in MSF demonstrated a significant improvement in a number of professional behaviours including communicating effectively with the patient and family (35%; 95% confidence interval, 11.0%-58.0%), timeliness of completing tasks (30%; 95% confidence interval, 7.9%-53.0%), and demonstrating responsibility and accountability (26%; 95% confidence interval, 2.9%-49.0%) compared to a control group of a further 18 who did not take part in the MSF. However, it is unclear whether this improvement would have occurred without the tailored coaching session. Furthermore, the wide confidence intervals reported in this study mean that these results need to be interpreted with caution and further research is therefore needed, to confirm the findings.
Three studies did not specify the types of changes made [20, 32, 33]. Of those that did, the most commonly reported change in behaviour related to improvement in communication, either with colleagues or patients [19, 21, 22, 25, 28, 29, 31, 34]. Other changes included better follow-up of patients , maintaining medical records,  improving information provided to patients [24, 30] and managing stress .
In two studies intention to change rather than actual change was reported and there was evidence that participants were more likely to contemplate and intend to change their behaviour in some way rather than actually implementing changes [23, 29]. For example, one study of 356 participants found that 42% of the physicians expressed an intention to make changes in their communication with patients and 28% intended to change communication strategies with peers in response to the feedback . However, the study did not explore whether these intended changes resulted in actual change. Also, as the authors acknowledge, it is unclear whether any changes made actually led to improvements in performance.
Main factors of MSF that influence its impact on medical professionals’ professional practice
Source of the feedback
The influence of the raters on the participants’ acceptance and use of the feedback was highlighted by three of the studies [29–31]. A focus group study identified that feedback was only perceived as useful when physicians considered the raters (colleagues or co-workers) to be familiar with their work and had experience of either working with them or observing their practice . This familiarity was related to how useful they perceived the feedback, namely if they felt that the rater was credible (was familiar with their work) the participants were more likely to perceive the feedback as useful and were more likely to consider changing their behaviour as a result . Burford et al. (2010)  also highlighted that acceptance and use of feedback was influenced by whether the physician felt the rater had sufficient knowledge of their work. In their study participants reported that raters were mainly selected for the perceived value of their feedback . Another study identified that the characteristics of the raters were found to be the main influence on physicians’ use of the feedback, especially honesty on the part of the raters . This honesty was also highlighted as a factor by Burford et al. who identified that raters need to aim to be unbiased in their feedback . Nonetheless, nearly half the raters in the study reported using indirect evidence (discussion with colleagues, absence of negative behaviour, inference from other observed behaviour) to assist in the development of their feedback, calling into question the usefulness of their feedback.
Feedback from patients was also found to have a positive influence on physicians’ acceptance and use of the feedback. Indeed, in two studies feedback from patients was shown to have a greater impact on behaviour change than feedback from colleagues [29, 31]. In these two studies the main behavioural change made concerned communication with patients; this was probably due to the fact that patients directly observe the behaviour of this particular group (family doctors) to a greater extent than colleagues or co-workers. In addition, the MSF tool used - the Physician Achievement Review – included questionnaire items which provided a clear direction for physicians to change their communication behaviour .
Format of the feedback
In six of the studies feedback was facilitated, either by an appraiser,  mentor, [25, 27] facilitator,  supervisor  or coach . This facilitation was said to enable the acceptance of the feedback and the setting of achievable goals . In one study, the quality of this facilitation was also key to the physicians’ acceptance and subsequent behavioural change . In the other 11 studies the feedback was non-facilitated and in the form of a confidential report. While four of the studies identified that contact was provided if needed, they did not specify the format of this contact, how many physicians accessed this support and their reaction to this support [30–33]. In six studies, where the feedback was non-facilitated, a need for facilitated feedback was identified as it was felt that facilitated feedback would positively influence the acceptance of the feedback or the management of the emotional response associated with the feedback [20, 22, 26, 29, 32, 33].
MSF reports can comprise numerical scores only or may incorporate narrative comments as well. There was evidence that participants preferred to receive written comments rather than just a numerical score [20, 25]. While one study identified only a small, though significant, preference for free-text comments from raters,  another found that written comments from the raters were essential to the physicians satisfaction with and acceptance of the feedback .
Content of the feedback
Three studies identified that receiving negative comments influenced whether physicians contemplated or initiated changes [21, 31, 32]. For example, Fidler et al. found that physicians who had more negative ratings in their feedback report contemplated or initiated more behavioural changes than physicians who did not receive as many negative ratings . Three of the studies found that when the physicians received positive feedback that confirmed good professional practice, they saw no need for change [31–33]. Receiving several comments from different sources about the same behaviour was also found to facilitate the use of the feedback received .
Response to the feedback
Two studies found that the process of reflecting on the MSF feedback was an important factor in its use [26, 33]. Sargeant et al. found that reflection was integral to whether the feedback received was learned from and used to change professional practice . The participants in this study highlighted that, for the feedback to be accepted and used, reflection needed to be facilitated, with the facilitator supporting and guiding the reflective process. However, it is unclear whether this reflection would have been as effective had it not been facilitated. This was mirrored by Overeem et al. (2009) who highlighted that the facilitator supported the reflective process ensuring that both strengths and weaknesses of the physician are focussed on .
The physicians’ emotional response to the feedback was also found to influence the acceptance and use of the feedback [30–33]. Physicians who received feedback that was negative and/or inconsistent with their self-perceptions, were more likely to respond with emotional distress [32, 33]. In an interview study with 28 family physicians who had participated in MSF, Sargeant et al. (2008) found that while some eventually accepted the feedback and went on to initiate change, others did not and, rather than accept the feedback, questioned the feedback process .
In two of the studies, it was found that when the physicians disagreed with the feedback they questioned its credibility and asserted that they would not make any changes to their practice until they verified the responses by conducting their own independent reviews of their practice [30, 31].
This systematic review synthesised the available evidence concerning the educational impact of MSF, defined by Overeem et al. (2012) as “the impact of MSF on change in practice”,  and its influence in changing behaviours and improving doctors’ performance. While a number of systematic reviews have looked at the effect of various types of workplace based assessment on the professional behaviours of medical doctors, to date no review has been carried out that focuses specifically on multisource feedback [7, 8, 11, 35]. This review sought to address this gap.
Considering the importance now placed on workplace based assessment, in particular MSF, there are surprisingly few published articles exploring the impact of MSF on physician behaviour and the factors that influence the effectiveness of MSF. Research has tended to focus more on the validity and reliability of different tools and on the acceptability of MSF to those using this assessment tool, rather than on its educational impact on improving professional performance and behaviours where judged to be necessary.
One of the main factors identified that could enhance MSF acceptance and use was whether feedback is facilitated, by an appraiser, mentor or coach. Furthermore, in five of the studies where feedback was not facilitated, physicians identified a need for facilitated feedback [20, 22, 29, 31, 32]. It was felt that facilitation would assist doctors in accepting the feedback and in using it to change their behaviour [31, 32].
It has been hypothesised that receiving negative or discrepant feedback (ratings from others that are lower than self-ratings) could be a catalyst for change, however evidence supporting this is limited . In reality, it has been found that physicians who rated themselves higher than others are more likely to react negatively to the feedback and not change their behaviour  The influence of negative feedback, or feedback that was not consistent with their own self perceptions was also highlighted by six studies in this current review [21, 22, 26, 30, 31, 33]. Specifically, Overeem et al. (2012) identified that the quality of the mentoring and the receipt of negative scores by colleagues were the main motivators for change for participating physicians . The regression analysis undertaken found that the two variables, quality of mentoring and scores by colleagues (physicians who received lower ratings from their medical colleagues reported more change in practice) accounted for 34% of the variance . While the interaction between these two variables is unclear in this study, two studies highlighted that the role of the facilitator is key to the physicians’ acceptance of the feedback [31, 33]. The facilitator was said to encourage reflection and subsequent acceptance of the feedback which in turn led to a change in behaviour.
The findings of this review are similar to other reviews of MSF in non-healthcare settings. For example Smither et al. (2005) carried out a meta-analysis of 24 longitudinal studies of MSF in non-healthcare organisational settings, focussing on how managers use the MSF feedback over time . The authors concluded that following participation in MSF, eight factors were identified as playing a part in influencing the extent of the change in behaviour and performance improvement . These factors are:
Characteristics of the feedback, initial reactions to feedback, personality, feedback orientation, perceived need for change, beliefs about change, goal setting, and taking action .
As with this current review, if the recipients reacted negatively and rejected the feedback they were unlikely to use it to change their behaviour. However, unlike this current review, Smither (2005) did not look at the content or format of the feedback. Bracken and Rose (2011) carried out a review of the literature focussing on the factors of the MSF process that facilitates behaviour change, and like this current review, identified that the credibility of the rater was an important factor .
Strengths and limitations
Strengths of the study were that a recognised systematic review process was followed  and papers meeting the study criteria were appraised by two researchers. Data extraction using a pre-designed proforma was independently undertaken by both researchers ensuring that no relevant data were overlooked. Applying Buckley et al’s  quality criteria ensured that studies of very poor quality were eliminated. To evaluate outcomes, Barr’s (2000) adaptation of Kirkpatrick’s four level evaluation model  was applied to all study findings; this ensured that only the outcomes relevant to the study were included.
This study has some limitations. Firstly, while the literature search was extensive and covered a number of databases, the articles retrieved were restricted to English-language publications only, and therefore publication bias cannot be ruled out. Secondly, methodological quality varied extensively between the articles and most of the studies were conducted on small volunteer-based samples. Also, as this review only focussed on published empirical studies some relevant unpublished studies may have been missed. Lastly, two studies included in this review have been reported in multiple articles. The articles by Fidler et al. (1999)  and Hall et al. (1999)  report on different aspects of the same study and the five articles reported by Sargeant et al. (2003, 2005, 2007, 2008, 2009) [29–33] all relate to the same sample population, from the pilot study reported by Sargeant et al. (2003) . This could lead to over-reporting of the findings .
Whatever review method is employed, the resultant synthesis is only equivalent to the primary data included in the synthesis and the studies identified in this review had a number of limitations. Firstly, 15 of the 16 studies were descriptive and therefore the strength of the findings may be limited by the uncontrolled nature of the studies. However, it is asserted that, given the methodological difficulties of evaluating health professionals’ performance and the educational impact of interventions, descriptive and observational studies can still provide valuable information . In fact, in this review the strongest evidence for improved performance following the MSF intervention and for the influencing factors for the effectiveness of the MSF intervention was provided by a focus group study,  and interview studies [31, 33]. However, it should be noted that all three articles are from the same sample population and therefore publication bias may exist.
Fifteen of the 16 studies described self-reported rather than measured changes in behaviour. While self-reported measures of change in behaviour are highlighted as being quicker and easier than measuring actual behaviours, it is argued that healthcare systems are not concerned with changing health professionals self-reported behaviour, rather they want to change actual behaviour in the belief that this should lead to improved patient care . Both Eccles et al. (2006)  and Hrisos et al. (2009)  purport that while self-reported change in behaviour can be a useful proxy for actual behaviour the extent to which these measures accurately reflect actual behaviour is unclear.
Some of the studies reported intention to change rather than actual change in behaviour [20, 23, 26, 27]. A number of health psychology theories, such as the theory of planned behaviour [42–44] and protection motivation theory [45–47] assume that intentions (a person’s motivation or decision to engage in a course of action ) cause behaviour. However, as studies focussing on the intention-behaviour relationship have generally been correlational rather than causational this causality is still to be determined . While Webb and Sheeran (2006) cite a number of meta-analyses that illustrate a relationship between intention and behaviour they highlight that this relationship could be in the reverse direction (that behaviour caused the intention) . Therefore, it cannot be conclusively determined whether intention actually leads to behaviour change as a result of an MSF intervention.
Implications for practice
An MSF tool which is designed to incorporate feedback from ‘credible’ raters (i.e. people who know the physician’s practice well, including patients) and which includes narrative comments as well as numerical scores is desirable. Furthermore, it seems to be helpful if the feedback is facilitated as this can influence the physician’s response to the feedback, how they deal with any negative comments and can promote reflection, all of which can impact on how the physician accepts and uses the feedback. Therefore when implementing an MSF intervention, particularly on an organisational or national basis, these factors need to be considered in order to optimise the effectiveness of the process.
Implications for research
Considering the emphasis now placed on MSF as a method of formative performance assessment, there are surprisingly few published articles exploring these areas, and the strength of the study findings is questionable. While the studies identified in this review did examine both the behaviours influenced and the influencing factors, further research needs to be carried out to identify how the feedback influences behaviour, the strength of the influencing factors, and what impact behaviour change has on the quality and safety of patient care. In addition, only one study  looked at the change in behaviour over time, and in that study the follow-up was only five months. Therefore, it is not known whether the changes made are sustained over a longer period of time. Furthermore, while feedback is increasingly being used with health professionals other than primary and secondary care physicians, this review identified no eligible published studies involving these clinical groups which may be a cause for concern if MSF is also to be implemented as a performance measure by these professions.
The positive influence of a facilitator in the MSF process was highlighted in this review. However further research is clearly necessary to uncover the role and influence of the facilitator in the feedback process, particularly given the resource and logistical implications of providing such facilitation at scale as part of a national MSF system. Furthermore, research is also required to determine whether facilitation on its own is the key influence underpinning the impact of MSF, or in combination with the other factors identified as important (such as the source and format of the feedback).
This review has identified that participation in MSF by medical doctors can lead to improved performance, although the volume and quality of supporting evidence is limited. Several factors were found to influence the extent to which professional performance is improved, which are summarised below.
Source of the feedback:
How the feedback is delivered:
Content of the feedback:
Receiving a number of comments about the same behaviour encourages acceptance and change 
- Gopee N: Self assessment and the concept of the lifelong learning nurse. Brit J Nurs. 2000, 9: 724-729.View ArticleGoogle Scholar
- Epstein RM, Hundert EM: Defining and assessing professional competence. JAMA. 2002, 287: 226-235. 10.1001/jama.287.2.226.View ArticleGoogle Scholar
- Epstein RM: Assessment in medical education. New Engl J Med. 2007, 356: 387-396. 10.1056/NEJMra054784.View ArticleGoogle Scholar
- Hayes OW, Reisdorff EJ, Walker GL, Carlson DJ, Reinoehl B: Using standardized oral examinations to evaluate general competencies. Acad Emerg Med. 2002, 9: 1334-1337. 10.1111/j.1553-2712.2002.tb01597.x.View ArticleGoogle Scholar
- Wilkinson TJ, Wade WB, Knock LD: A blueprint to assess professionalism: results of a systematic review. Acad Med. 2009, 84: 551-558. 10.1097/ACM.0b013e31819fbaa2.View ArticleGoogle Scholar
- Norcini J, Burch V: Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007, 29: 855-871. 10.1080/01421590701775453.View ArticleGoogle Scholar
- Miller A, Archer J: Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2007, 341: 710-715.View ArticleGoogle Scholar
- Overeem K, Faber MJ, Arah OA, Elwyn G, Lombarts KM, Wollersheim HC, Grol RP: Doctor performance assessment in daily practise: does it help doctors or not? A systematic review. Med Educ. 2007, 41: 1039-1049. 10.1111/j.1365-2923.2007.02897.x.View ArticleGoogle Scholar
- van der Vleuten CPM, Schuwirth LWT, Scheele MD, Driessen EW, Hodges B: The assessment of professional competence: building blocks for theory development. Best Pract Res Cl Ob. 2010, 24: 703-719.View ArticleGoogle Scholar
- van der Vleuten CPM, Norman GR, De Graaf E: Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ. 1991, 25: 110-118. 10.1111/j.1365-2923.1991.tb00036.x.View ArticleGoogle Scholar
- General Medical Council: Supporting information for appraisal and revalidation. 2012, London: General Medical CouncilGoogle Scholar
- Alexander DM: How do 360 degree performance reviews affect employee attitudes, effectiveness and performance?. University of Rhode Island, Schmidt Labor Research Center Seminar Research Series [online]. 2006, 1-12. [http://www.uri.edu/research/lrc/research/papers/Alexander_360.pdf]Google Scholar
- Brett J, Atwater L: 360 degree feedback: Accuracy, reactions and perceptions of usefulness. J Appl Psychol. 2001, 86: 930-942.View ArticleGoogle Scholar
- Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD: Audit and feedback: effects on professional practice and health care outcomes. Cochrane Db Syst Rev. 2006, 2: Art. No.: CD000259Google Scholar
- Violato C, Lockyer JM, Fidler H: Changes in performance: a 5‒year longitudinal study of participants in a multi‒source feedback programme. Med Educ. 2008, 42: 1007-1013. 10.1111/j.1365-2923.2008.03127.x.View ArticleGoogle Scholar
- Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009, 6: e1000100-10.1371/journal.pmed.1000100.View ArticleGoogle Scholar
- Barr H, Freeth D, Hammick M, Koppel I, Reeves S: Evaluations of Interprofessional Education: A United Kingdom Review of Health and Social Care. 2000, London: CAIPEGoogle Scholar
- Buckley S, Coleman J, Davison I, Khan KS, Zamora J, Malick S, Morely D, Pollard D, Ashcroft T, Popovic C, Sayers J: The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Med Teach. 2009, 31: 282-298. 10.1080/01421590902889897.View ArticleGoogle Scholar
- Brinkman WB, Geraghty SR, Lanphear BP, Khoury JC, Gonzalez Del Rey JA, De Witt TG, Britto MT: Effect of multisource feedback on resident communication skills and professionalism: a randomized controlled trial. Arch Pediat Adol Med. 2007, 161: 44-49. 10.1001/archpedi.161.1.44.View ArticleGoogle Scholar
- Burford B, Illing J, Kergon C, Morrow G, Livingston M: User perceptions of multi-source feedback tools for junior doctors. Med Educ. 2010, 44: 165-176. 10.1111/j.1365-2923.2009.03565.x.View ArticleGoogle Scholar
- Fidler H, Lockyer JM, Toews J, Violato C: Changing physicians' practices: the effect of individual feedback. Acad Med. 1999, 74: 702-10.1097/00001888-199906000-00019.View ArticleGoogle Scholar
- Hall W, Violato C, Lewkonia R, Lockyer J, Fidler H, Toews J, Jennett P, Donoff M, Moores D: Assessment of physician performance in Alberta: the physician achievement review. Can Med Assoc J. 1999, 161: 52-57.Google Scholar
- Lipner RS, Blank LL, Leas BF, Fortna GS: The value of patient and peer ratings in recertification. Acad Med. 2002, 77: S64-10.1097/00001888-200210001-00021.View ArticleGoogle Scholar
- Lockyer J, Violato C, Fidler H: Likelihood of change: a study assessing surgeon use of multisource feedback data. Teach Learn Med. 2003, 15: 168-174. 10.1207/S15328015TLM1503_04.View ArticleGoogle Scholar
- Overeem K, Lombarts MJMH, Arah OA, Klazinga NS, Grol RP, Wollersheim HC: Three methods of multi-source feedback compared: a plea for narrative comments and coworkers' perspectives. Med Teach. 2010, 32: 141-147. 10.3109/01421590903144128.View ArticleGoogle Scholar
- Overeem K, Wollersheim H, Driessen E, Lombarts K, Van De Ven G, Grol R, Arah O: Doctors’ perceptions of why 360-degree feedback does (not) work: a qualitative study. Med Educ. 2009, 43: 874-882. 10.1111/j.1365-2923.2009.03439.x.View ArticleGoogle Scholar
- Overeem K, Wollersheimh HC, Arah OA, Cruijsberg JK, Grol RP, Lombarts KM: Factors predicting doctors’ reporting of performance change in response to multisource feedback. BMC Med Educ. 2012, 12: 52-10.1186/1472-6920-12-52.View ArticleGoogle Scholar
- Owens J: Is multi-source feedback (MSF) seen as a useful educational tool in primary care? A qualitative study. Educ Prim Care. 2010, 21: 180.View ArticleGoogle Scholar
- Sargeant JM, Mann KV, Ferrier SN, Langille DB, Muirhead PD, Hayes VM, Sinclair DE: Responses of rural family physicians and their colleague and coworker raters to a multi-source feedback process: a pilot study. Acad Med. 2003, 78: S42-10.1097/00001888-200310001-00014.View ArticleGoogle Scholar
- Sargeant J, Mann K, Ferrier S: Exploring family physicians' reactions to multisource feedback: perceptions of credibility and usefulness. Med Educ. 2005, 39: 497-504. 10.1111/j.1365-2929.2005.02124.x.View ArticleGoogle Scholar
- Sargeant J, Mann K, Sinclair D, Van der Vleuten C, Metsemakers J: Challenges in multisource feedback: intended and unintended outcomes. Med Educ. 2007, 41: 583-591. 10.1111/j.1365-2923.2007.02769.x.View ArticleGoogle Scholar
- Sargeant J, Mann K, Sinclair D, Van der Vleuten C, Metsemakers J: Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Adv Health Sci Educ Theory Pract. 2008, 13: 275-288. 10.1007/s10459-006-9039-x.View ArticleGoogle Scholar
- Sargeant JM, Mann KV, van der Vleuten CP, Metsemakers JF: Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ Theory Pract. 2009, 14: 399-410. 10.1007/s10459-008-9124-4.View ArticleGoogle Scholar
- Shepherd A, Lough M: What is a good general practitioner (GP)? The development and evaluation of a multi-source feedback instrument for GP appraisal. Educ Prim Care. 2010, 21: 149-164.View ArticleGoogle Scholar
- Veloski J: Systematic revew of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach. 2006, 28: 117-128. 10.1080/01421590600622665.View ArticleGoogle Scholar
- Smither JW, London M, Reilly RR: Does performance improve following multisource feedback? A theoretical model, meta-analysis and review of empirical findings. Pers Psychol. 2005, 58: 33-66. 10.1111/j.1744-6570.2005.514_1.x.View ArticleGoogle Scholar
- Bracken DW, Rose DS: When does 360-degree feedback create behavior change? And how would we know it when it does?. J Bus Psychol. 2011, 26: 183-192. 10.1007/s10869-011-9218-5.View ArticleGoogle Scholar
- Centre for Reviews and Dissemination: Systematic Reviews CRD’s Guidance for Undertaking Reviews in Health Care. 2008, Heslington: CRD, University of YorkGoogle Scholar
- Von Elm E, Poglia G, Walder B, Tramer MR: Different patterns of duplicate publication. JAMA. 2004, 291: 974-980. 10.1001/jama.291.8.974.View ArticleGoogle Scholar
- Eccles MP, Hrisos S, Francis J, Kaner EF, Dickinson HO, Beyer F, Johnston M: Do self-reported intentions predict clinicians' behaviour: a systematic review. Implement Science. 2004, 1: 28.View ArticleGoogle Scholar
- Hrisos S, Eccles MP, Francis JJ, Dickinson HO, Kaner EF, Beyer F, Johnston M: Are there valid proxy measures of clinical behaviour? A systematic review. Implement Science. 2009, 4: 37-10.1186/1748-5908-4-37.View ArticleGoogle Scholar
- Ajzen I: From intentions to actions: A theory of planned behavior. Action-control: From cognition to behavior. Edited by: Kuhl J, Beckmann J. 1985, Heidelberg: Springer, 11-39.View ArticleGoogle Scholar
- Ajzen I, Madden TJ: Prediction of goal-directed behavior: attitudes, intentions, and perceived behavioral control. J Exp Soc Psychol. 1986, 22: 453-474. 10.1016/0022-1031(86)90045-4.View ArticleGoogle Scholar
- Ajzen I: Attitudes, personality and behavior. 1988, Chicago: Dorsey PressGoogle Scholar
- Rogers RW: A protection motivation theory of fear appeals and attitude change. J Psychol. 1975, 91: 93-114. 10.1080/00223980.1975.9915803.View ArticleGoogle Scholar
- Rogers RW: Cognitive and physiological processes in fear appeals and attitude change: A revised theory of protection motivation. Social psychophysiology: A source book. Edited by: Cacioppo BL BL, Petty LL. 1983, London: Guildford Press, 153-176.Google Scholar
- Rogers RW: Attitude change and information integration in fear appeals. Psychol Rep. 1985, 56: 179-182. 10.2466/pr0.19188.8.131.52.View ArticleGoogle Scholar
- Webb TL, Sheeran P: Does changing behavioral intentions engender behavior change? A meta-analysis of the experimental evidence. Psychol Bull. 2006, 132: 249-268.View ArticleGoogle Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/14/76/prepub
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.