Skip to main content
  • Research article
  • Open access
  • Published:

Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: a systematic review

Abstract

Background

Multisource feedback (MSF) is currently being introduced in the UK as part of a cycle of performance review for doctors. However, although it is suggested that the provision of feedback can lead to a positive change in performance and learning for medical professionals, the evidence supporting these assumptions is unclear. The aim of this review, therefore, was to identify the key factors that influence the effectiveness of multisource feedback in improving the professional practice of medical doctors.

Method

Relevant electronic bibliographic databases were searched for studies that aimed to assess the impact of MSF on professional practice. Two reviewers independently selected and quality assessed the studies and abstracted data regarding study design, setting, MSF instrument, behaviour changes identified and influencing factors using a standard data extraction form.

Results

A total of 16 studies met the inclusion criteria and quality assessment criteria. While seven studies reported only a general change in professional practice, a further seven studies identified specific changes in behaviour. The main professional behaviours that were found to be influenced by the feedback were communication, both with colleagues and patients and an improvement in clinical competence/skills. The main factors found to influence the acceptance and use of MSF were the format of the feedback, specifically in terms of whether it was facilitated, or if narrative comments were included in the review, and if the feedback was from sources that the physician believed to be knowledgeable and credible.

Conclusions

While there is limited evidence suggesting that MSF can influence professional performance, the quality of this evidence is variable. Further research is necessary to establish how this type of feedback actually influences behaviours and what factors have greatest influence.

Peer Review reports

Background

Medical practitioners are now professionally accountable via periodic revalidation for the standard of patient services they provide [1]. The assessment of professional performance is one way to demonstrate fitness to practise medicine and provide evidence of accountability. Medical regulators and educational bodies are endeavouring to identify reliable and robust methods to routinely assess the competence of trainee and qualified doctors to contribute supporting evidence of ongoing performance [2, 3]. However, while there is agreement on the need for regular performance assessment of doctors, how best to do this is unclear. Assessment methods involving simulated patients, video observation, audits of clinical records, critical incident analysis and multisource feedback are all suggested, but these approaches involve subjective judgements and lack adequate psychometric evaluation [2, 47]. An earlier systematic review of a diverse range of assessment methods highlighted the lack of rigour, effectiveness and overall utility of most approaches [8]. However, in terms of costs and time considerations, feedback from colleagues (also known as multisource feedback) was deemed to be the most appropriate and practical method. Moreover, whilst MSF may be seen as a collection of subjective assessments, studies of assessment methods have concluded that subjective methods can be as reliable as objective methods provided the sample size of assessors is adequate [9, 10].

As such, multisource feedback (MSF) is being introduced by the UK medical regulator, the General Medical Council (GMC), [11] as part of a cycle of performance review for all medical doctors. The GMC asserts that this feedback needs to be from more than a single source (e.g. self, medical colleagues and non-medical co-workers) and cover the whole of the individual’s professional practice [11].

MSF is a questionnaire-based process, which allows for the feedback to be collected in a systematic way [11]. The intent of the feedback is to guide behaviour change and improve performance [12]. The participant usually receives a copy of their own data (presented as aggregate scores for each element of perceived professional performance being assessed by colleagues) along with comparison data for their group. Graphical as well as numerical data may be provided to the participant. Obtaining more than one person’s view of colleague performance means that a “more complete and accurate assessment of the employees’ competencies” [12] is generated. Many MSF tools also require participants to self-assess, by completing the questionnaire themselves as a means of comparing self and others’ perceptions of professional performance [13].

Although it is suggested that the provision of feedback can lead to learning and a positive change in performance for medical professionals, the evidence supporting these assumptions is unclear [14]. Even with studies which have reported performance improvement over time, it is uncertain if this is attributable to the MSF process or some other mechanism. For example, a prospective longitudinal study which collected MSF data from 250 family physicians on two occasions, five years apart, did find improvements in behaviour, but due to the large gap in time between the two assessments, whether this improvement was attributable to behaviour change as a result of MSF participation was not clear [15].

Therefore, while participation in MSF is now compulsory for all medical doctors, an exploration of the factors that influence the effectiveness of MSF is a timely one, as it would ensure that MSF is delivered in a way that facilitates acceptance and behaviour change. The purpose of this literature review, therefore, was to identify the key factors that influence the effectiveness of multisource feedback in improving the professional practice of medical doctors. To achieve this we conducted a systematic review of the published evidence to address the following questions:

  • What impact does MSF have on the professional practice of medical doctors?

  • What are the main factors of MSF that influence its impact on the professional practice of medical doctors?

Methods

This study was conducted and reported according to PRISMA guidelines [16].

Search Strategy

The following electronic bibliographic databases were searched from inception to November 2012 for primary studies: MEDLINE; EMBASE; CINAHL; PsycINFO; Psychology and Behavioral Sciences Collection; Cochrane CENTRAL Database; Applied Social Sciences Index and Abstracts (ASSIA). The search strategies employed included the following terms: peer review, colleague feedback, multisource feedback, psychological feedback, 360 degree feedback and health professional. These words were searched as a combination of key words, and MeSH subject heading terms. The full Medline search strategy is in below:

Search strategy for MEDLINE (searched from inceptionNovember 2012)

  1. 1.

    Exp Peer Review, Health Care/

  2. 2.

    (feedback adj3 (review$ or colleague or “360” or peer)).mp

  3. 3.

    (multisource or multi-source or “multisource) adj3 feedback.

  4. 4.

    ((colleague or peer) adj3 assessment).mp

  5. 5.

    “360 degree evaluation”.mp

  6. 6.

    Feedback, Psychological/

  7. 7.

    1 or 2 or 3 or 4 or 5 or 6

  8. 8.

    Exp Health Personnel/

  9. 9.

    7 and 8

To augment the database searches the reference lists of previous literature reviews and of all retrieved eligible studies were reviewed. The journals, Medical Education, Academic Medicine, British Medical Journal, Education for Primary Care, Journal of the American Medical Association and Journal of Continuing Education in the Health Professions were also hand searched from November 2002-November 2012.

Inclusion and exclusion criteria

Published empirical studies with a focus on the development and application of MSF methods by all healthcare professional groups were initially to be included (except literature reviews, as the primary studies reviewed would already have been included). However, a preliminary scoping review identified potentially eligible studies which involved medical professionals only. The focus of this review, therefore, is on published work of relevance to the medical professions.

To evaluate outcomes, Barr’s (2000) adaptation of Kirkpatrick’s four level evaluation model [17] was applied to all study findings (Table 1). Our focus was on capturing changes in professional behaviours, so levels 3a (self-reported change in behaviour) and 3b (measured change in performance) were of direct relevance. Identified studies, therefore, had to report evidence of the assessment of at least one of these outcomes in order to be included. Studies concerned with psychometric development and validation and others that only measured knowledge or performance in a test situation, or behaviour not unique to the clinical environment (e.g. teaching or academic research) were excluded.

Table 1 Barrs adaptation of Kirkpatricks four level evaluation model

Studies identified through the literature searches underwent an inclusion review process. Both primary reviewers (JF and JW) independently screened the titles and the abstracts of the citations identified by the electronic searches. Using the screening criteria in Table 2, the reviewers made eligibility judgments on whether studies should be included or excluded. Disagreements between reviewers’ recommendations were resolved by retrieving the paper and reaching a consensus after re-review.

Table 2 Study eligibility criteria

All studies which met the criteria for inclusion were retrieved. Both reviewers independently read all of the retrieved studies and recommended which studies should be included in the review. Disagreements that arose were reconciled by discussion.

Study quality assessment

We adopted the quality criteria developed by Buckley et al. (2009), which has been utilised in a previous systematic review of workplace based assessment incorporating qualitative, quantitative and mixed methods studies, [7] to assess the quality of quantitative and qualitative studies included [18]. These criteria consist of a series of 11 quality ‘indicators’ (Table 3) with higher quality studies considered to be those which meet a minimum of seven indicators [18].

Table 3 Assessment of study quality

Data extraction

Two reviewers (JF and JW) independently undertook data extraction of all studies using a pre-designed proforma. The list below outlines the data retrieved from each article.

Information retrieved from included papers

  1. 1.

    Aim/objectives of study

  2. 2.

    Type of study and Study design

  3. 3.

    Setting and population

    1. a.

      Location

    2. b.

      Specialty

    3. c.

      Number of doctors included in study

  4. 4.

    Information regarding MSF instrument

    1. a.

      MSF questionnaire

    2. b.

      Format of feedback

  5. 5.

    Kirkpatrick outcomes measured/reported

  6. 6.

    Key findings

    1. a.

      Changes identified

    2. b.

      Influences on change

  7. 7.

    Conclusions

Results

The combined search of electronic databases yielded 3425 articles (Figure 1). After initial review, 87 were considered to be potentially relevant and the full article texts were obtained. Of these, 70 articles were then excluded as they did not meet the eligibility criteria (Table 3). The remaining 17 articles were assessed for study quality, with a further article being discounted, as it was deemed to be of insufficient quality. A total of 16 articles, therefore, met the inclusion criteria and were judged to be of sufficient quality to answer our study questions.

Figure 1
figure 1

Review process.

Included articles employed a range of study designs: RCT (n = 1); cross sectional survey questionnaires (n = 7); mixed methods (n = 2); and qualitative methods (n = 6). The studies focused on primary care physicians (n = 9), secondary care physicians (medical specialists (n = 3), surgeons (n = 1), paediatricians (n = 1)), foundation trainees (n = 1) and an unspecified physician group or setting (n = 1). The studies were conducted in Canada (n = 8), the UK (n = 3), the Netherlands (n = 3) and the USA (n = 2). A summary of study designs, populations and demographics is outlined in Table 4.

Table 4 Summary of characteristics of included studies

Impact of MSF on medical professionals’ professional practice

Of the 16 studies, only one, an RCT, identified a measured change in behaviour (Kirkpatrick level 3b) [19]. Alongside the MSF report, the intervention group also filled in a self assessment form and took part in a tailored coaching session to assist them in identifying their strengths and weaknesses and in setting specific behavioural goals. The 18 physicians who engaged in MSF demonstrated a significant improvement in a number of professional behaviours including communicating effectively with the patient and family (35%; 95% confidence interval, 11.0%-58.0%), timeliness of completing tasks (30%; 95% confidence interval, 7.9%-53.0%), and demonstrating responsibility and accountability (26%; 95% confidence interval, 2.9%-49.0%) compared to a control group of a further 18 who did not take part in the MSF. However, it is unclear whether this improvement would have occurred without the tailored coaching session. Furthermore, the wide confidence intervals reported in this study mean that these results need to be interpreted with caution and further research is therefore needed, to confirm the findings.

Three studies did not specify the types of changes made [20, 32, 33]. Of those that did, the most commonly reported change in behaviour related to improvement in communication, either with colleagues or patients [19, 21, 22, 25, 28, 29, 31, 34]. Other changes included better follow-up of patients [21], maintaining medical records, [24] improving information provided to patients [24, 30] and managing stress [24].

In two studies intention to change rather than actual change was reported and there was evidence that participants were more likely to contemplate and intend to change their behaviour in some way rather than actually implementing changes [23, 29]. For example, one study of 356 participants found that 42% of the physicians expressed an intention to make changes in their communication with patients and 28% intended to change communication strategies with peers in response to the feedback [23]. However, the study did not explore whether these intended changes resulted in actual change. Also, as the authors acknowledge, it is unclear whether any changes made actually led to improvements in performance.

Main factors of MSF that influence its impact on medical professionals’ professional practice

Source of the feedback

The influence of the raters on the participants’ acceptance and use of the feedback was highlighted by three of the studies [2931]. A focus group study identified that feedback was only perceived as useful when physicians considered the raters (colleagues or co-workers) to be familiar with their work and had experience of either working with them or observing their practice [30]. This familiarity was related to how useful they perceived the feedback, namely if they felt that the rater was credible (was familiar with their work) the participants were more likely to perceive the feedback as useful and were more likely to consider changing their behaviour as a result [30]. Burford et al. (2010) [20] also highlighted that acceptance and use of feedback was influenced by whether the physician felt the rater had sufficient knowledge of their work. In their study participants reported that raters were mainly selected for the perceived value of their feedback [20]. Another study identified that the characteristics of the raters were found to be the main influence on physicians’ use of the feedback, especially honesty on the part of the raters [34]. This honesty was also highlighted as a factor by Burford et al. who identified that raters need to aim to be unbiased in their feedback [20]. Nonetheless, nearly half the raters in the study reported using indirect evidence (discussion with colleagues, absence of negative behaviour, inference from other observed behaviour) to assist in the development of their feedback, calling into question the usefulness of their feedback.

Feedback from patients was also found to have a positive influence on physicians’ acceptance and use of the feedback. Indeed, in two studies feedback from patients was shown to have a greater impact on behaviour change than feedback from colleagues [29, 31]. In these two studies the main behavioural change made concerned communication with patients; this was probably due to the fact that patients directly observe the behaviour of this particular group (family doctors) to a greater extent than colleagues or co-workers. In addition, the MSF tool used - the Physician Achievement Review – included questionnaire items which provided a clear direction for physicians to change their communication behaviour [31].

Format of the feedback

In six of the studies feedback was facilitated, either by an appraiser, [34] mentor, [25, 27] facilitator, [26] supervisor [28] or coach [19]. This facilitation was said to enable the acceptance of the feedback and the setting of achievable goals [26]. In one study, the quality of this facilitation was also key to the physicians’ acceptance and subsequent behavioural change [27]. In the other 11 studies the feedback was non-facilitated and in the form of a confidential report. While four of the studies identified that contact was provided if needed, they did not specify the format of this contact, how many physicians accessed this support and their reaction to this support [3033]. In six studies, where the feedback was non-facilitated, a need for facilitated feedback was identified as it was felt that facilitated feedback would positively influence the acceptance of the feedback or the management of the emotional response associated with the feedback [20, 22, 26, 29, 32, 33].

MSF reports can comprise numerical scores only or may incorporate narrative comments as well. There was evidence that participants preferred to receive written comments rather than just a numerical score [20, 25]. While one study identified only a small, though significant, preference for free-text comments from raters, [20] another found that written comments from the raters were essential to the physicians satisfaction with and acceptance of the feedback [25].

Content of the feedback

Three studies identified that receiving negative comments influenced whether physicians contemplated or initiated changes [21, 31, 32]. For example, Fidler et al. found that physicians who had more negative ratings in their feedback report contemplated or initiated more behavioural changes than physicians who did not receive as many negative ratings [21]. Three of the studies found that when the physicians received positive feedback that confirmed good professional practice, they saw no need for change [3133]. Receiving several comments from different sources about the same behaviour was also found to facilitate the use of the feedback received [28].

Response to the feedback

Two studies found that the process of reflecting on the MSF feedback was an important factor in its use [26, 33]. Sargeant et al. found that reflection was integral to whether the feedback received was learned from and used to change professional practice [33]. The participants in this study highlighted that, for the feedback to be accepted and used, reflection needed to be facilitated, with the facilitator supporting and guiding the reflective process. However, it is unclear whether this reflection would have been as effective had it not been facilitated. This was mirrored by Overeem et al. (2009) who highlighted that the facilitator supported the reflective process ensuring that both strengths and weaknesses of the physician are focussed on [26].

The physicians’ emotional response to the feedback was also found to influence the acceptance and use of the feedback [3033]. Physicians who received feedback that was negative and/or inconsistent with their self-perceptions, were more likely to respond with emotional distress [32, 33]. In an interview study with 28 family physicians who had participated in MSF, Sargeant et al. (2008) found that while some eventually accepted the feedback and went on to initiate change, others did not and, rather than accept the feedback, questioned the feedback process [32].

In two of the studies, it was found that when the physicians disagreed with the feedback they questioned its credibility and asserted that they would not make any changes to their practice until they verified the responses by conducting their own independent reviews of their practice [30, 31].

Discussion

This systematic review synthesised the available evidence concerning the educational impact of MSF, defined by Overeem et al. (2012) as “the impact of MSF on change in practice”, [27] and its influence in changing behaviours and improving doctors’ performance. While a number of systematic reviews have looked at the effect of various types of workplace based assessment on the professional behaviours of medical doctors, to date no review has been carried out that focuses specifically on multisource feedback [7, 8, 11, 35]. This review sought to address this gap.

Considering the importance now placed on workplace based assessment, in particular MSF, there are surprisingly few published articles exploring the impact of MSF on physician behaviour and the factors that influence the effectiveness of MSF. Research has tended to focus more on the validity and reliability of different tools and on the acceptability of MSF to those using this assessment tool, rather than on its educational impact on improving professional performance and behaviours where judged to be necessary.

One of the main factors identified that could enhance MSF acceptance and use was whether feedback is facilitated, by an appraiser, mentor or coach. Furthermore, in five of the studies where feedback was not facilitated, physicians identified a need for facilitated feedback [20, 22, 29, 31, 32]. It was felt that facilitation would assist doctors in accepting the feedback and in using it to change their behaviour [31, 32].

It has been hypothesised that receiving negative or discrepant feedback (ratings from others that are lower than self-ratings) could be a catalyst for change, however evidence supporting this is limited [13]. In reality, it has been found that physicians who rated themselves higher than others are more likely to react negatively to the feedback and not change their behaviour [13] The influence of negative feedback, or feedback that was not consistent with their own self perceptions was also highlighted by six studies in this current review [21, 22, 26, 30, 31, 33]. Specifically, Overeem et al. (2012) identified that the quality of the mentoring and the receipt of negative scores by colleagues were the main motivators for change for participating physicians [27]. The regression analysis undertaken found that the two variables, quality of mentoring and scores by colleagues (physicians who received lower ratings from their medical colleagues reported more change in practice) accounted for 34% of the variance [27]. While the interaction between these two variables is unclear in this study, two studies highlighted that the role of the facilitator is key to the physicians’ acceptance of the feedback [31, 33]. The facilitator was said to encourage reflection and subsequent acceptance of the feedback which in turn led to a change in behaviour.

The findings of this review are similar to other reviews of MSF in non-healthcare settings. For example Smither et al. (2005) carried out a meta-analysis of 24 longitudinal studies of MSF in non-healthcare organisational settings, focussing on how managers use the MSF feedback over time [36]. The authors concluded that following participation in MSF, eight factors were identified as playing a part in influencing the extent of the change in behaviour and performance improvement [36]. These factors are:

Characteristics of the feedback, initial reactions to feedback, personality, feedback orientation, perceived need for change, beliefs about change, goal setting, and taking action [36].

As with this current review, if the recipients reacted negatively and rejected the feedback they were unlikely to use it to change their behaviour. However, unlike this current review, Smither (2005) did not look at the content or format of the feedback. Bracken and Rose (2011) carried out a review of the literature focussing on the factors of the MSF process that facilitates behaviour change, and like this current review, identified that the credibility of the rater was an important factor [37].

Strengths and limitations

Strengths of the study were that a recognised systematic review process was followed [38] and papers meeting the study criteria were appraised by two researchers. Data extraction using a pre-designed proforma was independently undertaken by both researchers ensuring that no relevant data were overlooked. Applying Buckley et al’s [18] quality criteria ensured that studies of very poor quality were eliminated. To evaluate outcomes, Barr’s (2000) adaptation of Kirkpatrick’s four level evaluation model [17] was applied to all study findings; this ensured that only the outcomes relevant to the study were included.

This study has some limitations. Firstly, while the literature search was extensive and covered a number of databases, the articles retrieved were restricted to English-language publications only, and therefore publication bias cannot be ruled out. Secondly, methodological quality varied extensively between the articles and most of the studies were conducted on small volunteer-based samples. Also, as this review only focussed on published empirical studies some relevant unpublished studies may have been missed. Lastly, two studies included in this review have been reported in multiple articles. The articles by Fidler et al. (1999) [21] and Hall et al. (1999) [22] report on different aspects of the same study and the five articles reported by Sargeant et al. (2003, 2005, 2007, 2008, 2009) [2933] all relate to the same sample population, from the pilot study reported by Sargeant et al. (2003) [29]. This could lead to over-reporting of the findings [39].

Whatever review method is employed, the resultant synthesis is only equivalent to the primary data included in the synthesis and the studies identified in this review had a number of limitations. Firstly, 15 of the 16 studies were descriptive and therefore the strength of the findings may be limited by the uncontrolled nature of the studies. However, it is asserted that, given the methodological difficulties of evaluating health professionals’ performance and the educational impact of interventions, descriptive and observational studies can still provide valuable information [7]. In fact, in this review the strongest evidence for improved performance following the MSF intervention and for the influencing factors for the effectiveness of the MSF intervention was provided by a focus group study, [27] and interview studies [31, 33]. However, it should be noted that all three articles are from the same sample population and therefore publication bias may exist.

Fifteen of the 16 studies described self-reported rather than measured changes in behaviour. While self-reported measures of change in behaviour are highlighted as being quicker and easier than measuring actual behaviours, it is argued that healthcare systems are not concerned with changing health professionals self-reported behaviour, rather they want to change actual behaviour in the belief that this should lead to improved patient care [40]. Both Eccles et al. (2006) [40] and Hrisos et al. (2009) [41] purport that while self-reported change in behaviour can be a useful proxy for actual behaviour the extent to which these measures accurately reflect actual behaviour is unclear.

Some of the studies reported intention to change rather than actual change in behaviour [20, 23, 26, 27]. A number of health psychology theories, such as the theory of planned behaviour [4244] and protection motivation theory [4547] assume that intentions (a person’s motivation or decision to engage in a course of action [47]) cause behaviour. However, as studies focussing on the intention-behaviour relationship have generally been correlational rather than causational this causality is still to be determined [48]. While Webb and Sheeran (2006) cite a number of meta-analyses that illustrate a relationship between intention and behaviour they highlight that this relationship could be in the reverse direction (that behaviour caused the intention) [48]. Therefore, it cannot be conclusively determined whether intention actually leads to behaviour change as a result of an MSF intervention.

Implications for practice

An MSF tool which is designed to incorporate feedback from ‘credible’ raters (i.e. people who know the physician’s practice well, including patients) and which includes narrative comments as well as numerical scores is desirable. Furthermore, it seems to be helpful if the feedback is facilitated as this can influence the physician’s response to the feedback, how they deal with any negative comments and can promote reflection, all of which can impact on how the physician accepts and uses the feedback. Therefore when implementing an MSF intervention, particularly on an organisational or national basis, these factors need to be considered in order to optimise the effectiveness of the process.

Implications for research

Considering the emphasis now placed on MSF as a method of formative performance assessment, there are surprisingly few published articles exploring these areas, and the strength of the study findings is questionable. While the studies identified in this review did examine both the behaviours influenced and the influencing factors, further research needs to be carried out to identify how the feedback influences behaviour, the strength of the influencing factors, and what impact behaviour change has on the quality and safety of patient care. In addition, only one study [15] looked at the change in behaviour over time, and in that study the follow-up was only five months. Therefore, it is not known whether the changes made are sustained over a longer period of time. Furthermore, while feedback is increasingly being used with health professionals other than primary and secondary care physicians, this review identified no eligible published studies involving these clinical groups which may be a cause for concern if MSF is also to be implemented as a performance measure by these professions.

The positive influence of a facilitator in the MSF process was highlighted in this review. However further research is clearly necessary to uncover the role and influence of the facilitator in the feedback process, particularly given the resource and logistical implications of providing such facilitation at scale as part of a national MSF system. Furthermore, research is also required to determine whether facilitation on its own is the key influence underpinning the impact of MSF, or in combination with the other factors identified as important (such as the source and format of the feedback).

Conclusions

This review has identified that participation in MSF by medical doctors can lead to improved performance, although the volume and quality of supporting evidence is limited. Several factors were found to influence the extent to which professional performance is improved, which are summarised below.

Key findings

Source of the feedback:

  • The raters need to be perceived as credible and familiar with the work of the physician [2931, 34]

  • Feedback from patients can increase acceptance of feedback and encourage positive change [29]

How the feedback is delivered:

  • Facilitated feedback is desirable [20, 22, 25, 26, 29, 31, 33]

  • Narrative comments from raters are valuable [26]

Content of the feedback:

  • Negative comments may stimulate behavioural change, except where comments are inconsistent with physicians’ own perceptions [21, 27, 32, 33]

  • Receiving a number of comments about the same behaviour encourages acceptance and change [28]

  • A gap between a peer rating and how the physician rates themselves encourages acceptance and change [22, 24]

  • Response to feedback – having time to reflect on the feedback was seen as an important facilitator for behavioural change [25, 31, 33]

References

  1. Gopee N: Self assessment and the concept of the lifelong learning nurse. Brit J Nurs. 2000, 9: 724-729.

    Article  Google Scholar 

  2. Epstein RM, Hundert EM: Defining and assessing professional competence. JAMA. 2002, 287: 226-235. 10.1001/jama.287.2.226.

    Article  Google Scholar 

  3. Epstein RM: Assessment in medical education. New Engl J Med. 2007, 356: 387-396. 10.1056/NEJMra054784.

    Article  Google Scholar 

  4. Hayes OW, Reisdorff EJ, Walker GL, Carlson DJ, Reinoehl B: Using standardized oral examinations to evaluate general competencies. Acad Emerg Med. 2002, 9: 1334-1337. 10.1111/j.1553-2712.2002.tb01597.x.

    Article  Google Scholar 

  5. Wilkinson TJ, Wade WB, Knock LD: A blueprint to assess professionalism: results of a systematic review. Acad Med. 2009, 84: 551-558. 10.1097/ACM.0b013e31819fbaa2.

    Article  Google Scholar 

  6. Norcini J, Burch V: Workplace-based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007, 29: 855-871. 10.1080/01421590701775453.

    Article  Google Scholar 

  7. Miller A, Archer J: Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ. 2007, 341: 710-715.

    Article  Google Scholar 

  8. Overeem K, Faber MJ, Arah OA, Elwyn G, Lombarts KM, Wollersheim HC, Grol RP: Doctor performance assessment in daily practise: does it help doctors or not? A systematic review. Med Educ. 2007, 41: 1039-1049. 10.1111/j.1365-2923.2007.02897.x.

    Article  Google Scholar 

  9. van der Vleuten CPM, Schuwirth LWT, Scheele MD, Driessen EW, Hodges B: The assessment of professional competence: building blocks for theory development. Best Pract Res Cl Ob. 2010, 24: 703-719.

    Article  Google Scholar 

  10. van der Vleuten CPM, Norman GR, De Graaf E: Pitfalls in the pursuit of objectivity: issues of reliability. Med Educ. 1991, 25: 110-118. 10.1111/j.1365-2923.1991.tb00036.x.

    Article  Google Scholar 

  11. General Medical Council: Supporting information for appraisal and revalidation. 2012, London: General Medical Council

    Google Scholar 

  12. Alexander DM: How do 360 degree performance reviews affect employee attitudes, effectiveness and performance?. University of Rhode Island, Schmidt Labor Research Center Seminar Research Series [online]. 2006, 1-12. [http://www.uri.edu/research/lrc/research/papers/Alexander_360.pdf]

    Google Scholar 

  13. Brett J, Atwater L: 360 degree feedback: Accuracy, reactions and perceptions of usefulness. J Appl Psychol. 2001, 86: 930-942.

    Article  Google Scholar 

  14. Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD: Audit and feedback: effects on professional practice and health care outcomes. Cochrane Db Syst Rev. 2006, 2: Art. No.: CD000259

    Google Scholar 

  15. Violato C, Lockyer JM, Fidler H: Changes in performance: a 5‒year longitudinal study of participants in a multi‒source feedback programme. Med Educ. 2008, 42: 1007-1013. 10.1111/j.1365-2923.2008.03127.x.

    Article  Google Scholar 

  16. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, Clarke M, Devereaux PJ, Kleijnen J, Moher D: The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. PLoS Med. 2009, 6: e1000100-10.1371/journal.pmed.1000100.

    Article  Google Scholar 

  17. Barr H, Freeth D, Hammick M, Koppel I, Reeves S: Evaluations of Interprofessional Education: A United Kingdom Review of Health and Social Care. 2000, London: CAIPE

    Google Scholar 

  18. Buckley S, Coleman J, Davison I, Khan KS, Zamora J, Malick S, Morely D, Pollard D, Ashcroft T, Popovic C, Sayers J: The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Med Teach. 2009, 31: 282-298. 10.1080/01421590902889897.

    Article  Google Scholar 

  19. Brinkman WB, Geraghty SR, Lanphear BP, Khoury JC, Gonzalez Del Rey JA, De Witt TG, Britto MT: Effect of multisource feedback on resident communication skills and professionalism: a randomized controlled trial. Arch Pediat Adol Med. 2007, 161: 44-49. 10.1001/archpedi.161.1.44.

    Article  Google Scholar 

  20. Burford B, Illing J, Kergon C, Morrow G, Livingston M: User perceptions of multi-source feedback tools for junior doctors. Med Educ. 2010, 44: 165-176. 10.1111/j.1365-2923.2009.03565.x.

    Article  Google Scholar 

  21. Fidler H, Lockyer JM, Toews J, Violato C: Changing physicians' practices: the effect of individual feedback. Acad Med. 1999, 74: 702-10.1097/00001888-199906000-00019.

    Article  Google Scholar 

  22. Hall W, Violato C, Lewkonia R, Lockyer J, Fidler H, Toews J, Jennett P, Donoff M, Moores D: Assessment of physician performance in Alberta: the physician achievement review. Can Med Assoc J. 1999, 161: 52-57.

    Google Scholar 

  23. Lipner RS, Blank LL, Leas BF, Fortna GS: The value of patient and peer ratings in recertification. Acad Med. 2002, 77: S64-10.1097/00001888-200210001-00021.

    Article  Google Scholar 

  24. Lockyer J, Violato C, Fidler H: Likelihood of change: a study assessing surgeon use of multisource feedback data. Teach Learn Med. 2003, 15: 168-174. 10.1207/S15328015TLM1503_04.

    Article  Google Scholar 

  25. Overeem K, Lombarts MJMH, Arah OA, Klazinga NS, Grol RP, Wollersheim HC: Three methods of multi-source feedback compared: a plea for narrative comments and coworkers' perspectives. Med Teach. 2010, 32: 141-147. 10.3109/01421590903144128.

    Article  Google Scholar 

  26. Overeem K, Wollersheim H, Driessen E, Lombarts K, Van De Ven G, Grol R, Arah O: Doctors’ perceptions of why 360-degree feedback does (not) work: a qualitative study. Med Educ. 2009, 43: 874-882. 10.1111/j.1365-2923.2009.03439.x.

    Article  Google Scholar 

  27. Overeem K, Wollersheimh HC, Arah OA, Cruijsberg JK, Grol RP, Lombarts KM: Factors predicting doctors’ reporting of performance change in response to multisource feedback. BMC Med Educ. 2012, 12: 52-10.1186/1472-6920-12-52.

    Article  Google Scholar 

  28. Owens J: Is multi-source feedback (MSF) seen as a useful educational tool in primary care? A qualitative study. Educ Prim Care. 2010, 21: 180.

    Article  Google Scholar 

  29. Sargeant JM, Mann KV, Ferrier SN, Langille DB, Muirhead PD, Hayes VM, Sinclair DE: Responses of rural family physicians and their colleague and coworker raters to a multi-source feedback process: a pilot study. Acad Med. 2003, 78: S42-10.1097/00001888-200310001-00014.

    Article  Google Scholar 

  30. Sargeant J, Mann K, Ferrier S: Exploring family physicians' reactions to multisource feedback: perceptions of credibility and usefulness. Med Educ. 2005, 39: 497-504. 10.1111/j.1365-2929.2005.02124.x.

    Article  Google Scholar 

  31. Sargeant J, Mann K, Sinclair D, Van der Vleuten C, Metsemakers J: Challenges in multisource feedback: intended and unintended outcomes. Med Educ. 2007, 41: 583-591. 10.1111/j.1365-2923.2007.02769.x.

    Article  Google Scholar 

  32. Sargeant J, Mann K, Sinclair D, Van der Vleuten C, Metsemakers J: Understanding the influence of emotions and reflection upon multi-source feedback acceptance and use. Adv Health Sci Educ Theory Pract. 2008, 13: 275-288. 10.1007/s10459-006-9039-x.

    Article  Google Scholar 

  33. Sargeant JM, Mann KV, van der Vleuten CP, Metsemakers JF: Reflection: a link between receiving and using assessment feedback. Adv Health Sci Educ Theory Pract. 2009, 14: 399-410. 10.1007/s10459-008-9124-4.

    Article  Google Scholar 

  34. Shepherd A, Lough M: What is a good general practitioner (GP)? The development and evaluation of a multi-source feedback instrument for GP appraisal. Educ Prim Care. 2010, 21: 149-164.

    Article  Google Scholar 

  35. Veloski J: Systematic revew of the literature on assessment, feedback and physicians’ clinical performance: BEME Guide No. 7. Med Teach. 2006, 28: 117-128. 10.1080/01421590600622665.

    Article  Google Scholar 

  36. Smither JW, London M, Reilly RR: Does performance improve following multisource feedback? A theoretical model, meta-analysis and review of empirical findings. Pers Psychol. 2005, 58: 33-66. 10.1111/j.1744-6570.2005.514_1.x.

    Article  Google Scholar 

  37. Bracken DW, Rose DS: When does 360-degree feedback create behavior change? And how would we know it when it does?. J Bus Psychol. 2011, 26: 183-192. 10.1007/s10869-011-9218-5.

    Article  Google Scholar 

  38. Centre for Reviews and Dissemination: Systematic Reviews CRD’s Guidance for Undertaking Reviews in Health Care. 2008, Heslington: CRD, University of York

    Google Scholar 

  39. Von Elm E, Poglia G, Walder B, Tramer MR: Different patterns of duplicate publication. JAMA. 2004, 291: 974-980. 10.1001/jama.291.8.974.

    Article  Google Scholar 

  40. Eccles MP, Hrisos S, Francis J, Kaner EF, Dickinson HO, Beyer F, Johnston M: Do self-reported intentions predict clinicians' behaviour: a systematic review. Implement Science. 2004, 1: 28.

    Article  Google Scholar 

  41. Hrisos S, Eccles MP, Francis JJ, Dickinson HO, Kaner EF, Beyer F, Johnston M: Are there valid proxy measures of clinical behaviour? A systematic review. Implement Science. 2009, 4: 37-10.1186/1748-5908-4-37.

    Article  Google Scholar 

  42. Ajzen I: From intentions to actions: A theory of planned behavior. Action-control: From cognition to behavior. Edited by: Kuhl J, Beckmann J. 1985, Heidelberg: Springer, 11-39.

    Chapter  Google Scholar 

  43. Ajzen I, Madden TJ: Prediction of goal-directed behavior: attitudes, intentions, and perceived behavioral control. J Exp Soc Psychol. 1986, 22: 453-474. 10.1016/0022-1031(86)90045-4.

    Article  Google Scholar 

  44. Ajzen I: Attitudes, personality and behavior. 1988, Chicago: Dorsey Press

    Google Scholar 

  45. Rogers RW: A protection motivation theory of fear appeals and attitude change. J Psychol. 1975, 91: 93-114. 10.1080/00223980.1975.9915803.

    Article  Google Scholar 

  46. Rogers RW: Cognitive and physiological processes in fear appeals and attitude change: A revised theory of protection motivation. Social psychophysiology: A source book. Edited by: Cacioppo BL BL, Petty LL. 1983, London: Guildford Press, 153-176.

    Google Scholar 

  47. Rogers RW: Attitude change and information integration in fear appeals. Psychol Rep. 1985, 56: 179-182. 10.2466/pr0.1985.56.1.179.

    Article  Google Scholar 

  48. Webb TL, Sheeran P: Does changing behavioral intentions engender behavior change? A meta-analysis of the experimental evidence. Psychol Bull. 2006, 132: 249-268.

    Article  Google Scholar 

Pre-publication history

Download references

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to Julie Ferguson.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors’ contributions

JF conceived of and designed the study, carried out the database searches, study selection, data extraction and interpretation and drafted the manuscript. JW acted as a second reviewer, participating in the study selection and data extraction and in the drafting and revising of the manuscript. PB participated in the drafting of and revising of the manuscript. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly credited. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Cite this article

Ferguson, J., Wakeling, J. & Bowie, P. Factors influencing the effectiveness of multisource feedback in improving the professional practice of medical doctors: a systematic review. BMC Med Educ 14, 76 (2014). https://doi.org/10.1186/1472-6920-14-76

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/1472-6920-14-76

Keywords