International experiences show that implementing the superordinate role concept in UME is not an automatic process [2, 4, 10] but needs to be closely monitored with suitable tools. There are several ways to evaluate the implementation of intrinsic roles at certain stages, mostly relying on (focus group) interviews, surveys or observation of practice of the different target groups involved. The strengths of these common approaches can be especially seen in including individual viewpoints, identification of practical needs and fundamental freedom of positioning. Mapping approaches provide another resource that can be combined with existing methods or form their basis. Mapping approaches, especially in web-based databases, enable e.g. comprehensive curricular description and visualisation, a common reference, different perspectives and scalability in focus and permanent availability.
In a previous, preliminary mapping study, multi-site curricular weightings of role’s objectives and programs’ agreement were compared in a matrix map. By applying Roger’s theory of diffusions of innovations [23], the role-specific patterns gave an orienting overview to identify the roles in various stages of curricular development [20]. In a next step, the present multi-site study provides more detailed diagnostic data focussing on the curricular status quo of the (sub-)competencies and objectives of the Collaborator, Health Advocate and Leader/Manager. These roles are all highly relevant for safe patient management and optimization of the healthcare system in rehabilitation and prevention. In overview, the Health Advocate can be highlighted as a positive example of how sub-competencies are consistently well-integrated in curricula, though in wide ranges of generally high curricular weightings. In contrast, the Collaborator role indicates average curricular representation, but reveals signs of ongoing curricular development in relevant parts, as well as obvious weaknesses regarding assessment and achieved outcome. Finally, the Leader/Manager displays the consistently lowest curricular weightings of its objectives with several substantial deficiencies in curricular representation, constructive alignment and/or outcome level [24].
The benchmarking approach with a common mapping database and consented procedures [22] applied in this study, appears an appropriate strategy to support monitoring of CBME implementation [21]. Mapping data can be applied in that way at any time during curricular developmental processes. The current data set documents a crosscut snapshot to indicate the program’s current positioning in relation to others in the field. In the context of UME, this approach allows to gain (external) reference data, to identify potential for optimization and to realize best practices. In any case, the data are considered as non-normative but descriptive in benchmarking process. However, the anonymity of the program is ensured since individual data is accessible only by the respective faculties.
Implications of role profiles
Mapping current UME curricula against consented standards (here: the German CBME framework NKLM [19]) reveals detailed information on conformities and discrepancies between curricular reality and given standards in teaching, assessment and competency level. On the one hand, these diagnostic data can help curriculum developers to identify curricular challenges in their local program. Based on that information they can decide whether and how to deal with these problems and set priorities. Whilst, on the other hand, multi-site practice-based evidences support framework reviews by critically reflecting its content and currently valid standards for perspective adjustment. There are typical constellations of diagnostic findings down to detailed objective level, affecting both or one of the target groups. In the following, frequently occurring challenges are exemplified in increasing degree of difficulty; it is discussed how they may be interpreted and dealt with, from different perspectives.
Role parts exceeding given standards
The Health Advocate offers characteristic examples for this data constellation. At a first glance it appears rather unproblematic: In line with international demands [1, 14,15,16], this role is an essential part of the UME curricula and well-integrated in many programs, though varying in frequency and intensity. It is assessed in all programs. A closer inspection shows that many sites clearly surpass the given minimal competency level in most sub-competencies, except in SC-9.1.3 and SC-9.2.3 addressing interprofessional health promotion in population groups and systems. Here, few programs fall short of the desired competence level and thus give the responsible local curriculum developers cause to act. In the overall evaluation, the Health Advocate shows itself as a positive role that does not currently require immediate urgent attention except in some local curricula.
Non-achievement of competency level
Despite the wide range of curricular weightings, heterogeneous attainments of competency levels were mapped, in some UME programs below the minimum requirements (Table 2). Typical examples are some sub-competencies of the Collaborator role, esp. SC-8.2.1 to SC-8.2.3, encompassing interpersonal skills for interdisciplinary and interprofessional collaboration. These topics are evidenced as a key aspect of successful inter-professional teams being closely related to patient safety [1, 13]. In case of sub-standard representation, sub-competencies appear to be taught rather in theory (Level 2: applied knowledge and skills in training) than in practice as specified in the NKLM (Level 3a: competency in practice, supervised). Presumably this is because of missing learning opportunities, inadequate context or impeding cultural environment [25]. After review of the NKLM framework and its re-acceptance, German curriculum developers are most likely challenged to revise and intensify UME interventions ensuring that graduates are appropriately prepared for mastering collaborative practice on day 1 of residency [10].
Low curricular weighting but (potentially) underrepresented
Typical examples of this characteristic feature are (sub-)competencies and objectives of the Leader/Manager. They are mapped in only few or none courses but well-consented in this lowest amount of curricular representation: e.g. SC-10.10.1 and SC-10.10.2 focusing leadership personality and styles as well as management functions. Internationally, collaborative leadership skills are increasingly recognized as indispensable for every physician - a fact that has already been considered in the drafting (and revision) of various national frameworks [1, 18]. Because of the well-known leadership impact on patient care and safety, these objectives are recommended to be integrated stably into UME programs for advanced medical students in their clinical years. Thus, basic competency levels achieved should be further developed in the practical year and residency [17, 26]. However, inappropriate educational context and environment as well as lacking students’ involvement could result in too rare opportunities to: (1) see role modelling (e.g. cooperative leadership), (2) reflect on the realisation and (3) practice it themselves [10]. The given representation may indicate deep-rooted curricular patterns. In view of the low curricular representation in German UME programs and the development of societal needs, both curriculum developers and framework reviewers are recommended to rethink competency levels and weightings as well as to place greater emphasis on underrepresented qualities.
Taught but not assessed content
Several examples illustrate missing formal constructive alignment in a considerable number of programs (e.g. in the Collaborator SC-8.2.3 addressing interprofessional conflicts; or in the Leader/Manager: again SC-10.10.1 and SC-10.10.2). There is an ongoing debate regarding the testability as well as the necessity of obligatory assessment of every competency facet, especially considering the rare opportunities to explicitly experience and practice certain role aspects in clinical context [27,28,29]. Faculties often perceive classical assessment methods as suboptimal for non-medical content. At the same time, unawareness and lack of familiarity with alternatives like qualitative methods lead to adherence to traditional habits. However, multifaceted assessment facilitating developmental progression of competence is crucial in CBME. More information and training are essential to “create a shared mental model of required learner’s behaviour and expected level of performance” [28] and build up “continuous, comprehensive and elaborate assessment and feedback systems” [4, 27]. Besides highlighting the role of feedback, research efforts are to be intensified for further development of additional formative assessment instruments and formats [25, 29, 30]. Thus, particularly faculty and curriculum development as well as quality management measures are required to facilitate institutional and programmatic change regarding Collaborator and Leader/Manager.
General accumulation of curricular weaknesses
Some framework content is characterized by clear deficiencies in the majority of programs regarding the criteria included: none or very low curricular weightings, sub-standard level attainment and missing assessment (e.g. non-specific sub-competencies from Leader/Manager field addressing time management, career planning and personal qualification needs; or Collaborator role features focusing on advanced aspects of interprofessional work). This may be caused by lack of conceptual clarity in terms of definitions, role characteristics, personal-specific and context-specific features, as systematically reviewed for the Leader/Manager [31]. Notwithstanding, some roles (esp. Collaborator, Leader/Manager) appear to be less affected by external pressure like e.g. legal regulation or politics than others (e.g. Health Advocate) during the last decades. Instead, moderate signs of increasing curricular emphasis like e.g. in the Collaborator role, seem to be based rather on internal efforts of individual programs. More than in any other case mentioned above, reviewers of the framework may seek clarification on whether, and if so to what extent, a (sub-)competency should be integrated into UME (e.g. personal-specific features of planning, system-related interprofessional collaboration). If re-affirmed, every effort for institutional change must be strengthened. If identified as inappropriate, it should be removed from UME and potentially transferred to PME.
Limitations
Some restrictions of our approach have to be considered. Over−/underestimations of competency representation cannot be excluded despite the instruction of mappers and data quality control. Mapping data may be positively or negatively biased by certain factors: e.g. knowledge, framework terminology, perception of intrinsic roles, CBME acceptance, institutional culture and enthusiasm for teaching. Therefore, mapping data cannot be taken as hard and accurate values for curriculum depiction. It is rather an actual snapshot and crosscut of curricula in the continuously changing field of UME, which must be regularly updated to show its merit. The teachers view is an important but single-sided perspective on the explicit curriculum (taught curriculum), even though results were controlled in probability checks by Dean’s offices and senior teachers. However, the students view, addressed in another project, is relevant for a multi-perspective curriculum evaluation (learned curriculum). Regarding the graphical representation of data, the sample size is considered to be rather small for visualization in boxplots and following definite interpretation about underlying structures. Nevertheless, the display in boxplots instead of dot columns provides more clarity and enough orientation to catch an informative tendency of roles development on objective level at a glance.