Measuring informal workplace learning outcomes in residency training: a validation study

Abstract

Background

Informal workplace learning (WPL) has no concrete learning objective and takes place without a responsible supervisor, which makes it difficult to assess its learning outcomes. Formal learning situations, as they are known from universities or schools, do not exist in this context and make a conventional assessment of learning goals and achievements impossible. Informal learning in the workplace is of central importance, and the assessment of informal learning outcomes in medical education is an under-researched area. The aim of our study was to adapt and validate an informal WPL questionnaire (originally developed for social workers) to assess learning outcomes due to informal WPL in residency training.

Methods

A total of 528 residents (n = 339 female; age: M = 29.79; SD = 3.37 years) completed an adapted questionnaire on informal WPL outcomes and the Freiburg Questionnaire to Assess Competencies in Medicine (i.e. medical knowledge, communication, and scholarship). Exploratory factor analysis was used to determine the underlying factor structure. The reliability of the factors was tested using McDonald’s omega, and the correlation between the factors and the three subscales of the Freiburg questionnaire was tested using Spearman’s rho correlation coefficient. To investigate construct validity, a structural equation model was calculated to examine the relationships between medical competencies and informal learning outcomes.

Results

The exploratory factor analysis yielded a four-factor solution that best fit the data. The scores of all four factors (GLO-CD: generic learning outcomes—competence development, GLO-R: generic learning outcomes—reflection, JSLO: job-specific learning outcomes, and OLLO: organisational learning outcomes) showed good internal consistency (Ω ≥ .69). The structural equation model showed that "medical expertise" had an impact on all four factors of informal learning at work. “Scholarship” seemed to predict GLO-CD and GLO-R.

Conclusions

Our four-factor model reveals meaningful determinants of informal WPL in relation to residency training. The instrument is therefore the first promising attempt to assess informal WPL in the broader context of medical education during residency, thus supporting its construct validity.

Background

After graduation from medical school, physicians begin their residency training. Residency training takes place almost entirely in the workplace, where, in contrast to medical school, formal learning is largely replaced by informal workplace learning (WPL; [1, 2]). WPL occurs when physicians apply skills in the workplace they have learned in medical school to clinical practice, or when they acquire new skills they were not previously familiar with [1, 3]. The outcomes of informal WPL seem to be different from formal learning outcomes, such as academic knowledge or practical skills. In one approach to defining informal WPL outcomes, generic learning outcomes, organisational learning outcomes, and job-specific learning outcomes are distinguished [1]. Specifically, Kyndt et al. [1] developed and validated a questionnaire to determine informal WPL outcomes for social care workers and capture its different aspects. However, little is known about whether these informal WPL outcomes also apply to residency training. To address this research gap, we have adapted the measure proposed by Kyndt et al. [1] for implementation with young physicians in their residencies and investigated how physicians’ perceived competencies, which include “communication”, “scholarship”, and “medical expertise”, as they have learned in their undergraduate medical education, predict informal WPL outcomes in their residencies. In the following, we provide a short overview of the theoretical concept of WPL, the Questionnaire of Informal WPL Outcomes used to measure these outcomes, and the influence of prior knowledge on informal WPL outcomes.

Learning in the workplace can be described by the social constructive learning theory. Social constructivism assumes that learners learn due to experiences and apply their prior knowledge to particular (working) tasks [4]. Workplace activities foster learning through different learning opportunities, such as allowing individuals to do the work themselves, obtain feedback and collaborate with colleagues and supervisors, deal with problems and challenges, and reflect on their own behaviour at work [5]. Previous findings show that the interplay of these learning conditions in the workplace (also provided by supervisors and the organisation) can facilitate knowledge and competence acquisition among employees [5,6,7].

Knowledge and competence acquistion can be facilitated through diffferent WPL approaches. These approaches are differentiated into formal, informal, and non-formal WPL. Formal WPL refers to clearly defined learning conditions, including learning goals, learning “off the job” in classrooms or workshops, and having a supervisor who teaches a particular skill or topic to the resident in a planned and deliberate way [5, 8]. On the contrary, informal WPL is embedded in everyday work as it takes place, without any structure, particular learning goal, or responsible supervisor [3, 5, 7]. Nonetheless, informal WPL is crucial for becoming an expert in one’s respective medical field, since it emerges from the process of participation in activities that characterise a specific community of practice and a professional domain [9]. Previous research has assumed that almost 80% of WPL opportunities are informal [1, 2]. Finally, the concept of non-formal learning merges aspects of formal and informal WPL. Non-formal learning also occurs in the workplace during daily practice. However, even though it involves a supervisor or mentor and explicit learning goals, it is still not as structured and planned as formal learning [3]. A typical non-formal learning setting would be workplace training organised by the company (e.g. training on a new resuscitation device conducted by other colleagues or supervisors who have already been trained).

Approaches to defining informal WPL outcomes focus either on competence-related differentiation by defining knowledge, skills, and attitudes or on a holistic approach offering learning principles for different occupational levels (for a detailed summary, see [1]. Kyndt et al. [1] have defined informal WPL outcomes as “[…] sustainable changes in knowledge, skills or attitudes that result from engagement in informal and formal learning processes and that influence individuals’ present and future professional achievement and/or organizational performance” [1]. By combining the competence-related and the holistic approach, the questionnaire of informal workplace learning developed by Kyndt et al. [1] aims to investigate informal WPL outcomes of employees and to identify potential for improvement in the working environment. The generic learning outcomes (GLO) refer to outcomes for all employees of a company (i.e. the product of reflection on the daily work routine, independent of one’s own position in the team or organisation and particular profession). The organisation-level learning outcomes (OLLO) are related to subgroups or teams within an enterprise in learning how to facilitate a teamworking environment. Finally, the job-specific learning outcomes (JSLO) describes the learning process required of a particular position (i.e. to develop the competencies needed to be successful in a given profession, like surgery or radiology). This classification, as described by Kyndt et al. [1], seems to be applicable to residency training as well. In each clinical area during residency, residents are supervised by a variety of consultants with differing clinical, scientific, and didactic expertise and interests, these characteristics of supervisors properly refers to GLO and OLLO. Care has to be provided in individual medical contexts and for conditions that require residents’ medical action, which might account for JSLO. With time during residency, along with the increasing competence of the resident, the complexity of treated cases increases, as does independence from supervisors. In this process, residents become an essential part of the medical team and health care system. Additionally, desired learning outcomes are necessarily specified as learning objectives a priori. Furthermore, while employers define the conditions under which WPL occurs [5], the employees’ individual biographies, previous knowledge, and/or personal engagement in other social scopes foster their individual WPL [10]. The outcomes of informal WPL could be influenced by the prior knowledge and competencies of residents [6, 10]. Learning in general refers not only to the emergence of new knowledge, but also to the combination of new knowledge with prior knowledge [9]. Self-awareness of one’s own prior knowledge or competencies might play a crucial role in (furthering) the development of competencies due to (informal) WPL during residency. Such self-awareness of competencies and knowledge can support self-efficacy, which has been mentioned as a further resource required for developing the intention to learn [11]. Self-efficacy refers to individual confidence in successfully handling challenging situations based on one’s own competencies [12, 13]. Among these factors, each resident’s motivation to gain a deeper understanding of a clinical topic, even outside of work, is important. We therefore assume that (informal) WPL plays a substantial role in the success of one’s residency [7, 14, 15].

Since the learning processes in residency training are less structured, but represent an innumerable variety of learning content and learning opportunities, which in turn can have a particular impact on patient care, we consider it urgently necessary to investigate the informal learning outcomes in residency training as well. Thus far, empirical research on informal WPL in residency training in general and on its outcomes in particular is scarce [16,17,18,19,20]. The present study addresses this gap by adapting the Questionnaire of Informal Workplace Learning Outcomes developed by Kyndt et al. [1] for use in residency training. Thus, we aim to answer the following research questions (RQ):

• RQ1: Does the German version of the instrument support a three-factorial model like the original instrument (cf. Kyndt et al. [1])?

• RQ2: Does the German version of the instrument provide reliable data (α ≥ .7)?

• RQ3: Is there a (predictive) relationship between medical expertise, communicative competence, and scholarship and informal WPL outcomes (r ≥ .3, p ≤ .05)?

Method

Residency in Germany

In Germany, residency training is independent from university-based undergraduate medical education. The medical associations (Ärztekammern) of the federal states are responsible for this, as they determine the learning content of specialist training. This includes mandatory sections in certain clinical areas (e.g. four years of anaesthesiology and one year of intensive care for an anaesthesiologist), as well as a correspondingly documented and performed minimum number of certain procedures (e.g. 50 central venous catheters). While the residents rotate through the different areas of their clinic to become familiar with the entire spectrum of their specialty, as well as to learn the necessary skills, the medical association decides at the end in a “collegial examination interview” whether to grant the specialist certification.

Currently, while competency-based curricula have been well integrated into undergraduate medical education, they are not generally applied to postgraduate specialist training. Furthermore, validated tools to assess competency do not yet exist or have not been accepted by German licensing institutions [21,22,23]. Although there is more competition for employment in the United States than in Germany, calls for changes in medical training have risen in recent years [24]. However, the structure of residency still differs substantially from that of American hospitals [25, 26]. In contrast, in countries like Great Britain and Ireland, for instance, competency-based assessments have been introduced to guarantee structured surgical training. These include specific teaching clinics, necessities for the ongoing education of the educators, and regular training days for trainees [27,28,29]. In Ireland, surgery residents need to complete lab-based operative skills assessments to be entrusted with professional activities [27, 29]. As mentioned above, German residency programmes may define the required content and length, but they do not emphasise the skills and competencies residents must possess at the end. Accordingly, the end of residency is defined by acquired time and knowledge, rather than skills and competencies.

Sample

Data collection occurred in the context of a survey study conducted among Bavarian medical graduates (Bayerische Absolventenstudie Medizin, MediBAS) during October 2018 and January 2019. Invitations to participate were distributed among 1,610 physicians who graduated between October 1, 2016 and September 30, 2017 at one of five Bavarian medical schools (Friedrich-Alexander Universität Erlangen, Ludwig-Maximilians-Universität München, Technische Universität München, Universität Regensburg, and Julius-Maximilians-Universität Würzburg). The survey was distributed via mail (paper-based) or email (Questback, Globalpark Inc.; for more details, see [30]. A total of 528 participants (n = 339 female; age: M = 29.79, SD = 3.37 years) completed the survey and were included in the present study. Of these participants, 88.8% were still employed in their first job after graduation, 9.3% had already started another job, and only three physicians were not employed at the time of the survey. The average number of working hours per week was 52.37 (SD = 9.42, min = 8.50, max = 85.00). A total of 58.3% worked in institutions with more than 500 employees, 25.2% worked in hospitals with 50–499 employees, 11.4% worked in companies with 2–49 employees, and 5.1% did not provide any information. The distribution of medical disciplines can be found in Table 1, and the occupation type can be obtained from Table 2. At the time of the survey, the participants had been in graduate medical training for an average of 13.5 months (SD = 7.2). Participants who had not (yet) received graduate medical training were excluded from these analyses, as we could rule out WPL in residency training for these participants. The participants were informed about the content and purpose of the study and gave their informed consent to participate in the survey in advance. The data collection was anonymous.

Table 1 Distribution of medical disciplines among participants

Table 2 Occupation type of participants

Validation approach

Our validation approach is based on the construct validity concept, which postulates that the trustworthiness of a score can be determined by revealing the relationship between a (theoretical) construct and the measure under consideration through construct validity [31]. The first step in investigating construct validity was to examine the psychometric properties of the German translation of the original questionnaire. We started by examining the dimensionality (factor structure) of the scale using explorative and confirmatory factor analyses. This approach aimed to answer RQ1. We then examined the reliability of the instrument to answer RQ2.

Finally, we compared the results of the factor analysis with the three subscales of “medical expertise”, “communication”, and “scholarship” on the Freiburg Questionnaire to Assess Competencies in Medicine (FKM; [32]), for construct validity to answer RQ3. To investigate the construct validity of the WPL measure, we assume that a higher degree of competencies would be related to higher informal WPL outcomes.

Instrument

The Questionnaire of Informal Workplace Learning Outcomes was developed and first validated for social care workers in Belgium [1]. A second Belgian study investigated informal WPL outcomes by adapting the instrument for policy inspectors [6]. Both studies applied the instrument in the Flemish language. The validation of the original instrument [1] consisted of three factors (GLO, OLLO, and JSLO) that were replicated in the study with policy inspectors [6]. The model fit in both studies was in an acceptable range ( [1]: CFI = 0.95; SRMR = 0.04, RMSEA = 0.05; [6]: CFI = 0.92; SRMR = 0.072, RMSEA = 0.078). Internal consistency was also satisfactory in both studies ( [1, 6]: α ≥ 0.73).

As the work environments of socio-educational care workers and healthcare providers are different, the original Questionnaire of Informal Workplace Learning Outcomes by Kyndt et al. [1] was adapted to the work environment of healthcare (see item wording in Table 3). Then, the questionnaire was translated into German. In translating the instrument, we followed the recommendations of Wild et al. [33]. The translation of the instrument was done by two educational researchers with English language skills at the C1–C2 level. A back-translation was not performed due to time constraints (fixed start of study from the administration committee of data collection). The translated version was discussed by the research group in order to ensure the substantive accuracy of the translation and its suitability to residency training. Based on the suggestions of Kyndt et al. [1], we adjusted the scale of the JSLO to our professional field of medicine. We eliminated item No. 10 on the original JSLO, “to support clients in their social participation”, as the item describes a (medical) activity that applies to only a small proportion of the medical profession (e.g. psychiatry or psychosomatics). As a result, we reduced the number of items from five to four. In addition, JSLO items No. 3 and No. 4 were adjusted to the medical context. The items were appraised on a 5-point Likert scale (1 = “disagree”, to 5 = “agree”).

Table 3 Overview of factor structure of measure by Kyndt et al., Janssens et al., and the present study

Further data included age, gender, and the FKM score with the three subscales of “medical expertise”, “communication”, and “scholarship”. The scale measuring medical expertise referred to the knowledge and skills needed to conduct basic diagnostics and to develop treatment plans [32]. The scale for communication covers different communicative situations with patients and how to build trustworthy relationships with them [32]. Finally, the scale for scholarship covers the scientific competencies of reading, interpreting, and applying medical research findings in one’s daily work. The competencies described by the FKM are estimated on a 5-point Likert scale (1 = “not at all”, to 5 = “very much”; [32]).

Analyses

Missing data were imputed using the Random Forest imputation method [34] in R by applying the R-packages “missForest” [35] and “randomForest” [36]. We started the instrument validation calculations with a CFA to answer RQ1. The aim of this CFA was to investigate the fit of the factor structure of the original scale developed by Kyndt et al. [1] for our data. Due to the poor data fit to the original model structure, we tested the common variance of the intercorrelation matrix using the Kaiser–Meyer–Olkin measure and Bartlett’s test of sphericity. The Kaiser–Meyer–Olkin measure of sampling adequacy provided a value of 0.91, and we found a significant result for Bartlett’s test (p < 0.001), showing that our data were suitable for exploratory factor analysis (EFA; [37]). We conducted an EFA with a robust maximum likelihood estimator to estimate the appropriate number of factors. Only variables with a loading ≥ 0.4 were considered in the respective factor [6, 37]. We then considered a four-factor solution for further analysis. The item OLLO No. 11, “…to fulfil managerial tasks autonomously”, did not fit any of the factors and was therefore excluded from further analyses.

We compared the two models with each other and found the most suitable structural equation model (SEM) using ANOVA. First, we compared the Akaike information criterion (AIC) and the Bayesian information criterion (BIC) of the two models. The smaller these values were, the better the model fit the data. Second, we interpreted the Chi² differences and their significance. The Chi² differences of the two models were significant (p < 0.001). Based on these comparisons, Model 1 performed better (Table 4). Then, we calculated the measurement models for each factor of Model 1 (cf. Table 5). The model fit values of each measurement model were found to be satisfactory [38]. For further analysis, we examined the explained variance for each factor and for the overall scale. The reliability of the factors was tested using McDonald’s omega and Cronbach’s alpha to answer RQ2. We assumed omega and alpha ≥ 0.7 to be adequate for a reliable test instrument [39].

Table 4 Comparison of fit indices of the two structural equation models

Table 5 Fit indices of measurement models of the four-factor solution

To answer RQ3, we then investigated the relationship between the WPL factors and the three FKM scales of “medical expertise”; “communication”, and “scholarship” using Spearman’s rho correlation coefficient due to a missing normal distribution of the data. According to the work of Field [40], we assumed a correlation coefficient of r ± 0.1 as a small effect, r ± 0.3 as a medium effect, and r ± 0.5 as a large effect. The significance level was set at p ≤ 0.05. The analyses were conducted with the R package lavaan (version 3.6.1; [41, 42]) and IBM SPSS statistical software (version 28; [43]).

To further investigate the construct validity of the instrument, we calculated an SEM by applying the three FKM scales as predictors of the four WPL factors. In the first step, a measurement model was calculated that covered the four latent variables of informal WPL outcomes. In the second step, a regression model was conducted to add the three FKM scales as predictors to the measurement model [39]. The SEM was calculated by using the software Mplus 8.7 [44] by applying maximum likelihood. The cut-offs of the fit indices for SEM applied by Kline [39] were used here (CFI ≥ 0.90, TFI ≥ 0.95, SRMR ≤ 0.05; RSMEA ≤ 0.08).

Results

Factor analysis

To answer the first research question regarding whether the German version of the instrument supported the same factor structure as the original instrument, we conducted a CFA. All items were modelled to load onto the respective factors in accordance with the original scale [1]. In contrast to the original scale, the model fit indices were poor (cf. Table 4, Model 2). We did not find a factor structure similar to that of the original scale as postulated in RQ1. After the CFA, we investigated the factor structure using EFA. The EFA revealed a four-factor solution as the best fit for our data (Model 2).

Reliability

The second research question focused on the reliability of the factors within the measure, which was calculated using McDonald’s omega. The values of all four factors were in an acceptable range, thereby providing evidence of good internal consistency (α ≥ 0.69, Ω ≥ 0.69; cf. Table 5).

Correlation and regression analyses

The test of normal distribution of the factor means and the means of communication competence, scholarship, and medical expertise were significant. Therefore, we could not assume normally distributed data. To investigate the construct validity of the scale, we calculated an SEM with the four factors of the informal WPL outcome questionnaire and the three scales of the FKM, including “communication”, “scholarship” and “medical expertise” (cf. Table 6). Spearman’s rho correlations between the factor means and the three competency means of “communication”, “scholarship”, and “medical expertise” were of medium size (between r = 0.199 and r = 0.395). Therefore, we could assume that the four factors of the WPL questionnaire were related to the three competencies of communication, medical expertise, and scholarship, indicating comparable but different underlying constructs. In addition, we investigated whether the three perceived competencies predicted informal WPL outcomes by applying an SEM. The SEM indicated that medical expertise in particular had an impact on all four factors of the informal WPL outcomes measure. In contrast, scholarship seemed to predict GLO-CD and GLO-R, and communication had no predictive effect on any of the four factors (Table 7).

Table 6 Correlation matrix between the four factor means and communication, scholarship, and medical expertise of the FKM

Table 7 Regression of the four WPL factors on communication, scholarship, and medical expertise

Discussion

Our study examined the psychometric properties of the adapted German version of the Questionnaire of Informal Workplace Learning Outcomes [1] for medical residency. Furthermore, we examined how the questionnaire was related to the residents’ perceived competencies of medical expertise, scholarship, and communication [32]. In the following sections, we discuss the answers to our research questions and the limitations and practical implications of our study.

The first research question investigated whether the German version of the questionnaire, when adapted for medical residency, supported the three-factorial model of the original instrument by applying a CFA (cf. [1]). In this case, the CFA provided insufficient model-fit values. One explanation could be that the reference frame with which the doctors completed the questionnaire was different from that used by social care workers in the original questionnaire [1] or by policy inspectors in an additional adaptation of the instrument [6]. Thus, the items may have been scored differently than in the original study. This, in turn, explains the different scale formation, since the outcomes of a consecutive EFA suggested a four-factor model with acceptable fit values. In comparison to the original three-factorial model (see Table 3), our four-factorial model also included the JSLO and OLLO factors. The GLO-CD and GLO-R depicted different aspects of generic learning outcomes. The GLO-CD included items about competence development, and the GLO-R emphasised items that describe reflective processes of (generic) learning. The distinction between job-specific and GLO was an adaptation of the original classification of items by Kyndt et al. [1]. Below, we briefly discuss each factor.

The GLO-CD, referring to generic learning outcomes—competence development, included items that described generic learning outcomes with a focus on competence development. We considered this to be an important dimension, as it reflects one’s cognitive independence and freedom, for which some critical thinking and some cognitive spare capacity might be prerequisites. Furthermore, this perspective on competence development could refer to the aspect of becoming a physician during residency. Following the idea of professional identity formation, gathering experiences should facilitate identity formation and allow for a transition from imitating the physician’s role to being a physician [45]. This factor extends to some extent to the professional identity formation of becoming an independent and autonomous thinking and acting physician who is able to break the boundaries of their profession in an appropriate manner [46].

The GLO-R, or generic learning outcomes—reflection, consisted of items that described reflection processes in the occupational environment. Reflection is a crucial part of learning in general and of WPL in particular. Through reflection, trainees can learn from their experiences and transfer their learning to comparable future situations [47]. The aspect of reflection is also described in the Canadian Royal College of Physicians and Surgeons’ physician competency framework (CanMEDS) with respect to the role of the “professional”, where the physician’s commitment to self includes self-monitoring to constantly improve their behaviour; this is relevant to both how they treat patients and how they manage their own resources [48]. In the field of expertise development, the concept of deliberate practices also refers to reflection as a crucial skill for improving one’s competencies and performance [49]. These different excerpts are only a brief indication of the manifold relevance of reflection in undergraduate medical education and residency training, and support the assumption that reflection also represents a crucial aspect of informal WPL in residency training [6].

The JSLO, or job-specific learning outcomes, focused on interpersonal relationships with the patient and effective, informed patient treatment. A good doctor-patient relationship facilitates a patient’s compliance and a physician’s job satisfaction and reduces the physician’s distress and burn-out risks [50]. Aspects of this factor are also covered in CanMEDS’ roles of the “communicator” and “collaborator”, where the communicator facilitates the doctor-patient relationship and shared decision making, and the collaborator fosters a successful relationship “with other health care professionals to provide safe, high-quality, patient-centred care” [48]. Given that patient encounters are central to the medical profession [51], and also represents a plausible facet of informal WPL outcomes.

OLLO accounts for organisational-level learning outcomes. This factor is comparable to the original factor introduced by Kyndt et al. [1]. OLLO covers the general aspects of a professional occupation that can be applied regardless of the specific work domain. Specifically, these include competencies for taking organisational and social responsibility, such as developing and implementing new ideas, considering one’s activities in the larger context, and dealing with ethical and social issues [1, 52]. These aspects of informal WPL outcomes seem to be transferable across different disciplines and play a crucial role in the medical field, as well as in social care or policy work environments [1, 6].

Finally, the CFA provided acceptable model fit values and showed comparable relationships between the factors, like the original instruments. The results support construct validity and indicate that the four factors support a similar construct [1, 39].

Our second research question aimed to examine the reliability of the instrument. Good internal consistency supports the assumption that the items measure the same underlying construct, which is also a prerequisite for a latent-variable analysis, like applying SEM [39], and for interpreting the validity of the measure [53]. All four factors provided good internal consistency. Compared to the original scale and the adaptation for police inspectors, similar internal consistency was measured. Therefore, we assume a reliable measure, which allowed for the further assumption that the content of the items of the subscales was homogeneous [39].

According to the results of our first and second research questions, we investigated our third research question on whether there was a relationship between the responses for informal WPL outcomes and the three competences of medical expertise, scholarship, and communication. Our study shows the relationships between these three competences and the four factors of the informal WPL outcomes measure. These correlations provide support for the assumption of construct validity [31, 39]. Applying the three competences as predictors of the informal WPL outcomes shows that “medical expertise” predicts the informal WPL outcomes of all four factors of the measure. This seems plausible due to the necessity of building baseline knowledge and skills to conduct anamnesis and to develop diagnoses and appropriate treatment plans [32, 48]. In addition, the predictive characteristics of scholarship for the two generic learning factors are plausible due to the ability to identify learning needs and to generate new knowledge [32, 48, 54]. The concept of scholarship contains the abilities of reading, interpreting, and applying scientific medical findings to the daily work at ward and to the particular patient’s needs and concerns [32], as well as to lifelong learning [48]. Communication had no predictive effect on any factor. The missing predictive relationship between communication and informal WPL outcomes could be due to the different aspects of the two measures. The communication factor of the FKM refers only to communication with patients and within teams, whereas the WPL factors refer more to the broader aspects of professional behaviour in the clinical setting and to skills such as scientific communication, conflict management, and giving and asking for feedback. Previous research has provided evidence that asking for and giving feedback are related to informal WPL outcomes [6, 55]. While the team communication aspect of the FKM communication scale could indicate a positive correlative relationship between communication and informal WPL outcomes, this does not explain why communicative competence predicts informal WPL outcomes.

The results pertaining to the relationship between informal WPL outcomes and the three medical competencies are in line with previous research on the antecedents of workplace learning, where the level of competence or basic-level skills relate to WPL outcomes [11, 55]. Our research findings thus provide initial exploratory results confirming that the informal WPL outcomes measure can provide appropriate information about different learning outcomes and their antecedents during residency.

While interpreting the data, we had to consider some limitations of our study. First, we used a convenience sample. We asked only graduates of one year in one federal state to participate via mail and email. Therefore, motivation and interest in medical education might have had an impact on their participation, and we cannot preclude the possibility that participants did not differ significantly from non-participants on these points. Second, competence assessments are based on self-assessments, which can be influenced by different biases. For instance, graduates have been shown to overestimate their abilities more often, especially those with lower competencies [56, 56]. In this case, the competencies of medical expertise, scholarship, and communication refer to the perceived competencies at the end of undergraduate studies, so a distorted perception through time might be plausible [57]. Furthermore, the recalling of experiences often refers to selective situations instead of all-day situations; therefore, the participants might not be able to accurately remember their demonstrated competencies and learning experiences in the workplace. These circumstances might influence the relationship between perceived competencies and WPL outcomes. Future studies could therefore relate the assessments of work samples to informal WPL learning outcomes to substantiate the validation process of the measure.

The practical implementation of our work here still has the characteristics of a research study. Our study is the first attempt to apply and validate this particular measure in (graduate) medical education. Our study results thus contribute to the research on learning conditions in residency training. However, to date, no recommendations or conclusions can be made about “good” WPL at the personal or organisational level. Further research on the predictors of WPL in residency training is thus needed to draw conclusions about the quality of WPL through the application of this measure.

Conclusion

This study provided evidence to validate the use of an existing instrument to assess informal WPL outcomes during residency. Although we were unsuccessful in replicating the three-factor model of the original questionnaire, our four-factor model nonetheless revealed meaningful factors of informal WPL related to residency training. The four factors were predicted by the competence facets of medical expertise and scholarship.

We are convinced that the tool is critical for identifying points for action and structural improvement with respect to the non-technical aspects of learning during residency. A longitudinal perspective should also be considered to examine learning outcomes during residency years.