Skip to main content
Download PDF

Research capacity and culture in an Australian metropolitan public mental health service: scoping the skills and experience of social workers and occupational therapists

BMC Medical Education volume 22, Article number: 864 (2022) Cite this article

Abstract

Background

Investment in a clinical research culture appears to be associated with benefits for consumers, staff, and overall organisational performance. The validated 55-item Research Capacity and Culture (RCC) tool was developed specifically to gauge the research capacity and culture of health professionals and workplace settings within which they work. Results of some individual studies suggest that professional discipline and workplace setting may impact RCC results however it has never been used in a dedicated public mental health setting. Therefore, this study will explore the research capacity and culture of allied mental health clinicians (Part 1). Another aim is to explore potential connections between workplace settings, locations and disciplines based on published RCC-based data to help signpost potential impediments to service improvements (Part 2).

Methods

Part 1: An RCC-based online survey canvased Australian Social Workers and Occupational Therapists (n = 59) based in a metropolitan public mental health service. Non-parametric analyses explored links between research-related experience and participant characteristics.

Part 2: Comparative analyses explored the potential influence of workplace settings and professional disciplines on published RCC results.

Results

Part 1: Overall, the research capacity and experiences of mental health Social Workers and Occupational Therapists seemed modest. Discipline was statistically associated with level of research-activity experience, weighted towards occupational therapy; demographic characteristics were not. Only two items in the RCC were rated high; many more items were rated low.

Part 2: Published studies exploration found no link between RCC ratings and workplace location, setting, or professional discipline. Sampling biases and use of modified, non-validated RCC versions likely impacted the results.

Conclusions

Allied mental health clinicians may not be sufficiently experienced, knowledgeable, or confident with a range of research-related activities given the emphasis on workforce research capability in policy and practice nowadays. This may be commonplace across health-based organisations. We recommend the systematic implementation of research training programs in (mental) health services, and a ‘whole-of-service levels’ approach be used i.e., transform policy, culture and leadership as well as provide practical resources with individual training. Potential benefits include a positive impact on organisation functioning, clinicians’ confidence and practice, and improved consumer outcomes.

Peer Review reports

Background

Disciplines such as occupational therapy and social work report being under pressure to deliver clinically relevant, cost-effective and knowledge (or research) informed practices [1, 2]. Allied health clinicians develop basic research-skills training during their undergraduate course, yet it is relatively unknown how these skills develop within the workforce [3]. Investment in a clinical research culture appears to be associated with benefits for consumers, staff and overall organisational performance [4]. Hardy [4] defines research culture as “a culture in which the application of evidence is valued, clinicians are encouraged to participate in research-related activities, opportunities are available for staff to acquire skills in research and evidence-based practice, research achievements are recognised and there is an investment of resources in research activity (p45)”.

Despite interest in research, allied health clinicians report many barriers including heavy workloads, staff turn-over and low staffing levels; lack of organisational resources and infrastructure; lack of own skills and knowledge; and low support from managers and colleagues [5,6,7]. These impediments seem to lie across individuals, practice-based teams, and organisations/services. Consequently, to remedy these barriers it is important to gauge the level of research-related skills, knowledge and experience within individuals, the capacity of the organisation infrastructure to support research, and the organisation’s culture or appetite for encouraging clinician-driven research.

A tool that was recently developed in Australia is the Research Capacity and Culture survey (RCC) [8]. The RCC was specifically developed to identify and understand the research capacity and culture of the health workforce and quantitatively measures the research capacity and culture across three levels: organisation, team and individual. The RCC was validated with excellent results [8] and has since been widely used in general medical health settings both in Australia (e.g. [9,10,11,12]) and overseas (e.g. [13, 14]).

Based on RCC results, workplace settings have been linked to the research capacity and confidence of clinicians. For example, Williams, Lazzarini et al. [15] found statistically significant differences in the self-rated research skills of Australian podiatrists across each of the RCC Individual domain items, with the breakdown from highest to lowest being non-clinical roles (managers, educators, and academics), public hospital, public community, private practice. Williams, Miyaki et al. [9] found that workplace location made a unique contribution to the variance in all RCC domain items rated by Victorian allied health clinicians, according to whether the health service was located in a metropolitan or regional area, weighted in favour of metropolitan area.

Similarly, professional discipline has been linked to the research capacity and confidence of clinicians. For example, Lee [11] found that individual research capacity differs according to discipline with both allied health staff and nursing staff being less confident across a range of research-based activities compared to medical staff in a Sydney-based general medical setting. Furthermore, two separate studies conducted within the same Queensland-based general health workplace and only two years apart, revealed medical doctors’ RCC scores [16] to be substantially higher than those of allied health professionals [17] indicating medical staff were more confident, skilled and experienced in research-related matters as well as evidently receiving more resources and support from their teams and the organisation as a whole compared to other health disciplines. This suggests that professional discipline may impact research confidence, skills, and perhaps even availability of resources.

While studies that have used the RCC often included mental health clinicians albeit in small proportions within their overall samples, all have been based within general medical settings (typically comprehensive medical service providers), and none were based in a dedicated mental health service setting. Therefore, the aim of this project is to explore the research-related knowledge and skills of allied health clinicians working within a metropolitan public mental health service and the capacity and culture of the organisation, using the RCC tool (Part 1).

There is some evidence that workplace settings and professional discipline may impact the research capacity and culture of clinicians. Therefore, a further aim was to explore the potential connections that workplace setting, location and professional discipline may have with research capacity and culture using the results of this study along with RCC-based data sourced from published studies (Part 2). Undertaking this extended exploration may provide clues about which of those factors, if any, may be linked with higher RCC scores and consequently best practice.

Methods

Part 1: within dedicated public mental health setting

An exploratory mixed methods study design was implemented consisting of an online survey and semi-structured interviews. The anonymous online survey consisted of demographic and work-related items, the validated RCC tool [8], eight open-ended questions and a call for volunteers willing to be interviewed. The qualitative results from the open-ended questions and semi-structured interviews will be reported elsewhere. Survey data were collected and managed using REDCap [18] an electronic data capture tool hosted by the Royal Melbourne Hospital Business Intelligence Unit.

Workplace setting

This study was conducted in a metropolitan public mental health service in Victoria, Australia. This area-based mental health service provides a comprehensive range of specialist, community, and hospital-based mental health services for youth, adult and aged people who are experiencing, or are at risk of developing a serious mental illness. Services are delivered across a range of locations, including most major hospitals within the service’s area of responsibility, and a range of community-based clinics. In addition, the service delivers several state-wide specialist services including the Neuropsychiatry Unit and an Eating Disorders Program. The project focused on social workers and occupational therapists for pragmatic reasons. Arguably these two MH-based disciplines receive very similar levels of research-related training during their undergraduate years. In addition, occupational therapy had employed a Research Lead (a standalone position dedicated to promoting research projects and knowledge translation activities) for some years while the social work department did not have an equivalent dedicated standalone position until recently.

Recruitment method

All social workers and occupational therapists employed within the service received a recruitment email comprising an invitation to participate in the project, an attached plain language statement and a link to the anonymous online survey, via their ‘discipline specific all-staff’ emailing list. The recruitment email was re-sent on a further two occasions within a six-week period during October and November 2020. The inclusion criteria were being a qualified social worker or occupational therapist employed within the mental health service at the time of recruitment, regardless of position/role, hours worked per week or whether employed on a full-time, part-time, contract or casual basis.

Participant demographic and discipline-related characteristics

The online survey comprised demographic and discipline related information including gender, age, discipline, highest education level, and years since completed last qualification. Educational level was partitioned into (a) undergraduate only (b) undertaking postgraduate coursework (c) completed postgraduate coursework (d) undertaking postgraduate research, and (e) completed postgraduate research. Time since last qualification was partitioned into 5-year subgroups and a 20-years + subgroup.

Participant research-related activity characteristics

Aside from the RCC tool (see below), participants were asked to indicate all the types of research-related activities with which they had been involved, over the last 5-years. They were asked to tick all that applied, with the options being (a) project work (b) quality assurance or QA (c) evaluation (d) research, and (e) none of the above; these subcategories were not mutually exclusive except for ‘none of the above’. This item provides evidence about the range or breadth of research-types with which participants had been involved, as well as the number of types within individuals. These qualities were used for descriptive purposes only. While these activities are ordinarily discussed as discrete types of undertakings, they operate on a continuum from quality assurance to original research [19,20,21]. Moreover, all involve the processes of identifying/appraising literature, identifying levels of risk and the ethical consequence of same, use scientific methodology to collect empirical data, and utilising the results [19,20,21] and consequently require a similar set of skills. Arguably, project work in mental health services also involve very similar processes i.e., identify a clinical problem then appraise the literature, identify risk, implement program, collect data/evaluate, and utilise results. Therefore, it is reasonable to surmise that a similar set of skills would be needed so for the purposes of this study, all terms were considered equivalent; for the analysis the variable was converted to a dichotomous variable of any research experience and no research experience.

Research capacity and culture tool (RCC)

The RCC [8] is a self-report tool with excellent internal consistency with Cronbach alpha coefficients reported across the three domains of Organisation (20 items), Team (20 items) and Individual (15 items): α = 0.95, 0.96 and 0.96, respectively. In this study, the Cronbach alpha coefficients for the domains of Organisation, Team and Individual were α = 0.97, 0.98 and 0.95 respectively, indicating excellent internal consistency and reliability of the scale. Each item can be scored from 1 = Lowest possible skill or success level, to 10 = highest possible skill or success level. Accordingly, the domain score ranges are Organisation 20 to 200, Team 20 to 200, and Individual 15 to150. Median and interquartile range (IQR) were calculated for the summary RCC item and domain statistics since they violated the assumption of normal distribution: almost every item was positively skewed, and kurtosis values were below zero. Median average scores were collapsed into three subcategories of low skill/success (scores 1–3.99), moderate skill/success (scores 4–6.99), and high skill/success (scores 7–10), in keeping with previous studies such as Alison [12], Friesen [6], and Matus [17].

Within study analyses

Non-parametric analyses (Chi-Square Test of Independence and Fisher’s Exact Probability Test) were used to explore associations between the participant characteristics and background with (any) experience in research-related activities, due to the categorical nature of the variables and the size and distribution of the sample.

To comply with the minimum cell frequency assumption of ≥ 80% of cells should have expected frequencies of ≥ 5 associated with Chi-square test for independence, the following variables were transformed: Age groups were condensed into three groups (20-34yrs, 35-49yrs, 50yrs +); Highest education level groups were condensed into three groups (undergrad only, PG-coursework (undertaking or completed), PG- research (undertaking or completed)); and, Time since last qualification groups were condensed into three groups (0-6yrs, 7-10yrs, 11yrs +).

A Friedman test was performed to determine whether there was a difference in the median scores between Organisation, Team and Individual RCC domains. Post hoc Wilcoxon Signed Rank Tests were performed to determine where the differences lay. Effect sizes (r) were calculated with interpretation based on Cohen’s criteria of 0.1 = small effect, 0.3 = medium effect, 0.5 = large effect. Mann–Whitney U Tests were performed to determine if RCC domain results were associated with any research-related experience [22].

Part 2: across published studies comparative analyses: workplace settings and discipline

An extended exploration was undertaken to look at factors (Workplace setting, professional discipline) that might be associated with higher RCC scores and consequently potential best practice. Three methods to identify relevant studies were used. The first method was to search Google Scholar using the term “research capacity and culture”. Since Google Scholar is less rigorous, more generous in search returns, it was thought likely most if not all studies would be listed. The second method was to repeat this method using Google itself. The third method was to use the online citation analysis tool CitationChaser which searched for citations across the LENS.org database (consisting of PubMed, PubMed Central, CrossRef, Microsoft Academic Graph and CORE), using the DOI of Holden’s 2012 foundation validation article. The first two methods were ‘topic-based” methods. The third method was a citation search based on the foundation article that introduced the RCC.

The inclusion criteria for studies to be included in our analyses were required to be:

  1. (a)

    Peer-reviewed

  2. (b)

    Based in Australia – since health systems across countries are too dissimilar

  3. (c)

    Report RCC sourced data across the three RCC categories of organisation, team and individual – since the objective of Part 2 is to compare RCC results across workplace settings and disciplines

  4. (d)

    Report results as medians and IQR – while both means and medians are both types of statistical averages, means should only be used when the distribution of data is normally distributed (i.e., bell shaped distribution) as means are unduly influenced by skewed data and outliers. In contrast, medians are robust measures of central tendency and used for non-normal distributions of data. Consequently, means and medians should not be used within the same comparative analysis

All articles that cited the foundation article published in 2012 up to July 2021 found by CitationChaser confirmed our Google Scholar results. Thirty citations were identified with 20 citations being excluded due to: 5 × citations mentioned RCC but did not include any data [23,24,25,26,27], 1 × citation included RCC domain Team data only [28], 2 × citations were based in educational settings [29, 30]; 5 × citations were based overseas [13, 14, 29,30,31], 10 × citations used means and standard deviations [6, 28, 32,33,34,35,36,37,38,39], and 2 × citations were not peer reviewed [37, 39] [NB: these categories are not mutually exclusive]. See Table 1 for the final list of 10 included studies and full breakdown of study sample characteristics.

Table 1 Descriptive features of samples used within peer-reviewed published studies that used the RCC (abridged version described)
Full size table

Settings and discipline were explored in two ways. The first was a graphical exploration whereby the median item scores from each study were collapsed into three subcategories of low skill/success (scores 1–3.99), moderate skill/success (scores 4–6.99), and high skill/success (scores 7–10), in keeping with previous studies such as Alison -[12], Friesen [6], and Matus [17]. The number of items located within each subcategory within each domain were then summed and the relative distribution displayed in graphs for visual inspection.

The second way was to apply three comparative bivariate statistical analyses on the reported RCC outcomes using the summed RCC domain median item scores from each study as the dependent variable with the independent variables being professional discipline (specified discipline or mixed disciplines), workplace setting (single service/organisation or multiple organisations), and location (metro, regional/rural, state/national). Please note, Lee’s study [11] reported RCC results as three discrete professional discipline types: doctors, nurses, and allied health professionals. For the comparator analyses where the independent variable was professional discipline, these groups were treated as discrete independent variables. For the comparator analyses where the independent variable was workplace setting, the mean of these three groups was used since each group was sampled from the same setting.

The 10 included publications were Australian studies that reported median and IQR per item per RCC domain. Excluded were 13 studies that reported means and standard deviations (n = 10) [6, 28, 32, 33, 35, 36, 38, 39], were set outside of Australia (n = 3) [13, 14, 31], were not peer-reviewed [37, 39] and/or did not report RCC results for each domain [34].

It is important to note that for comparative purposes, two items (#4 and #20) from the domain Organisation, and one item from each of the domains Team (#30) and Individual (#54) were omitted from Lazzarini [40], Holden [8] and the current study results, to match the abridged versions of the RCC used by the other comparator studies. Consequently, the range of potential summed scores became: Organisation = 18–180, Team = 19–190, and Individual = 14–140.

The data was analysed using IBM SPSS v26 and MS Excel 2013, and the Free Statistics Calculator v4 (see Free Statistics Calculators—Home (danielsoper.com) for the Fisher’s Exact Probability Test with Freeman-Halton extension for the 2 × 3 tables. P-values < 0.05 were considered statistically significant. Effect sizes were calculated for statistically significant results. Cohen’s recommended interpretation of the phi coefficient effect size is as follows: 0.10 for small effect, 0.30 for medium effect and 0.50 for large effect [22].

All applicable institutional and governmental regulations concerning the ethical use of human volunteers and in accordance with the Declaration of Helsinki, were followed during the course of this research. Each participant received a plain language statement and opportunity to ask questions about the study. Informed consent was deemed given with the voluntary completion of the survey. This project received full ethics approval from the Melbourne Health Ethics Committee: HREC/64816/MH-2020.

Results

The overarching aim of this project was to broadly explore the research-related knowledge, skills and experience of allied health staff working within a large Australian metropolitan public mental health service and the research-related capacity and culture of the organisation. A further aim was to consider the potential connections that workplace settings and professional discipline may have with research-related capacity and culture using the results of this study along with the RCC -based data sourced from published studies. Included studies were all peer-reviewed, Australian-based studies that reported median and interquartile range per item per RCC domain.

Part 1: participant characteristics and background

Survey participants

Overall, 59 social workers and occupational therapists responded to the survey invitation, representing an approximate 22% response rate. The most common participant characteristics were female, aged 30-39yrs, had completed postgraduate coursework within the last two years, and was a Social Worker. See Table 2. Most participants (58%) had experience with project work and to a lesser extent evaluation (39%) while 32% of participants indicated that they had no experience with any of the research-related activity types.

Table 2 Demographic characteristics of Survey completers (N = 59)
Full size table

Within study analyses

Associations between the categorical independent variables (Time since completed last qualification, Professional Discipline, Age group, and Gender) with any level of research-related activity experience were explored. Statistical non-significance was found for Time since completed last qualification (p = 0.53), Age group (p = 0.78), and Gender (p = 0.27). A statistically significant difference was found for Professional Discipline (p = 0.048) with a small to medium effect size (phi = 0.28) weighted towards occupational therapists more likely to have research-related experience.

See supplementary material for more detail.

Research capacity and culture tool (RCC)

RCC—Organisation

Within the domain Organisation, the most common median scores fell within the medium category. The median score for three items relating to organisation-provided resources was low: (#1) Provides resources to support staff research training; (#2) Provides funds, equipment or admin to support research training; and (12) Provides software programs for analyzing research data. In contrast, the median score for two items relating to the way clinicians are expected to practice was categorised as high: (#10) Promotes clinical practice based on evidence; and #20 Requires ethics approval for research activities. See Table 3.

Table 3 Median and Interquartile (IQR) range, and relative category per item in RCC domain Organisation
Full size table

RCC -Team

Within the domain Team, the most common median score fell within the low category. No team items could be classified as high. Seven team items could be classified as medium, tending towards the lower end of medium. Participants indicated that they had some confidence/success in team leaders who support research (#25), that consumer involvement in research is encouraged (#28), that experts are accessible for research advice (#34), and that disseminating research results with others at research forums/seminars is supported (#35). In contrast, items that could be linked to the provision of resources (#21), equipment & funds (#22) and other ‘doing’ activities such as applying for funds (#29), conducting research (#31), accessing literature (#30) were rated low. See Table 4.

Table 4 Median and interquartile (IQR) range and relative category per item in RCC domain Team
Full size table

RCC – Individual

Within the domain Individual, the most common median score fell within the low category. Six items could be classified as medium: finding (#41), reviewing (#42) & integrating (#54) the literature, designing questionnaires (#47), collecting data (#48), and analysing qualitive research data (#50). However, no item could be classified as high. See Table 5

Table 5 Median and interquartile (IQR) range and relative category per item in RCC domain Individual
Full size table

Differences between RCC domains within the mental health sample

The overall median and interquartile scores for each domain were Organisation Md = 5.0 (IQR: 3.0–6.0), Team Md = 3.0 (IQR: 1.0–5.0), and Individual Md = 3.0 (IQR:1.0–5.0). The results of the Friedman Test indicated that there was a statistically significant difference in RCC skills/success levels across the three domains (Organisation, Team, Individual, (χ2 (2, 59) = 21.8, p < 0.001). Three post hoc Wilcoxon Signed Rank Tests (using Bonferroni adjusted alpha values to control for Type 1 error: 0.05/3 = 0.017) compared Organisation with Team, Organisation with Individual, and Team with Individual. There was a statistically significant higher endorsement of Organisation skill/success levels compared to both Team (z = -4.7, p < 0.0005) with a medium effect size (r = 0.43) and Individual (z = -3.4, p = 0.001) with a medium effect size (r = 0.32). There was a non-significant difference between Team and Individual (p = 0.88, r = 0.01).

Mann–Whitney U Tests revealed no significant differences in overall RCC domains Organisation or Teams in participants with and without research-related experience. There was a statistically significant difference in skill/success levels in RCC domain Individual between participants with research experience and those without, weighted in favour of with experience (p = 0.04) however, the effect size was small (r = 0.27). See supplementary file.

Part 2: across published studies comparative analyses: workplace settings and professional discipline

Graphical display of the relative groupings of the RCC item ratings reveals seemingly strong differences between studies and therefore ratings as reported by the individuals within each study sample. Overall, the proportion of items within the poor category (Organisation = 7%, Team = 21%, Individual = 28%) seemed to increase the more proximal the domain is to the individual. The proportion of items within the high category (Organisation = 21%, Team = 20%, Individual = 21%) seemed to be relatively stable across the domains. However, the most common rating within each of the domains was moderate (Organisation = 71%, Team = 59%, Individual = 51%). See Fig. 1.

Fig. 1
figure 1

Graphical display of relative groupings of RCC items across published studies

Full size image

Comparative analyses were possible to explore whether the workplace setting (single health service or multiple health services) or location (metro, regional/rural, state/national), or professional discipline (discipline specific or mixed disciplines) influenced RCC domain scores. Item medians per RCC domain were summed within each study thus offering an overall representation of the perceived research capacity and culture in the various health settings investigated by the selected published studies. Two independent samples t-tests and one ANOVA were conducted. Neither workplace settings (single service or multiple services: Organisation p = 0.55; Team p = 0.91; Individual p = 0.37) nor the discipline (specific discipline or mixed discipline: Organisation p = 0.0.71; Team p = 0.88; Individual p = 0.72), nor workplace location (metro, regional/rural, state/national: Organisation p = 0.77; Team p = 0.23; Individual p = 0.32) were statistically significant across the domains. See supplementary material.

Discussion

To the best of our knowledge, this is the first study to use the RCC in a public mental health service in Australia or internationally. This discussion will begin by reflecting on the within mental health service results (Part 1) followed by the across studies results (Part 2). Overall, the reported research capacity and experiences of the allied mental health clinicians seem modest but not unlike some earlier studies e.g. Holden et al. [8]. Only two items across all the RCC items were rated highly; many more items were rated low. The domain Organisation seemed to elicit the most homogenous range of responses, having a somewhat normal distribution; the highest number of median item responses fell within the moderate range. Interestingly, it was two Organisation domain items that related to how clinicians are expected to practice (promoting clinical practice based on evidence and requiring ethics approval for research activities) that received the highest rating from participants.

There was more variability in the Team related items both within the current study based in a mental health service as well as other health services represented in the published studies e.g. Holden et al. [8], and Lee et al. [11]; clearly, some teams work better than others. This could be because there are many different teams within a mental health service. Leadership style has also been linked with team functioning e.g., see Corrigan et al. [41] and Hitch et al. [42]. Participants came from several teams within the mental health service and likely experienced a range of different leadership styles though this was beyond the remit of this study to investigate. Our results could be evidence that obstacles are in place that prevent the more positive overall organisational culture from filtering down adequately. If this was the case, then it would be reasonable to surmise that low ratings would increase as items became more closely related to the individual and consequently more distal from the organisation as a whole and indeed, that was found. An alternative explanation could be the survey terminology itself. Team can mean different things to different people, like the definitions of family. Within public mental health services, a team can be a discipline specific team within a service such as a team of Social Workers or Occupational Therapists or a multidisciplinary collaborative outreach team that could comprise social workers, occupational therapists, nurses, psychologists, medical staff and/or lived experience consultants. Teams can be based in the community or within an acute inpatient setting. In recognition of these differences, this study deliberately did not provide a definition of team, leaving it to the individual to decide where they saw themselves and perhaps, unintentionally allowed bias to creep into the results. A further possibility is that it may not be seen as the role of small units such as teams, to support research (e.g., many teams would not have the capacity to provide funds and if particular supports are provided by the organisation, then perhaps, need not be duplicated by individual teams). Alternatively, these results could simply reflect characteristics of a modest sized sample.

Most of the Individual items were classified as low suggesting that participants rated their research-related skills and knowledge as low. This may be expected when looking at the breadth of their research-related activity experience but perhaps not if one considers the level of post-graduate education represented in the cohort. Participants did feel moderately confident in finding, critically reviewing, and integrating the literature into their practice. Arguably these were the items that are the most amendable/transferable to everyday practice. In contrast, it was the items more closely associated with doing research, evaluation, and quality assurance projects [43,44,45] such as designing a study, submitting an ethics application, and writing for publication, which were reported low by the participants. Nevertheless, these results are in keeping with earlier studies, revealing this to be a common enough dilemma for allied health clinicians across various health settings (e.g. [7, 8, 12]), possibly reflecting an historical lack of research training and mentoring in the workplace [12].

Exploration of participants’ personal and professional characteristics revealed occupational therapists’ participation in research-related activities in the last five years was significantly higher than social workers however this was not unexpected. The service has a muddled history with long term discontinuities in research support whether occupational therapy or social work, over the years. However, since 2014, the service has employed part/time, an occupational therapist Research Lead with an increase in the quantity of quality assurance projects and knowledge translation activities able to be reported within a couple of years [46]. There has not been an equivalent dedicated standalone position within the social work department until recently and even then, with a smaller time fraction. The potential benefit of employing a discipline research lead is in keeping with the findings of Williams, Miyasaki et al. [9] and Wenke et al. [47], with both studies finding that employing a Research Lead was beneficial to the research capacity, skills and culture of allied health clinicians in other healthcare settings.

It is worth noting that more than half of the participants reported either no experience with research-related activities or experience with only one type, most commonly project work. The lack of experience was not statistically linked to age or time since participants completed their last qualification. As a rule, we are unlikely to participate in activities in which we have no confidence or are not viewed as part of our everyday remit. Fortunately, there is growing empirical evidence that the implementation of research training programs substantially improves confidence, engagement and practice that positively impact on organisational functioning and consumer care (e.g. [4, 48]). Our findings suggest that a systematic implementation of research training programs is needed, with the potential to positively impact organisation functioning, clinicians’ confidence and practice, and consumer outcomes.

Part 2: across studies commentary

It is worth remembering that comparative research methods underpin discovery, help detect associations and causes that enable theories to be constructed, and deepens our understanding; in short, comparative research methods underpins knowledge development and consequently guides best practice. A further aim of our study was to consider the potential influences that workplace setting, location and professional discipline may have on research capacity and culture using the results of this study along with RCC-based data sourced from published studies. It was thought that this extended exploration may provide clues about which of those factors, if any, may be linked with higher RCC scores and consequently best practice.

There was a variety of distribution patterns across the published studies however the results from the comparative analyses were not significant; neither workplace location, workplace setting, nor professional discipline were statistically linked with RCC ratings. In contrast, both studies led by Williams [9, 15] suggest workplace setting and location does make a difference to the research capacity and culture experienced by clinicians with a greater capacity found in metropolitan vs non-metropolitan locations, and greater capacity in public sector vs private practice. Similarly, the studies by Brandenburg [16] Matus [17] and Lee [11] suggest that professional discipline can make a meaningful difference to the perceived research capacity and culture of the organisation as a whole, the team within which one is based, as well as differing capacities across disciplines, .e.g., medical staff reported greater personal capacity than allied health staff who in turn reported greater personal capacity than nursing staff [11].

Two broad reasons could account for these inconsistencies, (a) Bias and (b) Sampling methods. (a) RCC versions: Our study used the original version of the RCC which meant it included the full complement of items within each domain while all, but the original published studies used abridged versions, though, for the between studies exploration, we omitted the same items for equivalence. Bias may have been introduced with the addition of an unsure/unknown option used by roughly 40% of the published studies (e.g. [7, 12, 16, 17]) but not present in the original version. It is possible that participants in those studies may have nominated usure/unknown rather than rating their workplace organisation and team within the low range thus potentially biasing those study results.

Bias may have been introduced with the variety of abridged RCC versions that have been used. While all publications cite Holden’s validation study that published the original wording, number of items and with forced responses [8], none had used the original version of the tool. Some of the changes have included relatively minor word changes such as changing the tense of verbs – finds becomes finding (e.g. [9]). Other changes have a more qualitative distinction potentially changing the intent of items. For example, Provides experts for research advice became has identified experts accessible for research advice (e.g. [15]) that further morphed into arranges experts to give research advice [7] by yet another investigator team. Changes made to an instrument should be acknowledged and well-considered. Further validation is required when it is deemed necessary to make such changes.

(b) Sampling methods: Selection bias that may have been introduced by the sampling method used in the published studies. It is unclear what the benefit would be to include corporate and clerical staff along with clinical health professionals in a single sample (e.g. [11]); similarly including clinical health professionals for whom research is a prescribed part of their role together with others where it is not, or including staff such as corporate and clerical staff, and/or allied health or welfare assistants who do not require tertiary education to fulfil their role and therefore would lack research-related training (e.g. [7]). Computing summary outcomes based on such heterogeneous samples is likely to undermine both relevance and generalisability in the results.

Nonetheless, there were limitations to our study too. All the participants in the current study were clinicians who volunteered to complete the survey. While there is no direct evidence available, it is reasonable to assume that survey participants were more interested in research-related activities compared to non-responders. There is no reason why all clinicians must be involved in research, so they need not score highly in the experience-based items within the RCC domain Individual. However, given that survey respondents may have been skewed to those interested in research then it is problematic that so many had little to no experience. All the investigators in our study were also employed by the organisation at the time of the study however we were under no obligation to find any particular result. Our sample size was relatively small resulting in modest power to find statistically meaningful outcomes however our research focus was both explorative and descriptive. Our comparative exploration across published studies was restricted to Australian-based studies that reported RCC outcomes as median and IQR per item per RCC domain. Excluded were studies that reported RCC outcomes in means and standard deviations, studies conducted outside of Australia and/or not peer reviewed. However interestingly, those other studies also made changes to the RCC tool that would make comparative analyses challenging. It would have been interesting to compare clinicians employed in a dedicated mental health setting to those based in general medical settings however this was found not to be a statistically viable option.

Notwithstanding the above, there were strengths to our study. To the best of our knowledge our study was the first study to use the RCC tool exclusively in a public mental health setting. We used the RCC tool in its original validated form enhancing rigor in our results. Our survey was limited to two disciplines thus minimising potential sampling biases. Our survey was both anonymous and used the original forced response methodology thus arguably, minimising potential response bias.

In conclusion, our study provides evidence that allied mental health clinicians based in public mental health settings in Australia may not be sufficiently experienced, knowledgeable, or confident with a range of research-related activities given the emphasis on workforce research capability in policy and practice nowadays. Clinicians had moderate confidence in their ability to find, critically review, and integrate the literature into their practice. However, they were substantially less knowledgeable, experienced, and confident with other research-related activities such as analysing qualitative or quantitative data, designing a study, or submitting an ethics application. Limited comparative analysis to explore the potential link between workplace setting, workplace location and professional discipline, with research capacity and culture was not statistically significant however drilling down into the published studies revealed several methodological biases that may have affected results. Clearly there is much work to be done across the levels of organisation, team, and individual to increase allied mental health clinicians’ skills, knowledge, confidence, and resources connected to research-related activities in health services including mental health services. There is growing empirical evidence that the employment of discipline research leads and the implementation of research training programs positively impacts organisation functioning, clinicians’ confidence and practice, and consumer outcomes. For these reasons, the following recommendations can be made

  1. (a)

    Health services, including mental health services, need to take a ‘whole of service levels’ approach to cultivate their research culture/ethos i.e., need to address each level of the organisation (organisation as a whole, teams and individuals) applying different strategies to the different levels

  2. (b)

    Health services, including mental health services, need to implement quality training programs that focus on research-related activities

  3. (c)

    Health services, including mental health services, need to recognise that some disciplines may require more active support to conduct research-related activities

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Abbreviations

RCC:

Research Capacity and Culture survey

QA:

Quality Assurance

Md:

Median

IQR:

Interquartile range

References

  1. Morris K, Smyth G. A survey of research capacity and partnerships among mental health occupational therapists in the UK. Ment Health Rev J. 2017;22(2):136–47.

    Article  Google Scholar 

  2. Humphreys C, Berridge D, Butler I, Ruddick R. Making Research Count: the development of knowledge based practice. Res Policy Plan. 2003;21(1):41–9.

    Google Scholar 

  3. Harding KE, Stephens D, Taylor NF, Chu E, Wilby A. Development and evaluation of an allied health research training scheme. J Allied Health. 2010;39(4):143E-148E.

    Google Scholar 

  4. Harding K, Lynch L, Porter J, Taylor NF. Organisational benefits of a strong research culture in a health service: a systematic review. Aust Health Rev. 2017;41(1):45–53.

    Article  Google Scholar 

  5. Pager S, Holden L, Golenko X. Motivators, enablers, and barriers to building allied health research capacity. J Multidiscip Healthc. 2012;5:53–9.

    Article  Google Scholar 

  6. Friesen EL, Comino EJ. Research culture and capacity in community health services: results of a structured survey of staff. Aust J Prim Health. 2017;23(2):123–31.

    Article  Google Scholar 

  7. Matus J, Tearne JE, Blyth K, Coates S, Pearson S, Cavalheri V. An evaluation of research capacity and culture in a sample of Western Australian Allied Health professionals. Tasman Med J. 2021;3(1):23–9.

    Google Scholar 

  8. Holden L, Pager S, Golenko X, Ware RS. Validation of the research capacity and culture (RCC) tool: measuring RCC at individual, team and organisation levels. Aust J Prim Health. 2012;18(1):62–7.

    Article  Google Scholar 

  9. Williams C, Miyazaki K, Borkowski D, McKinstry C, Cotchet M, Haines T. Research capacity and culture of the Victorian public health allied health workforce is influenced by key research support staff and location. Aust Health Rev. 2015;39(3):303–11.

    Article  Google Scholar 

  10. Gill SD, Gwini SM, Otmar R, Lane SE, Quirk F, Fuscaldo G. Assessing research capacity in Victoria’s south-west health service providers. Aust J Rural Health. 2019;27(6):505–13.

    Article  Google Scholar 

  11. Lee SA, Byth K, Gifford JA, Balasubramanian M, Fozzard CA, Skapetis T, Flood VM. Assessment of Health Research Capacity in Western Sydney Local Health District (WSLHD): A Study on Medical, Nursing and Allied Health Professionals. J Multidiscip Healthc. 2020;13:153–63.

    Article  Google Scholar 

  12. Alison JA, Zafiropoulos B, Heard R. Key factors influencing allied health research capacity in a large Australian metropolitan health district. J Multidiscip Healthc. 2017;10:277–91.

    Article  Google Scholar 

  13. Boyd M, Gall SB, Rothpletz-Puglia P, Parrott JS, King C, Byham-Gray L. Characteristics and Drivers of the Registered Dietitian Nutritionist’s Sustained Involvement in Clinical Research Activities: A Mixed Methods Study. J Acad Nutr Diet. 2019;119(12):2099–108.

    Article  Google Scholar 

  14. Luckson M, Duncan F, Rajai A, Haigh C. Exploring the research culture of nurses and allied health professionals (AHP s) in a research-focused and a non-research-focused healthcare organisation in the UK. J Clin Nurs. 2018;27(7–8):e1462–76.

    Article  Google Scholar 

  15. Williams C, Lazzarini P. The research capacity and culture of Australian podiatrists. J Foot Ankle Res. 2015;8(1):11.

    Article  Google Scholar 

  16. Brandenburg C, Noble C, Wenke R, Hughes I, Barrett A, Wellwood J, Mickan S. Relationship Between Research Culture and Research Activity of Medical Doctors: A Survey and Audit. J Multidiscip Healthc. 2021;14:2137–50.

    Article  Google Scholar 

  17. Matus J, Wenke R, Hughes I, Mickan S. Evaluation of the research capacity and culture of allied health professionals in a large regional public health service. J Multidiscip Healthc. 2019;12:83–96.

    Article  Google Scholar 

  18. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–81.

    Article  Google Scholar 

  19. Jarvis H. Differentiating among Research, Evaluation and Measures to Assure Quality. Healthc Manage Forum. 2000;13(4):34–6.

    Article  Google Scholar 

  20. Differences between research, quality assurance & quality improvement. https://www.ualberta.ca/research/research-support/research-ethics-office/human-research-ethics/research-qi.html

  21. Shirey MR, Hauck SL, Embree JL, Kinner TJ, Schaar GL, Phillips LA, Ashby SR, Swenty CF, McCool IA. Showcasing Differences Between Quality Improvement, Evidence-Based Practice, and Research. J Contin Educ Nurs. 2011;42(2):57–68.

    Article  Google Scholar 

  22. Pallant JF. SPSS survival manual : a step by step guide to data analysis using IBM SPSS / Julie Pallant. Crows Nest: Allen & Unwin; 2013.

  23. Friesen EL, Comino EJ, Powell Davies G. A Research Capacity Building Strategy for SWSLHD Division of Community Health 2015–2020, vol. NA. Liverpool: Primary & Community Health Research Unit (PCHRU); 2015. p. NA.

    Google Scholar 

  24. McBride KE, Steffens D, Duncan K, Bannon PG, Solomon MJ. Knowledge and attitudes of theatre staff prior to the implementation of robotic-assisted surgery in the public sector. PloS one. 2019;14(3):e0213840.

    Article  Google Scholar 

  25. Mickan S, Wenke R, Weir K, Bialocerkowski A, Noble C. Strategies for research engagement of clinicians in allied health (STRETCH): a mixed methods research protocol. BMJ Open. 2017;7(9):e014876.

    Article  Google Scholar 

  26. Oliver-Baxter J, Brown L, McIntyre E. Surviving or thriving in the primary health care research workforce: the Australian experience. Aust J Prim Health. 2017;23(2):183–8.

    Article  Google Scholar 

  27. Parker V, Lieschke G, Giles M. Ground-up-top down: a mixed method action research study aimed at normalising research in practice for nurses and midwives. BMC Nurs. 2017;16(1):1–8.

    Article  Google Scholar 

  28. Wenke RJ, Mickan S, Bisset L. A cross sectional observational study of research activity of allied health teams: is there a link with self-reported success, motivators and barriers to undertaking research? BMC Health Serv Res. 2017;17(1):114.

    Article  Google Scholar 

  29. Cha C, Hwang H, An B, Jeong S, Yang SJ. Nursing student and faculty competency improvement through a nurse-bridging program in Cambodia. Nurs Educ Today. 2020;93(NA):104523.

    Article  Google Scholar 

  30. Hand RK, Sears E, Harris SR. Research Involvement of Alumni From a Combined Dietetic Internship/Master’s Degree Program With a Research Concentration. Top Clin Nutr. 2020;35(4):329–40.

    Article  Google Scholar 

  31. Gimeno H, Alderson L, Waite G, Chugh D, O’Connor G, Pepper L, Gibson F, Wray J, Sell D. Frontline Allied Health Professionals in a Tertiary Children’s Hospital: Moving Forward Research Capacity, Culture and Engagement. Int J Pract Based Learn Health Soc Care. 2021;9(1):29–49.

    Article  Google Scholar 

  32. Borkowski D, McKinstry C, Cotchett M. Research culture in a regional allied health setting. Aust J Prim Health. 2017;23(3):300–6.

    Article  Google Scholar 

  33. Elphinston RA, Pager S. Untapped Potential: Psychologists Leading Research in Clinical Practice. Aust Psychol. 2015;50(2):115–21.

    Article  Google Scholar 

  34. Holden L, Pager S, Golenko X, Ware RS, Weare R. Evaluating a team-based approach to research capacity building using a matched-pairs study design. BMC Fam Pract. 2012;13:16.

    Article  Google Scholar 

  35. Howard AJ, Ferguson M, Wilkinson P, Campbell KL. Involvement in research activities and factors influencing research capacity among dietitians. J Hum Nutr Diet. 2013;26(s1):180–7.

    Article  Google Scholar 

  36. McBride KE, Young JM, Bannon PG, Solomon MJ. Assessing surgical research at the teaching hospital level. ANZ J Surg. 2017;87(1–2):70–5.

    Article  Google Scholar 

  37. Raschke N. The perceived research capacity and culture within non-metropolitan local health districts in NSW. Sydney: Heath Education and Training Institute; 2017.

    Google Scholar 

  38. Wenke R, Weir KA, Noble C, Mahoney J, Mickan S. Not enough time for research? Use of supported funding to promote allied health research activity. J Multidiscip Healthc. 2018;11:269–77.

    Article  Google Scholar 

  39. Goncalves J, White D. Survey on research capacity and culture. 2018.

    Google Scholar 

  40. Lazzarini PA, Geraghty J, Kinnear EM, Butterworth M, Ward D. Research capacity and culture in podiatry: early observations within Queensland Health. J Foot Ankle Res. 2013;6(1):1.

    Article  Google Scholar 

  41. Corrigan PW, Diwan S, Campion J, Rashid F. Transformational Leadership and the Mental Health Team. Adm Policy Ment Health Ment Health Serv Res. 2002;30(2):97–108.

    Article  Google Scholar 

  42. Hitch D, Lhuede K, Giles S, Low R, Cranwell K, Stefaniak R. Perceptions of leadership styles in occupational therapy practice. Leadersh Health Serv. 2020;33(3):295–306.

    Article  Google Scholar 

  43. Dawes M, Summerskill W, Glasziou P, Cartabellotta A, Martin J, Hopayian K, Porzsolt F, Burls A, Osborne J, Second International Conference of Evidence-Based Health Care Teachers Developers. Sicily statement on evidence-based practice. BMC Med Educ. 2005;5(1):1.

    Article  Google Scholar 

  44. Drisko JW, Grady MD. Evidence-Based Practice in Clinical Social Work. 2nd ed. Cham: Springer International Publishing; 2019.

    Book  Google Scholar 

  45. Hoffmann T, Bennett S, Del Mar C. Introduction to evidence-based practice. In: Hoffman T, Bennett S, Del Mar C, editors. Evidence-based practice across health professions. 3rd ed. Australia: Elsevier; 2018. p. 1–15.

    Google Scholar 

  46. Hitch D, Lhuede K, Vernon L, Pepin G, Stagnitti K. Longitudinal evaluation of a knowledge translation role in occupational therapy. BMC Health Serv Res. 2019;19(1):154.

    Article  Google Scholar 

  47. Wenke RJ, Ward EC, Hickman I, Hulcombe J, Phillips R, Mickan S. Allied health research positions: a qualitative evaluation of their impact. Health Res Policy Syst. 2017;15(1):6.

    Article  Google Scholar 

  48. Black AT, Balneaves LG, Garossino C, Puyat JH, Qian H. Promoting evidence-based practice through a research training program for point-of-care clinicians. J Nurs Adm. 2015;45(1):14–20.

    Article  Google Scholar 

Download references

Acknowledgements

Not applicable.

Funding

This study received funding from the NWMH Research Seed Grant #014–2020.

Author information

Authors and Affiliations

  1. NorthWestern Mental Health, Royal Melbourne Hospital, Parkville, Australia

    Christine Migliorini, Megan Turville & JoAnne Bevilacqua

  2. Centre for Youth Mental Health, The University of Melbourne, Melbourne, Australia

    Caitlin McDowell

  3. Psychosocial Research Centre, Department of Psychiatry, The University of Melbourne, Melbourne, Australia

    Carol Harvey

Authors
  1. Christine Migliorini
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Caitlin McDowell
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Megan Turville
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. JoAnne Bevilacqua
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Carol Harvey
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the conception and design of the study. CM and CMcD collected the survey data. CM organized the published literature database, the survey data database and performed the statistical analyses. CM wrote the first draft of the manuscript. All authors contributed to the manuscript revision, read, and approved the submitted version.

Corresponding author

Correspondence to Christine Migliorini.

Ethics declarations

Ethics approval and consent to participate

All applicable institutional and governmental regulations concerning the ethical use of human volunteers and in accordance with the Declaration of Helsinki, were followed during the course of this research. Each participant received a plain language statement and opportunity to ask questions about the study. Informed consent was deemed given with voluntary completion of the survey. This project received full ethics approval from the Melbourne Health Ethics Committee: HREC/64816/MH-2020.

Consent for publication

Not applicable.

Competing interests

While all the authors were employed by the organisation at the time of the study, no-one was under any pressure to find any particular outcome.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Migliorini, C., McDowell, C., Turville, M. et al. Research capacity and culture in an Australian metropolitan public mental health service: scoping the skills and experience of social workers and occupational therapists. BMC Med Educ 22, 864 (2022). https://doi.org/10.1186/s12909-022-03936-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12909-022-03936-0

Keywords

BMC Medical Education

ISSN: 1472-6920

Contact us