Establishing entrustable professional activities for psychiatry residents in China

Abstract

Purpose

The authors established entrustable professional activities for psychiatry residents in China.

Methods

The authors conducted a literature research and two expert consultation rounds following the Delphi method in 2022 to screen and optimize entrustable professional activities for psychiatry residents.

Results

The effective questionnaire recovery rate in the two consultation rounds was 100% (44/44). The expert authority coefficients of the first and second consultation rounds were 0.861 and 0.881, respectively. The Kendall harmony coefficients of the first and second expert consultation rounds were 0.279 (χ² = 405.43, P < .001) and 0.389 (χ² = 3456.83, P < .001), respectively. The arithmetic means of the various indicators’ evaluation results in the two consultation rounds ranged between 3.61 and 4.93, and the full score rates were between 13.6% and 93.2%. The authors established 17 entrustable professional activities for psychiatry residents and their contents with phase-based modularization and formulated the entrustable level of each at various stages.

Conclusions

Combined with standardized psychiatry training characteristics, the authors preliminarily established phase-specific and modular entrustable professional activities for psychiatry residents. The formulated entrustable professional activities are suitable for the practice and clinical environment of standardized psychiatry training in China. The devised system has good observability and measurability and provides a simple and feasible competency evaluation method for standardized psychiatry resident training.

Introduction

In 2010, the Commission on the Education of Health Professionals for the 21st century proposed the third-generation reform of medical education, emphasizing the competency-based medical education (CBME) training mode [1]. Since 2014, China has implemented a nationwide standardized resident training system that is gradually maturing. Medical education professionals in China have started to realize the importance of cultivating the competence of clinicians. Some scholars used behavioral event interviews, expert consultations, and questionnaire surveys to build a competency model suitable for clinicians in China [2]. However, its implementation and evaluation were too cumbersome, resulting in weakened clinical diagnosis and treatment integrity. The trainees cannot effectively integrate various abilities into clinical work, and such models show difficulties in achieving the expected practical effects [3, 4].

In 2005, Professor ten Cate of the Medical Center of Utrecht University in the Netherlands first introduced the concept of entrustable professional activities (EPAs) [5]. EPAs emphasize medical behavior integrity and direct observations based on the observable, measurable, and executable clinical working environment [6]. They have been widely adopted internationally and are considered a good CBME evaluation method [7]. At present, EPAs are implemented in the United States, Australia, New Zealand, the Netherlands, Canada, and other countries and are included in several postgraduate medical education courses, including in pediatrics, internal medicine, family practice, psychiatry, obstetrics and gynecology, and nursing [8]. Papers reported mainly EPA development and/or EPA implementation studies whereas EPA assessment studies were less frequent [9], including three studies on constructing psychiatry EPAs [10,11,12], six on implementing EPAs [13,14,15,16,17,18], and one on constructing and implementing EPAs [19]. A relatively complete framework and implementation experience in psychiatry EPAs are available only in the United States, Australia, New Zealand, and Canada [18,19,20]. As a theoretical framework under exploration, many aspects of psychiatry EPAs are still worth investigating and improving. An EPAs evaluation system suitable for China’s national conditions has not been established. Based on the current standardized resident training situation at Huilongguan Clinical Medical College of Peking University, we explored establishing EPAs for competency evaluation of psychiatry residents on the basis of twelve tips to develop entrustable professional activities [21], providing some reference for daily and phase-based psychiatry competency evaluations during postgraduate medical education.

Methods

Research methods

We conducted a literature research and two expert letter consultation rounds following the Delphi method [22] to establish entrustable professional activities for psychiatry residents in China. The study was performed in accordance with the latest version of the Declaration of Helsinki and approved by the ethics committee of Beijing Huilongguan Hospital.

Composition of the research group

The research group was composed of ten people, one with a senior professional title, four with a deputy senior professional title, and five with an intermediate professional title. One had a doctoral degree and nine had a master’s degree. Specifically, the team included the director of the education department, the ward director, and eight senior clinical front-line teachers, all with over ten years of teaching experience. The group members are all psychiatrists who have undergone two rounds of training in the early stages and are familiar with the EPA concept. The main team responsibilities included literature research, formulating the consultation form, expert selection, preparing an online Questionnaire Star version, distributing and recovering the consultation forms, index screening and revision, statistically analyzing the data, and writing this manuscript.

Literature research

We searched five electronic databases (Medline, CINAHL, Scopus, PsychINFO, and Web of Science) in January 2020 for publications from 2011 to 2020 using the following keywords: “entrustability and psychiatry” and “entrustable professional activit* and psychiatry,” retrieving 34 publications. We screened them for suitability based on their titles and abstracts, searching for articles on “confidence in professional behavior (EPA).” Twenty-eight papers remained after the first screening step. We assessed their suitability based on a full-text review, searching for publications with specific EPAs. Ten articles remained after this screening(Fig. 1). We limited the search to articles written in English, the type of literature is narrative.

Fig. 1

PRISMA protocol

The ten articles contained 61 EPAs, including duplicates. Following the 13 EPAs mentioned in a published guide from 2020 [19], we removed duplicates, summarized, and merged items, leading to the selection of 12 EPAs.

Preparing the expert inquiry form

Based on the literature review, referring to the 13 EPAs mentioned by the Association of American Medical Colleges Core Entrustable Professional Activities for Entering Residency, the 20 EPAs suggested by the Royal College of Physicians and Surgeons of Canada, and the rotation mode of standardized psychiatry training detailed in the Rules for Standardized Resident Training in Beijing (2013 version) [23], we developed a phase-based modular EPAs suitable for the three rotation stages, that is, the general, specialized, and mobile stages. The first draft expert consultation form for EPAs of psychiatry residents was formed after several discussion rounds within the research group and included 19 EPA indicators.

Selecting the consulting experts

We adopted an objective sampling method to select 44 experts from nine Grade-A psychiatry departments in Beijing, Tianjin, Shanxi, Heilongjiang, Henan, Zhejiang, Sichuan, and Yunnan Provinces. The expert inclusion criteria were: (i) bachelor’s degree or higher; (ii) a deputy senior professional title or higher; (iii) over ten years of clinical work and teaching experience; (iv) participated in the standardized training of psychiatry residents; and (v) able to complete at least two expert consultation rounds. All experts provided their informed consent for participation in this study.

Implementing the expert consultation

We distributed the consultation questionnaires as Questionnaire Star and conducted two consultation rounds, 2 to 3 weeks apart. In the first consultation round, The first part of the questionnaire is to introduce to the experts EPA concepts, the study background information, its purpose, and a preliminarily constructed evaluation index system and asked them to score the importance and familiarity levels of the indicators. We used a Likert 5-level scoring method, assigning scores from one point (very unimportant/unfamiliar) to five points (very important/familiar). The questionnaire also collected basic information on the experts, their degree of familiarity with the subject, and the basis for their judgment. We sorted, summarized, and statistically analyzed the returned expert opinions of the first round to generate the form for the second round of expert consultation, which was based on the results of the first round. We asked experts to evaluate the observability, authenticity, multiple competencies, effectiveness, repeatability, and feasibility of the indicators. Experts graded the entrustable level that should be achieved at the end of each stage and at the end of the course, referring to the five-level entrustable assessment scale [5] and Twelve tips to develop entrustable professional activities [20] proposed by ten Cate. Level 1 is “no permission to execute the EPA,” indicating that the resident’s knowledge and skills are not enough to implement the EPA and must be completed by the supervisor. Level 2 is “execute EPA under full supervision,” indicating that the resident could execute the EPA but required full supervision and guidance. Level 3 is “execute EPA under occasional guidance” indicating that the resident could execute the EPA, but the supervisor needs to respond to the resident’s needs, provide timely help, and review the outcomes. Level 4 is “execute EPA independently, without supervision,” indicating that the resident can work independently, knows the risks, and can perform safely without supervision. Level 5 is “guide others to execute EPA,” indicating that the resident can supervise others when implementing the EPA. We summarized and statistically analyzed the returned expert opinions of the second round. As the statistical analysis results showed consistent feedback results after the two expert consultation rounds, the consultation ended.

Index screening criteria

We calculated the mean and coefficient of variation (CV) of each index following a expert opinions. The index screening criteria were importance score > 3.75 and CV < 0.25. The research group revised the EPAs following the questionnaire results and expert opinions.

Expert enthusiasm, authority, coordination, and concentration

The effective questionnaire recovery rate expressed the expert enthusiasm degree.

The authority coefficient, determined by the experts’ judgment basis and familiarity degree coefficients, expressed the expert authority degree. We used the following formula: authority coefficient = (familiarity degree coefficient + judgment basis coefficient) / 2 [24]. Values above 0.7 indicated good expert authority.

The CV and Kendall harmony coefficient expressed the degree of coordination among the expert opinions. The smaller the CV, the more consistent the expert opinions. The greater the Kendall harmony coefficient, the better the coordination among the expert opinions.

The arithmetic mean and the full score rate reflected the concentration of expert opinions.

Statistical methods

For statistical analysis, we used IBM SPSS Statistics for Windows, Version 26.0 (IBM Corp., Armonk, NY, USA). The measurement data are expressed as means (standard deviations), and counting data are expressed as frequencies and percentages. The mean, full score ratio, and CV described the index scores. We considered differences with P < .01 to be statistically significant.

Results

Basic experts’ information

We consulted 44 experts. Their mean (standard deviation) age was 40.6 (5.3) years, and they had engaged in standardized resident training for 10.9 (4.6) years. The experts included three with a PhD degree (6.8%), 27 with a master’s degree (61.4%), and 14 with a bachelor’s degree (31.8%). Fifteen (34.1%) had full senior professional titles, and 29 (65.9%) had deputy senior professional titles. They included five residential training directors, nine teaching and research department directors, six teaching secretaries, five clinical department directors, eight ward directors, and 11 clinical teachers.

Expert enthusiasm, authority, coordination, and concentration

The effective questionnaire recovery rate in the two consultation rounds was 100% (44/44), indicating that the experts paid attention and had a high degree of enthusiasm for this study.

The expert authority coefficients of the first and second consultation rounds were 0.861 and 0.881, respectively. The expert familiarity and judgment basis coefficients were 0.818 and 0.904 in the first round and 0.847 and 0.915 in the second round, respectively.

In the two rounds of consultation in this study, except for one index, the coefficient of variation of the evaluation results of each index was between 0.05 and 0.24. The Kendall harmony coefficients of the first and second expert consultation rounds were 0.279 (χ² = 405.43, P < .001) and 0.389 (χ² = 3456.83, P < .001), respectively, indicating that the coordination degree of expert opinions was good.

The arithmetic means of the various indicators’ evaluation results in the two consultation rounds ranged between 3.61 and 4.93, and the full score rates were between 13.6% and 93.2%.

The establishment and validation of EPAs for psychiatric residents

Following the experts’ opinions, we deleted or modified some contents of the various EPAs, resulting in 62 content descriptions. After the first round of expert consultation, we deleted 15 content descriptions and modified ten. After the second round of expert consultation, we revised four content descriptions; the final content descriptions are presented in supplementary Table S1. See Table 1 for competencies [25] related to the 17 EPAs.

Table 1 Evaluation index of entrustable professional activities of psychiatric residents at stages

The importance, familiarity, observability, authenticity, multiple competencies, effectiveness, repeatability, and feasibility values were all greater than 3.5. Except for the observability of “guiding beginners,” the indicators’ CV values were all smaller than 0.25, indicating that the indicators’ setting was reasonable (Table 2).

Table 2 Results of the expert consultation on the entrustable professional activity indexes of psychiatry residents

Expected levels of supervision on the EPA for psychiatry residents at their various training stages

After the two consultation rounds, the expected levels of supervision on the EPAs at the various stages were similar. The expected levels of supervision CVs of the EPA indicators at the end of each stage and at graduation were all under 0.25 (Table 3).

Table 3 Expected entrustable levels of entrustable professional activities of psychiatry residents at various stages

Discussion

The EPA index for psychiatry residents is scientifically valid

In methodology terms, we adopted a literature review and the Delphi expert consultation method to formulate and select indicators following scientific standards. We invited experts from nine psychiatric hospitals across the country that included experts in teaching management, teaching research, and clinical teaching. Most experts had a graduate degree, senior title, and rich experience in resident training, teaching, and management. The experts consulted in this study showed good authority and enthusiasm. The CV and expert coordination coefficient of all EPA indicators were good, making the research results scientifically valid.

Phase-based modular characteristics of the EPAs in the psychiatry department

Relatively few studies on EPAs have been conducted in China; exceptions include, however, the research on constructing resident EPAs by Qi et al. of Peking University Hospital [26] and an obstetrics and gynecology EPA study [27]. The former flatly disassembled clinical behaviors into 17 key behaviors, while the latter listed key technical operations in obstetrics and gynecology as EPA indicators. However, the above-mentioned local EPA evaluation systems could not meet the particular training objective requirements at various stages and the standardized psychiatry training rules.

We adopted phase-based modular thinking to match the standardized training characteristics in the psychiatry department when constructing the EPA indicators, making them suitable for the clinical environment. The phase-based modularization also makes them observable, repeatable, and feasible. The formulated EPAs are more suitable for the specific characteristics of psychiatry specialty training than the clinical resident EPAs, and could be used when exploring EPAs for other specialty training programs. Unlike the standardized training of clinical residents, the psychiatry department EPAs were constructed into five modules, including clinical psychiatry communication, quantitative psychological assessment and judgment, using psychotherapy methods, evaluation, admission, and treatment of psychiatry outpatients, and modified electroconvulsive therapy treatment, highlighting the characteristics of the specialty. Psychiatrists serve patients with mental diseases, who face obstacles to understanding and affection. Establishing a treatment relationship with patients, reflecting their understanding, providing humanistic care, and communicating effectively are essential skills of psychiatric clinical work [28]. The diagnosis and treatment at the psychiatry department differ from those at other clinical medicine departments. They rely more on detailed medical history, professional mental examination and consultation, and necessary psychological examination scales. Psychiatry lacks the gold standard objective examination of internal medicine and surgery, making diagnosis and evaluation more varied and subjective [29]. Therefore, using psychological examination scales is particularly important and indispensable. Psychiatry residents should master selecting and interpreting common psychological scales. The particularity of psychiatry compared with other clinical disciplines is highlighted by its theoretical basis and clinical practice. The discipline’s nature originated from two equally important “genes,” natural and social sciences. Therefore, some scholars believe that psychiatrists should have certain psychological treatment abilities in addition to being able to perform biomedical diagnosis and treatment, so that they can learn to walk in both realms [30]. Another prominent feature of psychiatry is risks other than the disease itself, including behaviors that endanger society, other people, or the patient through self-inflicted injuries and suicidal behaviors. The mental health law defines the principles of involuntary hospitalization for six serious mental disease types, establishes a mental health monitoring network, and implements a reporting system for serious mental disorders [31]. Therefore, psychiatrists working at the psychiatry outpatient department should first assess and judge the risks of impulsive or self-inflicted injury and suicidal behaviors and treat these according to the mental health law provisions, in addition to undertaking routine diagnostic and therapeutic work. The EPAs under the above five modules obtained high scores in the expert consultation importance evaluation and had good consistency. Evidently, the EPAs constructed in this study are suitable for the overall objectives and discipline characteristics of standardized training of psychiatry residents, making them highly practical.

The localized nature of the EPA indicators for the psychiatry department

Boyce et al. [10] developed four EPAs for first-year residents using a questionnaire survey, making it an exploratory study for psychiatry EPA construction. Young et al. [12] did not consider the impact of the clinical environment and disease category and severity on the EPA indicators when constructing their suggested 13 EPAs. However, they pointed out that these indicators were too broad. Weiss et al. [11] constructed EPAs by years, showing the phased and dynamic development of clinical EPAs.

The standardized training developed by Royal College of Canada is divided into four stages, including Transition to Discipline, Foundations, Core, and Transition to Practice [20], which is equivalent to the second stage of resident physician training in China, and the duration is also 2 years. Our research is currently focused on the standardized training phase for resident physicians, which is the rotation phase of the first three years. Therefore, there are still certain differences in the phased training at home and abroad.

The training of resident psychiatrists in Beijing is conducted through rotation between psychiatry and related departments. The rotation is divided into three stages: general department stage (9 months), specialized department stage (24 months), and mobile stage (3 months). The rotation of related departments includes Neurology, emergency department or intensive care unit, and cardiovascular medicine department. The rotation of psychiatry includes psychiatric intensive care unit, psychiatric mild illness ward or clinical psychological department or open ward, community, rehabilitation department or geriatric department, or pediatric department, and psychiatric outpatient and emergency department, The mobile phase includes clinical research or other rotating departments.

Therefore, simply dividing by years or establishing EPAs based solely on foreign phased training systems cannot meet the standardized training requirements for resident physicians in China.

Our research and the standardized psychiatry training characteristics led to the construction of EPAs in a phase-based and modular manner, explaining the final 17 EPA indicators. Therefore, the proposed EPA set is suitable for the practice and clinical environment of standardized psychiatry training in China and has good observability and measurability.

The limitations of our research

In this study, we selected experts from nine psychiatric hospitals in various regions of China. Based on the detailed psychiatry training rules in Beijing, this study represents the current mode and characteristics of standardized psychiatry training. However, the consultation scope should be expanded further with a view to building and implementing an EPA evaluation system for psychiatry departments nationwide.

Our research has certain limitations. Firstly, due to participating in the Delphi expert consultation method, all mentors in the field of psychiatry were selected, and the subjects of consultation did not cover resident physicians, lacking a perspective as a guide, which may lead to a certain bias in the formulation of EPAs. Secondly, due to factors such as time and the impact of the epidemic, it was not possible to invite international experts to participate in this study. Thirdly, international standard for evaluating EPAs were not used.

Future research directions on psychiatry EPAs

We propose these future research directions: (i) compare the EPA and existing evaluation systems to further optimize the EPA evaluation system; (ii) focus on implementing and applying the EPAs in standardized psychiatry training; (iii) training courses should be developed following the evaluation indicators and the training-stage objectives, evaluating the indicators in evaluator and resident groups; and (iv) for educators, analyze and compare the EPA evaluation system with the existing Objective Structured Clinical Examination structured assessment system and analyze the relevant factors to provide some reference for further exploration of the training approach to combine standardization, homogeneity, and individualization.

Conclusion

We used a literature review and the Delphi method to establish a phase-based modular EPA index for psychiatry residents at their various training stages. These psychiatry department EPAs provide a simple and feasible competency evaluation method for resident training and clarify the training objectives for residents, clinical teachers, and management personnel. We intend to continue exploring quantitative research on the effectiveness of the psychiatry EPA evaluation system, with a view to establishing a standardized scheme for their implementation and evaluation.