Study design
The realist approach seeks to answer the question of "what works, why, and how, in what circumstances," with realism as the paradigm that lies between positivism and social constructivism [20, 21]. In this approach, we first establish a working hypothesis we wish to investigate in the context of an educational program. Next, from purposefully collected quantitative or qualitative data, we explore, test, and refine what mechanisms work under what conditions (context) and with what interventions (including opportunities or resources), from which we can describe explainable outcomes in an iterative fashion. This is described by the formula context + mechanism = outcome (following prior literature, the components are described as CMOs, an acronym for Context-Mechanism-Outcome) [22]. The key to validating CMOs is in the consistency and integration of the CMOs refined from the working hypotheses on the one hand with the data collected based on those hypotheses on the other [23]. This analysis process is described in detail in the Realist approach section below.
Curriculum of medical students in Japan
We briefly describe here the medical education curriculum in Japan to aid understanding of the learning context [24]. In Japan, students enter medical school after graduating from high school. The curriculum is presented in a six-year program that begins in April. In the first and second years, students begin lectures and practical training in basic medicine such as anatomy and physiology. In the third and fourth years, they study clinical medicine, including internal medicine and surgery. Students must pass a CBT (Computer-Based Testing) and OSCE (Objective Structured Clinical Examination) prior to clinical practice in order to be awarded the title of "student doctor" and be eligible for clinical practice. Clinical practice at the university in this study is conducted from January of the fourth year to November of the sixth year, for a total of 72 weeks. JH and colleagues in other universities previously developed a community-based medical education (CBME) curriculum within clinical practice that involves 16–19 students on rotations every four weeks, all spending one to two weeks in clinics or small hospitals to learn SDH in local communities [25]. This type of program is generally delivered as part of the general medicine component of clinical practice during the fourth or fifth year at a few Japanese universities. In the present study, we conducted a program evaluation focused on online community diagnosis in student-friendly hometowns at X university, which differs from prior literature [25].
General medicine in clinical practice
The general medicine in clinical practice component at X university was previously implemented from January of the fourth year to December of the fifth year. Clinical rotations consisted of 5–6 students every 2 weeks. Due to the corona virus epidemic, we changed to a full online clinical rotation from February to May 2020. Although clinical rotations at the university hospital resumed in August 2020, some online rotations remained. We therefore developed a community diagnosis program to make medical students aware of community health issues through online training in general medicine. Delivery of this online component and this study began in August 2020. JH is a general practice physician, has earned a PhD, and has published multiple qualitative studies, including studies involving realist approaches; TA is a general practice physician with 6 years of experience working in community hospitals and clinics, including qualitative research using reports; SF has been a physician for 40 years, has a PhD, and is program director of a general practice department; and JH is involved in education in collaboration with a general medicine department and has various opportunities to communicate with the other authors.
Learning theory
Community-based medical education has used experiential learning, communities of practice, and situated learning as theoretical frameworks [25]. However, few learning theories clearly explain the reality of the learning experience of medical students when learning about their own community [16]. Therefore, using variation theory, we considered core learning as discernment and variation in situations having time, space and social dimensions [26, 27]. We defined the learning about a community as “the important identification of things that occur under interaction between the learner and the learning object; and the variations that arise in this interaction are what take the essential characteristics of things into thought based on variation theory.” This helped in elucidating the learning experience of medical students.
Learning objectives
Two learning objectives were set.
Program schedule
During a one-hour zoom-based lecture on the first day of clinical practice, the significance of community diagnosis was presented, together with specific examples. Students could select their own area of residence or an area they were familiar with, such as their hometown, for community diagnosis. After one week, the medical students presented the community diagnosis they had researched, and received feedback from peers and an author (JH). On the final day of the two-week program, each student presented a structural report on community diagnosis which included their own feedback-based assumptions or opinions, and submitted a revised report. To avoid one researcher (JH) bias, TA interviewed the medical students on the last day of the program to obtain their feedback on their learning, in place of JH.
Student assessment
Summative assessment was implemented by the first author based on a rubric. This assessment included evidence from the midterm and final presentations, and structured reports of the community diagnosis. This rubric was shared with the medical students on the first day. The author participated in observations about the students' presentations on the midterm and final days of the program; their listening attitude during their peers' presentations; and how they responded to feedback from peers and the author. Formative assessment was also conducted throughout the midterm and final day presentations. Feedback was provided mainly on the validity of what was examined in the community diagnosis, the logic of their explanation of the issues in the community, and the relevance and feasibility of the action plan.
Setting and study participants
The setting of this study was the Department of General Medicine at X University in Tokyo, Japan, from August 2020 to December 2021. The academic year of Japanese universities begins in April. Our university starts clinical practice in January of the 4th year, and the subsequent 24-month period until December of the 5th year is the basic clinical practice period. The rate of female medical students in Japan is reported to be low; in 2018, only 21.9% of physicians were women [28]. Study participants included 4th- and 5th-year medical students who participated in a general medicine clinical clerkship in the basic clinical practice program.
Realist approach
The analytical method was carried out according to the four steps of Pawson [22]. First, we formulated a working hypothesis that we aimed to investigate in the program. Working hypotheses about the program were formulated using deductive and inductive methods [22]. On this basis, using purposefully collected, mainly qualitative data, we explored, tested, and refined what mechanisms worked, under what conditions (context), with what interventions (including opportunities or resources), and from which explainable outcomes could be described and verified iteratively. This heuristic is to remind us to think of realistic evaluation in terms of constructs, and not as formulas, as in mathematics.
Generating a working hypothesis about the components of CMOs
Context is the condition under which the program is introduced, and is related to the mechanism, which considers "under what circumstances" the program will work. Mechanism is the process of how an individual interprets and acts on an intervention. Outcomes are how a series of outcomes result in an effect or change. We began by hypothesizing about potential mechanisms. Astbury and Leeuw reported that a clear distinction should be made between mechanisms and program interventions [29]. For example, an outcome might be an increase in the learners’ knowledge or readiness resulting from some mechanism arising from some educational intervention. In other words, the mechanism is an explanatory model that can be described in relation to multiple context and educational intervention variables. Dalkin et al. used the formula presented by Pawson et al., Context + Mechanism = Outcome [22]. With regard to the mechanism of the program's intervention and the mechanism as an inference caused by the program, Dalkin et al. proposed a method of describing the context + mechanism (intervention) → mechanism (inference) = outcome [30]. With this representation, mechanisms are explained by both concrete facts and interpretive inferences. The contexts in which program interventions are introduced vary, as do the emerged patterns of mechanisms (inferences) and activated outcomes. Thus, CMOs describe multiple patterns of how the various components of a program harmonize and integrate. Here, to describe these patterns, we utilized the observational evaluation and community diagnostic structural reports submitted by the students on the final day of the program.
Observation and verification
The working hypothesis was tested by collecting data on CMOs. At the end of the fiscal year, program evaluation was conducted based on the medical students' verbal and written reflections and reports. The authors confirmed whether a series of CMOs could comprehensively explain the learning patterns of the community diagnostic program as an evaluation. When testing for consistency and integration, we did not focus solely on a single outcome, but examined whether CMOs as learning patterns of community diagnosis under various combinations of learner outcomes with contexts and mechanisms allowed readers to transfer the findings to other settings.
CMOs clarification
This process of validation and refinement resulted in a pattern of CMOs, namely a series of mechanisms and learner outcomes which could explain the complex learning process based on several contexts. Continued validation and refinement of the CMO patterns was conducted using data collected over the two years [31].
Ethical approval and consent to participate
This study was approved by the Ethics Committee of Keio University (approval number: 20211157), and was performed in accordance with the Declaration of Helsinki. All participants were given the opportunity to opt out in the web-page of the medical education center at Keio University. Informed consent was obtained from all participants.