Evaluation of FamMed essentials: a blended-learning program for capacity building of general practitioners in Pakistan

Abstract

Background

To provide access to primary care and universal health coverage, Pakistan requires 60,000 trained family physicians by 2030. At present, most primary care is provided by general practitioners (GPs) who do not have any post-graduate training. Empowering GPs through competency–based programs, that strengthen their knowledge and skills, may be a cost-effective strategy for improving healthcare quality. We describe the development and evaluation of FamMed Essentials, a modular, blended-learning program to improve clinical knowledge and skills of GPs.

Methods

This is a mixed method study. We used the CIPP (content, input, process and product) framework for course development and evaluation. We describe the steps used in content development, strategies for teaching and assessments, and evaluation of strengths and weaknesses of the program. In depth focus group discussions were conducted to gather insight on participants’ and faculty’s perceptions regarding the program’s effectiveness.

Results

Of the 137 participants who have completed the program, 72% were women and 49% had been practicing for more than five years. We saw a significant improvement in knowledge across all modules (p = < 0.001) and perceived confidence in clinical skills (p = < 0.001). An objective assessment showed participants’ competence in patient management. Participants reported a high level of satisfaction (4.4 ± 0.83 on a 5-point Likert Scale). Focus group discussions revealed a positive impact on clinical practice. Flexibility and use of different teaching and learning strategies were additional strengths. In addition, participants reported an interest in further training. Power outages were highlighted as a major challenge.

Conclusion

In resource-constrained health systems, a modular, blended-learning, competency-based program is helpful to upgrade GPs knowledge without impacting their busy schedules. Accreditation of such programs and provision of a career trajectory for the trained GPs are pivotal to expansion of such initiatives.

Background

The World Health Organization (WHO) attributes competent Family Physicians (cFP) as a key factor in positively influencing health indicators of a country [1]. Pakistan requires 60,000 cFP by 2030 to improve the primary health system for its population of 220 million [2]. Presently, medical graduates are licensed to practice independently as a general practitioner (GP) after completing a one-year internship without any training in Family Medicine. Family Medicine remains a nascent discipline, and to date, there are only eleven institutions across the country that offer the four-year post-graduate structured training program, after which residents can sit for the FCPS (Fellow of College of Physicians and Surgeons of Pakistan) qualifying exam and be considered a Family Medicine specialist [3]. Moreover, seven institutions across Pakistan offer a two-year structured training, that allows trainees to sit for the MCPS (Member of College of Physicians and Surgeons Pakistan) qualifying examination to be considered a Family Physician [4]. Thus, there aren’t enough programs to train the GP workforce that is required for the country’s healthcare system. As the country moves towards a Family Practice approach to Universal Health Coverage [1], the WHO proposed a one-year diploma in Family Medicine for capacity building of the existing GPs as a cost-effective strategy [5,6,7]. Recognizing this need [3], nine institutions have created one-year programs that have been endorsed by WHO as Regional Diplomas in Family Medicine [4].

The Department of Family Medicine at Aga Khan University developed “FamMed Essentials”, a one-year program, that uses a blended- learning approach to help GPs who are interested in honing their clinical skills.

Experiential and situated learning theories emphasize that learning occurs best when learners participate in authentic activities at the workplace [8]. Technology enhanced learning allows a learning experience with relatively limited resources [9]. However, it is still limited in development of some sub-competencies where physical contact that use human senses are required (e.g., physical examination, diagnostic & therapeutic procedures, etc.). Therefore, a program with a mix/blend of online and onsite learning is a practical solution. Blended learning (BL) is a growing approach for acquiring clinical skills, enabling interaction between participants, facilitators, and resources [10]. It allows continuous learning, while overcoming time and space constraints, and has the potential to improve clinical competencies such as clinical reasoning, history taking, and reflective thinking skills [11].

The goal of the program is to help GPs learn and apply core concepts of Family Medicine in their practices. This includes enabling them to provide evidence-based management of common health issues; offer comprehensive care incorporating preventive and health promotive strategies in patients’ health plans; provide contextual care recognizing the role of social determinants of health on patients’ health and wellbeing; and provide timely and appropriate referrals when required.

In this manuscript, we evaluate the effectiveness of the FamMed Essentials program, using the Context, Input Process and Product (CIPP) framework [12], and identify potential areas for improvement to enhance the program’s outcomes related to the needs of GPs in the region.

Methods

Study design

This was a mixed-method study utilizing the CIPP framework [13] in the development and evaluation of the program. We chose the CIPP framework as it evaluates the various stages of program development including, understanding the needs/goals, assessing designs/resources, evaluating implementation, and measuring outcomes/impacts. CIPP combines formative and summative assessment, thus aiding implementation and highlighting improvement areas of the program. The contents of the CIPP framework are outlined as below.

Context

A context evaluation includes a needs assessment, aligning goals with needs, anticipating barriers and identifying assets and opportunities [13, 14]. Prior to the program a literature search confirmed shortage of competent family physicians [15]. A survey of GPs [16] revealed lack of confidence in managing common outpatient conditions amongst 50% of the respondents. In addition, 62% GPs reported comfort in using technology; and 61% were interested in a flexible, blended-learning program. This evaluation facilitated funding for program development.

Input

The process of input and subsequent evaluation addresses whether the targeted needs are achieved by an appropriate plan [14]. Family Medicine faculty met weekly to develop the program’s learning outcomes, objectives, assessments and course content. They reviewed curricula of national and international programs such as Membership of the College of Physicians and Surgeons Pakistan (MCPS); [17] Membership of the Royal College of General Practitioners UK (MRCGP Int) [18]; College of Family Physicians of Canada (CFPC); [19] the Royal Australian College of General Practitioners (RACGP); [20] American Academy of Family Physicians (AAFP); [21] and World Health Organization (WHO) [22]. Topics were evaluated based on national burden of disease [1, 23]. The Delphi method was used to reach consensus on the final topic list [24].

The program content was organized into five modules: Pediatric and Adolescent Health, Reproductive Health, Non-Communicable Diseases (NCDs), Common infections and Health of special populations (palliative care, and geriatrics), and Beyond infections and NCDs (including common musculoskeletal, gastroenterological, and surgical problems). Faculty were trained in online and blending learning methods. The multimedia development team created engaging graphical voice-overpower Point presentations. The program was hosted on a virtual learning environment (VLE) platform. It was evaluated by the Department of Continuing Professional Education (DCPE) and accredited for 310 Accreditation Council of Continuing Medical Education (ACCME) credits. It was also endorsed by WHO-EMRO as a Regional Diploma in Family Medicine.

Process

The process describes if the program was effectively implemented and evaluates its strengths and weaknesses which help with further refinement of a program [25].

FamMed Essentials is promoted through university communication channels and social media platforms (Facebook page, Twitter, Linked in) and various WhatsApp groups.

Participants feedback is collected through a questionnaire to refine the program based on strengths and weaknesses.

Product

The product evaluation component in CIPP closely resembles a traditional “summative” program evaluation. It assesses the program outcomes, both positive and negative [12]. Table 1 shows the components used for evaluation of each element of the framework.

Table 1 Components in evaluation of each element of CIPP framework

Program participants and eligibility

The program has been specifically designed for GPs practicing in Pakistan who hold a valid license to practice. In addition, physicians trained in Pakistan, but residing abroad also showed interest in improving their clinical knowledge and skills and were included. Furthermore, the University is a strong proponent of empowering nurses, therefore, we also welcomed nursing students pursuing a master’s degree in the clinical stream and interested in becoming nurse practitioners into the program.

Setting

FamMed Essentials utilizes a modular and blended learning approach, allowing participants to access the program content through the Aga Khan University (AKU) virtual learning Environment (VLE). The onsite component of the program is conducted at the state-of-the-art Center for Innovation in Medical Education (CIME) at the Aga Khan University in Karachi, Pakistan.

Educational/teaching intervention

FamMed Essentials is a flexible one-year modular program designed for busy practitioners. The overview of the program and the structure of each module can be seen in Fig. 1.

Fig. 1

Overview of FamMed essentials blending learning program

Each module begins with a mandatory pre-module test consisting of multiple-choice questions (MCQs) to assess prior knowledge and identify knowledge gaps. The module content includes videos covering 15–16 topics, featuring text, figures, algorithms and narrative audios from Family Medicine faculty. These videos also include embedded quizzes to promote learner’s engagement [26]. Additional resources and clinical skills videos are also available (asynchronous independent activity). Learners can post their questions on a visual board using a Padlet® wall, which automatically sends an email to the module coordinator. Asynchronous collaborative activities include a monthly discussion forum. Monthly synchronous sessions called “Sunday Chai and Chat” (tea in local language) with faculty provide an opportunity to discuss these questions, reinforce key concepts through interactive quizzes, role plays and engage in small-group case-based discussions in breakout rooms. At the end of each module participants must complete a mandatory post-module test before accessing the next module. After completing five modules, participants attend a three-day onsite clinical skills session, to learn essential skills on simulators and simulated patients (e.g. breast and pelvic examinations, laceration repairs and musculoskeletal examinations).

This is followed by a 14-station Task Oriented Assessment of Clinical skills (TOACS) exam with feedback and sign off. It assesses history taking, physical examination, diagnosis management and/or counselling for outpatient conditions. Evaluation is conducted using a standardized marking grid.

Data collection tools and statistical analysis

We collected data for various parameters in the program using the following instruments:

• Improvement in knowledge was assessed through comparing the scores of pre- and post- module tests obtained from the VLE using paired t-test. A p-value of < 0.005 was considered statistically significant. We used STATA v. 16.0 for quantitative analysis.

• Level of confidence in clinical skills were evaluated through a feedback form completed by participants at the conclusion of the onsite clinical skills session.

• Clinical management skills were assessed objectively through a TOACS exam.

• All participants who completed the program were invited to fill in an online survey using Survey Monkey®. The survey included questions regarding the acceptability and perceived utility of each component of the program.

• To obtain an in-depth understanding of participants’ experiences, four focus group discussions (FGDs) were conducted with 12–15 participants in each group. Each FGD followed a question script and lasted 45–60 min. A separate FGD was conducted with the faculty who helped develop and run the program. FGDs were audio recorded and transcribed and checked for accuracy. Two members of the evaluation team (HF and SS) decoded and sorted content by themes. To develop consensus three members (HF, SS and UIK) crossmatched the themes.

• Sustainability evaluation was based on the operational budgetary requirements.

Ethics approval

We received approval from the University Ethics Review Committee (ERC: 2021-6431-2013) for conducting the evaluation. We obtained informed consent from all participants in the study during the focus-group discussions and whilst completing the online survey.

Results

Participants’ demographics

Five cohorts have completed FamMed Essentials. The initial three groups accommodated 20 participants each, with later cohorts expanding due to increased faculty confidence. During this time, 214 participants registered in the program and 28 (13%) formally withdrew. Of the 186 participants, 137 (74%) completed the program and 49 (26%) are at various stages of completion. Thus, this evaluation focuses on the 137 participants who have completed the program.

The majority of participants were women (n:99, 72%), reflecting the trend in Pakistani medical school admissions [27]. While the majority were from Karachi (n:94, 69%), participations also came from rural areas (n:11, 8%), and other urban cities (n:20, 14%). In addition, few participants were residing in the Middle East (n:8, 6%) and Canada (n:4, 3%). About half (n:67, 48%) were in practice for ≥ 5 years; (n:62, 45%) for < 5 years; and 8 (6%) had never practiced. A majority (n:102, 75%) were practicing in different settings including hospitals (n: 53, 39%); private clinics (n:15, 11%); and government clinics (n: 8, 6%). In addition, 26 (19%) were working in non-clinical settings; and 35 (25%) were unemployed.

Program outcomes

Quantitative analysis

Improvement in clinical knowledge

There was significant improvement in clinical knowledge in all modules, with the largest increase in scores noted in women’s health and management of non-communicable diseases (Table 2).

Table 2 Knowledge Change and Clinical Skills Confidence Assessment

Improvement in clinical skills

There was a significant improvement in participants’ perceived confidence in performing clinical skills taught during the onsite clinical skills session (Table 2).

Competence in management of common clinical issues

All participants completed a Task Oriented Assessment of Clinical Skills exam. Table 3 shows the average score and the pass rate at each station. The list of stations is longer than 14 as some stations were changed for each cohort. While most participants performed well on management of NCDs, they scored lower in management of pediatric conditions such as malnutrition. Participants improved in managing NCDs as they encounter more adult cases in their practices [16]. We recognize the limited pediatric teaching available at government and private medical institutions and are working on enhancing exposure to pediatric cases within the program.

Table 3 Results of Task Oriented Assessment of Clinical Skills

End of Program survey

Ninety-five participants completed the online survey evaluating all program components (Table 4). Overall, the participants reported high satisfaction with most components. The lowest scores were for the discussion fora (4.1 ± 0.9 on a 5-point Likert scale). Participants who were in clinical practice had higher satisfaction with components requiring application of clinical knowledge.

Table 4 Participants’ satisfaction with program and its components

Qualitative analysis

The analysis of participants’ FGDs yielded four main themes that are presented in Table 5. The focus group with faculty revealed both positive opinions and challenges. Most faculty felt that developing content and skills’ sessions helped update their own knowledge and skills. In addition, they recognized their own professional development through the training in blended teaching and learning skills, that they are now using in other classes. Faculty felt that the diverse learning needs of the participants were addressed by using multiple teaching and learning strategies (e.g. role plays, discussions, and use of independent learning resources). All faculty wanted to continue being part of FamMed Essentials.

Table 5 Participants’ perceptions and relevant quotes related to themes

Program sustainability

The program received support from the University Provost and Dean of the Medical College for development. Pfizer pharmaceuticals provided an educational grant for scholarship for 23 participants.

The program is currently offered twice a year with up to 100 participants per cohort, covering cost of faculty time and administrative needs. There is interest from other low-and-middle income countries (LMICs) to expand the program.

Discussion

Our evaluation findings demonstrate that FamMed Essentials is a sustainable and adaptable capacity building model for GPs in Pakistan and the region. Both quantitative and qualitative data indicate that the blended learning approach enhances clinical knowledge, clinical skills, and self-reflection. Participants also report improved patient-centeredness and communication skills. The program is valued for professional growth with participants intending to sit for qualifying examinations. FamMed Essentials fills a crucial gap in resource -constrained health systems by increasing competence in managing common outpatient conditions. Other programs in the country and the region have reported strengths and challenges. The one-year diploma offered through Khyber Medical University reports enhanced evidence-based practice in participants; and identified time constraints and technology availability as barriers [28]. Similarly, participants at a two-year, in-person diploma offered through Ziauddin University (a private institution) report improvement in problem solving skills, but identify taking time off from work as a major challenge [29]. The 12- month regional diploma offered by the American University of Beirut reports improvement in clinical skills of general practitioners. The authors identified accreditation/recognition by local regulatory bodies and career integration as a major challenge [30]. An evaluation of the diploma in Saudi Arabia reported development of proficient consultation skills and patient-centered care among the participants [31]. Other countries such as Iran have used short diplomas to strengthen the primary health care infrastructure and bridge health care delivery gaps [32].

Several challenges were identified by participants of FamMed Essentials, including frequent internet disruptions, which are common in LMICs. Other studies have recognized that connectivity issues hinder learning and the need to address this to enhance learning experience [33]. The flexibility of our program allows access that supports learning. Participants reported less confidence in managing pediatric cases due to insufficient pediatric exposure during their medical education. To address this, we have integrated additional child health topics and skills into the curriculum. We also noted that participants in clinical practice reported a higher satisfaction with components requiring application of clinical knowledge. A clinical attachment program may help aspiring practitioners in enhancing their skills.

Our study found real-time discussions with experts, videos and skills development sessions were the most effective methods. Other studies highlight the importance of interactions between learners and teachers in online courses [34,35,36]. The challenge noted by our participants regarding the use of discussion forums is also consistent with existing research [37]. There is a need to address the readiness of both faculty and students to engage in asynchronous discussions when designing online courses.

As the program is new, we have not yet observed participants’ career trajectory. In addition, we were unable to observe pre- and post- program patient care in the physicians’ workplace. This is an important area to ensure the application of knowledge and skills in clinical practice.

Conclusion

The evaluation of FamMed Essentials demonstrates the effectiveness of blended learning in enhancing GPs’ clinical skills in resource limited settings, thereby enhancing overall primary care. Accreditation of these programs and government support in providing a career trajectory to these trained GPs are pivotal to expansion of such initiatives. Lastly the CIPP framework proved crucial for this assessment and is recommended for similar health care programs aiming to adopt innovative teaching.