Program evaluation of a model to integrate internationally educated health professionals into clinical practice
© Greig et al.; licensee BioMed Central Ltd. 2013
Received: 5 November 2012
Accepted: 8 October 2013
Published: 11 October 2013
The demand for health professionals continues to increase, partially due to the aging population and the high proportion of practitioners nearing retirement. The University of British Columbia (UBC) has developed a program to address this demand, by providing support for internationally trained Physiotherapists in their preparation for taking the National Physiotherapy competency examinations.
The aim was to create a program comprised of the educational tools and infrastructure to support internationally educated physiotherapists (IEPs) in their preparation for entry to practice in Canada and, to improve their pass rate on the national competency examination.
The program was developed using a logic model and evaluated using program evaluation methodology. Program tools and resources included educational modules and curricular packages which were developed and refined based on feedback from clinical experts, IEPs and clinical physical therapy mentors. An examination bank was created and used to include test-enhanced education. Clinical mentors were recruited and trained to provide clinical and cultural support for participants.
The IEP program has recruited 124 IEPs, with 69 now integrated into the Canadian physiotherapy workforce, and more IEPs continuing to apply to the program. International graduates who participated in the program had an improved pass rate on the national Physiotherapy Competency Examination (PCE); participation in the program resulted in them having a 28% (95% CI, 2% to 59%) greater possibility of passing the written section than their counterparts who did not take the program. In 2010, 81% of all IEP candidates who completed the UBC program passed the written component, and 82% passed the clinical component.
The program has proven to be successful and sustainable. This program model could be replicated to support the successful integration of other international health professionals into the workforce.
KeywordsInternational health graduates Educational program development Program evaluation Integration
Health-care education around the world varies in educational standards, curriculum, and evaluation methods. The increasing demographic presence of ‘visible minorities’ and internationally-trained professionals, demands equal access to employment opportunities that support practice in occupational roles according to their qualifications and work experience. Support for this transition is essential to maximize skilled contribution to the development of a country . Canada is known as a country with a broad immigration policy which is reflected in its ethnic diversity, with over 71,559 people transitioning from temporary to permanent Canadian resident status in 2010 . The demand for health professionals continues to increase in Canada, partially due to the aging population and the high proportion of practitioners nearing retirement. Similar trends are predicted across all health professions [3–5], prompting the need for initiatives to increase the number of health professionals entering clinical practice. There is a need and desire to use readily available resources, the internationally-trained health professionals. To enter clinical practice, these health professionals need to demonstrate that they are safe and competent practitioners within the Canadian health care system. The knowledge and skills required to obtain a license to practice a profession vary by country and often by state or province, and most practitioners will require at least some orientation to the new health system and societal expectations of health care delivery. Others will require retraining in the basic sciences and clinical skills .
In order to obtain a license to practice Physiotherapy in Canada, in all provinces except Québec, Physiotherapists educated outside Canada must complete the two-part national Physiotherapy Competency Exam (PCE). In 2005 there was a notable discrepancy in pass rates between internationally trained Physiotherapists and Canadian graduates; less than 50% of internationally trained Physiotherapists exam attempts translated into passes, compared to greater than 92% of Canadian educated candidate attempts . The challenges to internationally trained Physiotherapists success are thought to be lack of familiarity with the Canadian healthcare system, differences in Physiotherapy practice between countries, issues with language fluency, specialisation in specific areas of practice that provide a challenge on a generalist examination, and lack of familiarity with the Canadian National examination format.
There is some literature about internationally trained health professionals; this literature mainly reflects the process of examining the credentials, competency, demographic characteristics and distribution of international medical graduates (IMGs) [8–11]. There is some literature which considers IMG’s and nurse’s experiences while recertifying, or the training programs ability to aid integration into the recipient country’s medical community and the country’s needs [12–18]. The conclusions from this work are that medical training programs need: i) support for international graduates to facilitate program completion, including faculty and peer mentoring, psycho-social counselling, and educational and orientation activities; ii) help to master the contextual areas of practice in addition to ensuring academic and technical competence; iii) understanding from administrators and faculty of cross-national medical training and practices (in terms of similarities and differences) . It is reasonable to suggest that the experiences and challenges experienced by IMGs and nurses as they recertify and integrate would be similar across other health professions.
To meet the demand for licensed Physiotherapists in Canada, and more specifically British Columbia (BC), and to provide support for internationally-trained physiotherapists who wish to continue to practice their skills and gain licensure, the “Internationally Educated Physiotherapists (IEP) Program Project” was established at the University of British Columbia.
The aim of this project was to create the educational tools, curriculum, and infrastructure needed to sustain a program supporting internationally trained Physiotherapists applying for licensure in BC. The primary objective of the program was to increase the pass rate of internationally trained Physiotherapists taking the PCE. The UBC ethics board states that ethical approval is not needed for program evaluation.
The IEP project was developed in a partnership with key BC Physiotherapy stakeholders (university, professional organization, regulatory body, health authorities) which provided oversight, guidance, communication services, advertising and marketing support, identification and recruitment of mentors and content experts. A key role in the management of the program was the Program Coordinator, who was a trained and licensed Physiotherapist with over 40 years of practice experience. The Program Coordinator was responsible for all activities of the program, including recruiting, scheduling, data collection, communication and feedback sessions related to exam modules. All inputs and activities are shown in Figure 1. Initial estimates projected that the program could admit four IEPs per cohort and six cohorts per year, with some flexibility to include additional students if there was excess demand.
The program consists of two separate streams to assist IEPs in the preparation for the PCE: one in preparing for the written component (QE), and the other to prepare for the clinical component (PNE) of the national exam. IEPs had the option to participate in both streams of the program, or to receive support for the preparation of only one component of the PCE (the written (QE) component or the clinical (PNE) component). The QE consists of 200 multiple-choice questions (MCQs) and the PNE is an objective structured clinical examination (OSCE) to evaluate candidate’s interactions in controlled scenarios with standardised clients. The primary objective of the program is to assist IEPs with their preparation for taking these exams. To achieve this objective, modules including examination writing skills, content area review, and practice exam experiences (using tested-enhanced learning theory) were developed . Many resources were available on-line; however, exams and workshops were initially delivered face-to-face.
Content experts developed initial versions of the curricular modules covering the four primary areas of Physiotherapy practice using the DACUM (Developing a Curriculum) model . All modules were assessed by an external expert for accuracy, ease of use, and adherence to competencies as outlined by The Canadian Alliance of Physiotherapy Regulators 2008 National Examination Blueprint . An updated version of the Alliance National Examination Blueprint was introduced in 2009 , that included a fifth area of practice (Multisystems); the IEP modules were revised to include this new area of practice. Two examination preparation modules were purchased from the national Physiotherapy Exam Skills Preparation Program and adapted to fit the IEP Program framework. Questions for the exam bank were developed by the content experts and reviewed by both a clinical expert and an educational expert to ensure the question format accurately reflected the questions found in the PCE.
Access to Physiotherapy mentors was initially identified as important to provide IEPs with the acquisition and reinforcement of practical skills in a Canadian context. Mentors were recruited from the clinical community based on area of clinical expertise, and experience teaching and supervising students. Three mentor-training workshops that focussed on facilitation techniques to support the learning of entry-level physiotherapy skills were offered to interested physiotherapists across the province. IEPs were matched with mentors who specialize in the primary areas of practice. The responsibilities of the mentors were to facilitate the development of the knowledge, clinical skills and clinical reasoning in each content area as expected in Physiotherapy practice in Canada. Mentors were also responsible for supporting the IEPs to achieve the competencies, as outlined in each module.
The program evolved since inception as a result of feedback from a variety of sources, including data collected from the IEPs, mentors and the IEP Advisory committee through focus groups and interviews. Feedback indicated that sample exam questions and practice exams were not perceived to accurately reflect the final PCE exam, and that there was a lack of detailed feedback after practice exams. Furthermore, IEPs suggested that modules were not specific enough and that IEPs were unfamiliar with terminology used within the Canadian Physiotherapy context. In response, study questions and a glossary of terms were added to each module, more complex vignette-based questions were developed, and based on the results of their exams IEPs are provided with guidance to further resources and study suggestions.
Several new initiatives resulted from the interviews with IEPs, including: clinical skills lab reviews; written tutorial workshops; mobile mentors; small group mentorship; delivery of written exam modules to remote communities; and transitioning the written program to an on-line platform. In addition, the written exam preparation was transitioned to a fully on-line program, so all aspects of the module was accessible to IEPs throughout BC, across Canada and even internationally.
Canadian alliance of physiotherapy regulators examination results
Written component (QE)
Clinical component (PNE)
Educational modular and curricular packages
Seven modules, based on the PCE blueprint and the core competencies of entry level physiotherapy in Canada, were developed and trialled.
More than 500 MCQs and 22 OSCE questions were developed, reviewed and, stored in the exam bank with marking guides and patient model instructions where appropriate. This supported the administration of internal written tests and internal clinical tests.
Initially 35 mentors were trained and paid for a maximum of 10 hours per IEP to provide one-on-one support and expertise during the exam preparation. The number of mentors trained was sufficient for the number of IEPs residing in the Lower Mainland of British Columbia, however, there were challenges pairing mentors with IEPs who resided out of the Lower Mainland, particularly those IEPs in rural areas. To address this, mobile and small group mentorship models were developed, which involved mentors from the Lower Mainland travelling to meet with IEPs in remote areas. This model proved to be cost effective and time efficient. In addition, higher IEP to mentor ratios (small group learning), maximised the mentor time and provided a more “dynamic” session for the IEPs. Feedback from IEPs and mentors, indicated that few IEPs utilized the allocated time with mentors, and that IEPs found the other components of the program (e.g. practice exams, workshops, clinical lab days etc.) of more benefit. The mentorship model therefore moved to self-regulated access; that is, IEPs were given contact details for mentors and were encouraged to make independent arrangements to meet with mentors according to need.
Characteristics of international educated physiotherapists accepted into program
Sex nMale (%Male)
Years of practice as physiotherapist
5 to 10
Country of physiotherapy education
English as 1st language
Written and clinical exams
Table 1 shows the PCE exam results for IEP program participants relative to all PCE candidates, Canadian-educated candidates, and all Internationally-educated candidates examination results. In the first year, 2008, there were very few graduates from the IEP program due to timing of the exams, three taking the written (QE) component and four the clinical (PNE) component, with 33% and 100% passing respectively. In 2010, 26 IEPs took the QE exam with 81% passing and, 33 took the PNE with 82% passing. The pass rate for IEPs taking the QE between 2008 and 2012 is 0.67, and for those taking the PNE is 0.92. These rates are significantly higher (p < 0.05 and p < 0.001, respectively using Fisher’s exact test) than the pass rate for all the internationally-educated physiotherapists who took the QE and PNE where the pass rate is 0.53 and 0.68 respectively. Taking the IEP program conveys a 28% increase in risk of passing the written exam, with a risk ratio (RR) in comparison to other international graduates taking the QE of 1.28 (95% CI, 1.02 to 1.59). The results from the PNE are even more positive with RR = 1.39 (95%CI, 1.27 to 1.52); IEP candidates have a 39% increase in risk of passing the clinical exam when compared to international candidates who do not take the IEP program.
Of the 65 IEPs taking the QE exam 65% had English as their first language, of these 63% passed the exam. Of the 35% who did not have English as their first language, 40% passed the QE. IEP candidates who have English as their first language have a non-statistically significant 58% increased risk of passing the QE, RR = 1.58 (95% CI 0.93 to 2.67, p = 0.09). Of the 83 IEPs taking the PNE exam, 74% had English as their first language, of these 93% passed the exam. Of the 26% who did not have English as their first language, 67% passed the PNE. For the PNE, IEP candidates who have English as their first language have a statistically significant, 40% increased risk of passing with an RR = 1.4(95% CI, 1.03 to 1.9, p = 0.03).
Of the 124 people who entered the IEP program 69 are known to have entered the Canadian physiotherapy workforce, with 34 of these known to be working in the public sector, five people are temporarily not in the Canadian workforce, due to child care or visa problems, and 20 are awaiting immigration approval for entry to Canada.
The aim of the IEP Program was to create the educational tools and infrastructure needed to sustain a program supporting internationally trained Physiotherapists applying for licensure in BC. Consideration was given to developing a program that included the necessary curriculum, resources, mentorship and practice opportunities to assist IEPs in their preparation for the exams and entering the Canadian workforce. As many resources as possible were made available on-line to serve the needs of IEPs from clinical practice sites distant from Vancouver. Evaluation was built into the program to inform continuing program improvements and the development of additional program initiatives. This has resulted in the Program attracting greater numbers of IEPs than projected, and providing the flexibility to support and accommodate these greater numbers. Projections identified the capacity to support and accommodate 8 IEPs per year in the practical and 16 in the written, and over the first 4 years of the program (2008–2011), 124 IEPs have participated. The program now recovers its expenses through charging the IEPs, and although participation has declined relative to the funded program, enrolment has been sufficient to cover costs. Using a framework to build the modules, gaining regular feedback and, making the necessary changes to develop an internationally educated health professional program has proven highly successful. Only 24 health professionals (19%) who entered the program were unable to succeed in national examinations and join the Canadian physiotherapy workforce. However, these professionals may retake the examination and enter the workforce in the future. The long-term data regarding entry of IEPs into practice indicates that at least 69 IEPs, who completed the IEP Program, entered the Canadian Physiotherapy workforce, with 34 of these working in the public sector.
The IEPs who participated appear to be representative of the population of Canadian immigrants; in the Canadian general populace the ratio of European to Asian immigrants is changing over time. In our population 58% of students came from the UK in 2009 and 12% in 2011, versus 10% of students from India in 2009 and 40% in 2011. The top four countries for physiotherapist immigrants practicing in Canada to have received their basic physiotherapy education, from 2007 to 2009, were UK (21%), India (15%), USA (10%) and Australia (8%) . The IEP program had fewer applicants from the U.S.A. and more from Australia; this could be explained by British Columbia having a lower U.S.A. and higher immigration rate from Asia and the Pacific than other provinces (Canadian average = 9% from USA, 25% from Asia and the Pacific; B.C. 6% and 59% respectively). Training, examination methods, healthcare context, and physiotherapist roles in the UK and Australia align more closely with Canadian standards than those from other countries (i.e. India and the Philippines); it is possible that Physiotherapists from countries with different academic procedures seek more opportunities to receive additional support for the exam preparation.
The primary objective of this program was to increase the pass rate of internationally trained Physiotherapists taking the PCE. The success of the program in increasing the pass rate of IEPs taking the PCE has been achieved with international graduates who took the program having a 28% greater possibility of passing the written section than their counterparts who did not take the program. The results for the clinical exam are even more impressive with the IEP program students having a 39% increase in risk of passing. A greater percentage of Canadian-educated candidates for the PCE continue to pass both elements of the PCE but the gap is closing, particularly in the clinical component. The program was developed partially due to the observation that in 2005 less than 50% of PCE attempts translated into passes in IEPs ; the students who participate in the program now far surpass this with more than 65% gaining a pass. Trends in IEP successes on the QE also indicate higher pass rates over time, suggesting a positive impact of the new initiatives including the written workshops, additional resources (expanded exam bank, additional written exam practice, glossaries and self-study questions), mobile mentors, small group mentorship, and transitioning the written program to an on-line platform to promote access throughout BC, across Canada and internationally.
It may be expected that those who do not have English as their first language would have more difficulty with participating in the course and passing the exams, in particular, the written element. This appears to be false for the QE, the relative risk is 1.58 but the confidence interval is very wide, crossing 1 (equality) and is not significant. However, in the slightly larger number of people taking the PNE (n = 81), the relative risk demonstrates that those with English as their first language have a significant, 40% greater risk of passing the clinical element. The IEP success rate on the PNE indicates a slight decrease over time, when more candidates who did not have English as their first language participated.
Although the IEP program targets internationally trained Physiotherapists, this model could be transferred to other health professions given the identified need to support internationally trained health professionals transitioning into the Canadian workforce , and the similarity in processes related to the passing competency examinations prior to registration in these professions . Using program evaluation methodology, including the logic model, proved a successful approach for the development of a program that aimed to integrate internationally educated health professionals into clinical practice.
As this is a relatively new program, there are not sufficient numbers of IEPs who have participated to enable sub-analyses of the data and examination of particular elements of the program. Collecting more qualitative data will allow for more in-depth analysis of the IEPs integration into Canadian practice and the impacts on acculturation and socialization.
Potentially this program model could be applied to support integration of international health graduates from any background into the workforce. While this program used as a primary outcome measure improved performance on national competency examinations, many program elements also support international practitioner’s integration into the Canadian health care system and practice. Further examination of the role specific program elements may have on improving internationally educated health professional’s understanding of the cultural context of Canadian practice is needed.
Physiotherapy Association of BC, College of Physical Therapists of BC, Vancouver Health Authority, Fraser Health Authority, and the University of British Columbia helped to design this program.
The pilot program was funded through a grant from the Ministry of Jobs Tourism and Innovation, Government of British Columbia.
- Racism in the Canadian job market; how widespread is discrimination against visible minorities in Canada’s labour market?: http://Canadaimmigrants.com.
- Research and Statistics: http://www.cic.gc.ca.
- Federal/Provincial/Territorial Advisory Committee on Health Delivery and Human Resources: How many are enough? Redefining self-sufficiency for the health workforce a discussion paper. 2009, Ottawa: Health CanadaGoogle Scholar
- Canadian Nurses Association: Planning for the future: nursing human resource projections. 2002, Ottawa: CNA, 87.Google Scholar
- Health Employers Association of BC, Ministry of Health Services: Health human resource planning projections for physiotherapists in British Columbia. 2010, Victoria: Ministry of Health ServicesGoogle Scholar
- Kidd MR, Zulman A: Educational support for overseas-trained doctors. Med J Aust. 1994, 160 (2): 73-75.Google Scholar
- Johnson K: Integrating internationally educated physiotherapists. 2007, Toronto: Canadian Alliance of Physiotherapy Regulators and Canadian Physiotherapy associationGoogle Scholar
- Boulet JR, Norcini JJ, Whelan GP, Hallock JA, Seeling SS: The international medical graduate pipeline: recent trends in certification and residency training. Health Aff Millwood. 2006, 25 (2): 469-477. 10.1377/hlthaff.25.2.469.View ArticleGoogle Scholar
- Crutcher RA, Banner SR, Szafran O, Watanabe M: Characteristics of international medical graduates who applied to the CaRMS 2002 match. CMAJ. 2003, 168 (9): 1119-1123.Google Scholar
- Fink KS, Phillips RL, Fryer GE, Koehn N: International medical graduates and the primary care workforce for rural underserved areas. Health Aff Millwood. 2003, 22 (2): 255-262. 10.1377/hlthaff.22.2.255.View ArticleGoogle Scholar
- Ko DT, Austin PC, Chan BT, Tu JV: Quality of care of international and Canadian medical graduates in acute myocardial infarction. Arch Intern Med. 2005, 165 (4): 458-463. 10.1001/archinte.165.4.458.View ArticleGoogle Scholar
- Bates J, Andrew R: Untangling the roots of some IMG’s poor academic performance. Acad Med. 2001, 76 (1): 43-46. 10.1097/00001888-200101000-00012.View ArticleGoogle Scholar
- Hall P, Keely E, Dojeiji S, Byszewski A, Marks M: Communication skills, cultural challenges and individual support: challenges of international medical graduates in a Canadian healthcare environment. Med Teach. 2004, 26 (2): 120-125. 10.1080/01421590310001653982.View ArticleGoogle Scholar
- Rukholm EE, Stamler LL, Talbot LR, Bednash G, Raines F, Potempa K, Nugent P, Clark DJ, Bernhauser S, Parfitt B: Scaling up the global nursing health workforce: contributions of an international organization. Collegian. 2009, 16 (1): 41-45. 10.1016/j.colegn.2009.01.001.View ArticleGoogle Scholar
- Searight HR, Gafford J: Behavioral science education and the international medical graduate. Acad Med. 2006, 81 (2): 164-170. 10.1097/00001888-200602000-00015.View ArticleGoogle Scholar
- Sochan A, Singh MD: Acculturation and socialization: voices of internationally educated nurses in Ontario. Int Nurs Rev. 2007, 54 (2): 130-136. 10.1111/j.1466-7657.2007.00564.x.View ArticleGoogle Scholar
- Whelan GP: Commentary: coming to America: the integration of international medical graduates into the American medical culture. Acad Med. 2006, 81 (2): 176-178. 10.1097/00001888-200602000-00017.View ArticleGoogle Scholar
- Wong A, Lohfeld L: Recertifying as a doctor in Canada: international medical graduates and the journey from entry to adaptation. Med Educ. 2008, 42 (1): 53-60.View ArticleGoogle Scholar
- Bordage G: Conceptual frameworks to illuminate and magnify. Med Educ. 2009, 43 (4): 312-319. 10.1111/j.1365-2923.2009.03295.x.View ArticleGoogle Scholar
- Centers for Disease Control and Prevention: Framework for program evaluation in public health. MMWR Recomm Rep. 1999, 48 (RR-11): 1-40.Google Scholar
- Hatry H, van Houten T, Plantz M, Greenway M: Measuring program outcomes: a practical approach. 1996, Virginia: United Way of AmericaGoogle Scholar
- Larsen DP, Butler AC, Roediger HL: Test-enhanced learning in medical education. Med Educ. 2008, 42 (10): 959-966. 10.1111/j.1365-2923.2008.03124.x.View ArticleGoogle Scholar
- Delta Pi Epsilon Society, L. R: Delta Pi Epsilon National Research Conference Proceedings (Columbus, Ohio, November 15-17, 1990). [S.l.]. 1990, : Distributed by ERIC ClearinghouseGoogle Scholar
- Canadian Alliance of Physiotherapy Regulators: Physiotherapy competency examination. Candidate handbook. 2008, Toronto: Canadian Alliance of Physiotherapy RegulatorsGoogle Scholar
- Canadian Alliance of Physiotherapy Regulators: Physiotherapy competency examination. Candidate handbook. 2009, Toronto: Canadian Alliance of Physiotherapy RegulatorsGoogle Scholar
- Canadian Institute for Health Information: Physiotherapists in Canada, 2009. 2010, Ottawa: Canadian Institute for Health InformationGoogle Scholar
- Sweatman L, Barry J, Liitle L, Davies J: International nurse applicants: initiatives, including LeaRN. Can Nurse. 2003, 99 (8): 34-36.Google Scholar
- The pre-publication history for this paper can be accessed here:http://www.biomedcentral.com/1472-6920/13/140/prepub
This article is published under license to BioMed Central Ltd. This is an open access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.