- Research
- Open access
- Published:
Evaluation of continuous quality improvement in accreditation for medical education
BMC Medical Education volume 20, Article number: 308 (2020)
Abstract
Background
Accreditation systems are based on a number of principles and purposes that vary across jurisdictions. Decision making about accreditation governance suffers from a paucity of evidence. This paper evaluates the pros and cons of continuous quality improvement (CQI) within educational institutions that have traditionally been accredited based on episodic evaluation by external reviewers.
Methods
A naturalistic utility-focused evaluation was performed. Seven criteria, each relevant to government oversight, were used to evaluate the pros and cons of the use of CQI in three medical school accreditation systems across the continuum of medical education. The authors, all involved in the governance of accreditation, iteratively discussed CQI in their medical education contexts in light of the seven criteria until consensus was reached about general patterns.
Results
Because institutional CQI makes use of early warning systems, it may enhance the reflective function of accreditation. In the three medical accreditation systems examined, external accreditors lacked the ability to respond quickly to local events or societal developments. There is a potential role for CQI in safeguarding the public interest. Moreover, the central governance structure of accreditation may benefit from decentralized CQI. However, CQI has weaknesses with respect to impartiality, independence, and public accountability, as well as with the ability to balance expectations with capacity.
Conclusion
CQI, as evaluated with the seven criteria of oversight, has pros and cons. Its use still depends on the balance between the expected positive effects—especially increased reflection and faster response to important issues—versus the potential impediments. A toxic culture that affects impartiality and independence, as well as the need to invest in bureaucratic systems may make in impractical for some institutions to undertake CQI.
Background
The design [1], as well as the value of accreditation systems [2, 3], are topics that require additional research. In this study we reflect on the virtues of continuous quality improvement (CQI) in medical education.
There is an implicit belief that the accreditation of medical education programs ensures the quality of the education these programs provide and the graduates they produce [4,5,6]. The importance of the accreditation of medical education programs has been further heightened by a recent announcement by the Educational Commission for Foreign Medical Graduates. The announcement states that, by the year 2023, the only candidates who can be certified, and therefore considered for residency education in the United States, will be graduates of medical schools that have been accredited by organizations approved by the World Federation of Medical Education as working to appropriately formulated and applied standards [7]. Despite the respect that accreditation is accorded, little has been written about whether the accreditation process actually achieves its intended outcomes. In many jurisdictions, accreditation of medical education programs has traditionally been based on episodic evaluation [8]. In general terms, an episodic evaluation is an external review by an accreditation authority that includes on-site visits and is repeated at specified intervals [8, 9]. More recently, the focus of accreditation has been shifting from episodic evaluation to CQI [10,11,12,13]. CQI, which originated in industry as an approach to quality control for services or products [14,15,16,17], is a never-ending cycle of organizational improvement that uses quality indicators [18].
This paper, one of a series arising from the first World Summit on Accreditation Outcomes in Medicine, explores the role of CQI in systems traditionally based on episodic evaluation. As explained before, the evidence in literature for the benefits of a CQI system in medical education is insufficient, although CQI has been introduced in several jurisdictions. The result is a new bureaucratic instrument that is being used without sufficient empirical evidence. As such, it is important to investigate the pros and cons of CQI in medical education. In this utility-focused evaluation [19] we use a set of quality criteria for governance oversight to shed more light on these pros and cons.
Methods
We chose to perform a naturalistic utility-focused evaluation [19]. Three of the authors (LP, CC, and FS) were responsible accreditation officers in their distinct fields of medical education who wished to add to the evidence and inform decision making around investments in CQI. The fourth author (NA) is a certified lawyer, medical doctor and PhD student researching accreditation systems. At the time of this evaluation, there were no conflicts concerning CQI that might have influence their opinions.
We chose to use a set of quality criteria for government oversight that did not originate from the field of medical education, because a firm theoretical background in the accreditation literature for medical education was not available.
Our author team was assigned to the roles of both descriptor of the accreditation cases and evaluator of CQI based on the criteria. The authors discussed and evaluated each system using the criteria and searched for patterns with respect to the quality criteria. During these discussions, detailed questions about each system were answered by the author responsible for that system. This process was repeated iteratively until the authors reached consensus for each criterion.
Case examples: three accreditation systems
In the next section, the accreditation systems are briefly introduced to give the reader a context for our discussion about the pros and cons of CQI. A comprehensive description of the systems may be found on the websites of the accreditation authorities (afmc.ca; cacms-cafmc.ca; lcme.org; www.knmg.nl; www.royalcollege.ca) or obtained from the authors. LP, CC and FS provided detailed descriptions of the systems—both written and oral.
The American and Canadian undergraduate accreditation systems
In North America, two authorities accredit medical degree-granting programs: the Committee on Accreditation of Canadian Medical Schools (CACMS) for medical schools in Canada, and the Liaison Committee on Medical Education (LCME) for medical schools in the United States and Canada. The CACMS and the LCME are external bodies with respect to the programs they accredit. The accreditation cycle includes an on-site visit by a peer-review team; the interval between one full survey and the next is 8 years. The internal review process for the full survey requires approximately 2 years and involves a major expenditure of human and financial resources. In 2009, the Association of Faculties of Medicine of Canada (AFMC) approved a mandatory interim review process (IRP) for Canadian medical schools that includes a review of CACMS accreditation standards at the mid-point of the formal accreditation cycle, along with CQI to achieve and maintain compliance with standards throughout the cycle [8]. The AFMC IRP is currently independent of the CACMS, and the effectiveness of the process is the subject of ongoing review. The LCME, with the endorsement of CACMS, has introduced an accreditation standard requiring schools to have a CQI process to monitor compliance with accreditation standards; this came into effect during the 2015–16 academic year [5]. Research is needed to determine the efficacy of these mandatory requirements for CQI.
The Dutch postgraduate accreditation system
In the Dutch system there are two important external authorities for the accreditation of residency training: the CGS (Legislative College for Accreditation of Residency Training) and the RGS (Specialist Registration Committee). These are bodies of the Royal Dutch Medical Association (KNMG). The CGS is responsible for setting regulations (training requirements) for accreditation, while the RGS’s role is to monitor compliance with these regulations. The medical specialties in teaching hospitals are visited every 5 years by a survey team commissioned by the RGS. There are no specific mandatory internal quality management systems regarding residency training in teaching hospitals; however, sufficient instruments and systems are made available to conduct internal quality control. The postgraduate medical education programs are required to perform some form of CQI.
Continuous professional development accreditation in Canada
There are two basic types of continuous professional development (CPD) accreditation systems: provider-based accreditation and activity-based accreditation. In activity-based CPD accreditation systems, providers submit evidence to the accreditation organization about whether or not an activity they have developed complies with established administrative, educational, and ethical standards. If the accreditation authority agrees that the standards have been met, the activity is approved. Activity-based accreditation is used, for example, by the College of Family Physicians of Canada. Provider-based CPD accreditation systems recognize organizations as institutional providers of continuing education on the basis of their ability to demonstrate adherence to a set of standards. The accreditation process frequently includes an on-site visit in which a peer-review team has the opportunity to discuss specific standards with, among others, the CPD organization’s leadership, administrative staff, and participants. Once an organization is approved, all activities it develops during its accreditation cycle are automatically approved for credit. A provider-based accreditation system is used by the Royal College of Physicians and Surgeons of Canada (RCPSC) to focus on national and provincial CPD organizations, including national specialty societies, while the Committee on Accreditation of Continuing Medical Education (CACME) focuses on the university offices of continuing medical education (CME) in the faculties of medicine. These two provider accreditation systems will be used to illustrate the process by which continuous accreditation is expressed and promoted.
Criteria for oversight
Accreditation is actually a method of oversight. Oversight functions of government and accreditation processes in medicine may struggle with similar difficulties in achieving quality improvement [20, 21], even though different stakeholders are involved. The Netherlands Scientific Council for Government Policy (WRR) identified seven criteria as the basis for societally responsive oversight [22] (Table 1). Given their general applicability, we used these criteria to evaluate the pros and cons of CQI in undergraduate medical education in the United States and Canada; postgraduate medical education in the Netherlands; and CPD in Canada.
Results
The results are summarized in Table 2 and reflect the authors’ evaluation of the pros and cons of CQI at the institutional level.
Discussion
Our findings show that an integrated system for accreditation of medical education may consist of external authorities and the local CQI system supporting timely reflection and action to serve the public interest. External authorities might even need less capacity due to local quality systems. The downside of CQI is the bureaucracy and costs involved, the risk of interference with other quality systems, the need for an open culture, and potential issues with impartiality and dependency of colleagues involved.
Our findings partially conflict with other literature. Since the 1980s CQI has been used in manufacturing and engineering, the financial sector, education, and health care. Various methods have been used to pursue CQI (e.g., the plan-do-check-act or plan-do-study-act cycle, the Six Sigma standard in manufacturing and other fields, “total quality management”) [23]. All of these share the concept of a “cycle of improvement.” In the manufacturing industry, CQI has been useful in reducing defects, lowering costs, and increasing quality and productivity [24, 25]. In the construction industry it has led to economic benefits, increased customer satisfaction, a reduced need to re-do work, and declining defects in housing projects [26, 27]. Further, CQI is readily applicable to standardized and repetitive processes [23]. In surgical care it has resulted in reduced infection rates and shorter delays in the operating room [23]. Comparing these positive literature reports with the evaluation of CQI in our contexts suggests a publication bias of positive results. It is difficult to measure economic effects and other outcomes in health care and medical education [28,29,30]. The cost–benefit ratio in our contexts may not always be positive. The difficulty in going through all the components of the plan-do-study-act cycle has been described before [30,31,32]. The bureaucracy and costs involved in CQI, the risk of interference with other quality systems, the need for an open culture, and potential issues with impartiality and dependency of colleagues involved are important findings. The criteria for oversight used in our analysis were instrumental to the detection of these issues.
Future research addressing CQI in medical education may address design variations [33], elaborate on various perspectives on the cost–benefit evaluation in different contexts [34], and quantitatively measure [35] the effects of CQI in medical education contexts. The gathering of evidence about accreditation systems and CQI in medical education seems to be still in its infancy [5, 9, 36].
The strength of this work is the use of criteria for oversight that provided a lens suitable to critically evaluate the use of CQI. The evaluation gains validity by using cases throughout the continuum of medical education and from three different countries. The cases warrant a naturalistic approach connected to real-life experiences. The evaluation would gain more reliability through the introduction of additional evaluators and cases. Moreover, a shift from qualitative research to a mixed methods design with an attempt to introduce quantitative elements would enhance the value of this kind of research.
Conclusion
We performed a utility-based evaluation in the hope of reducing complexity in decision making about the use of CQI. The result is one of both pros and cons. The introduction of CQI in the three cases studied may be regarded as an experiment. Further research is needed to find out the contexts in which itis worth investing in CQI. The use of criteria for oversight helped identify issues that will facilitate decision making around the introduction of CQI.
Availability of data and materials
The data in this study were collected mainly through the expertise and involvement of the authors FS, LP, and CC in accreditation activities and partly from the websites of the accreditation authorities. Because of our reliance on the authors’ expertise, we have provided an extensive description of their professional activities below.
Abbreviations
- AFMC:
-
Association of Faculties of Medicine of Canada
- CACME:
-
Committee on Accreditation of Continuing Medical Education
- CACMS:
-
Committee on Accreditation of Canadian Medical Schools
- CFPC:
-
College of Family Physicians of Canada
- CGS:
-
Legislative College for Accreditation of Residency Training
- CPD:
-
Continuous Professional Development
- CQI:
-
Continuous Quality Improvement
- IRP:
-
Interim Review Process
- KNMG:
-
Royal Dutch Medical Association
- LCME:
-
Liaison Committee on Medical Education
- RCPSC:
-
Royal College of Physicians and Surgeons of Canada
- RGS:
-
Specialist Registration Committee
References
Akdemir N, et al. How changing quality management influenced PGME accreditation: a focus on decentralization and quality improvement. BMC Med Educ. 2017;17(1):98.
Braithwaite J, et al. Health service accreditation as a predictor of clinical and organisational performance: a blinded, random, stratified study. Qual Saf Health Care. 2010;19(1):14–21.
Greenfield D, Braithwaite J. Health sector accreditation research: a systematic review. Int J Qual Health Care. 2008;20(3):172–83.
Maniate JM. Redesigning a resident program evaluation to strengthen the Canadian residency education accreditation system. Acad Med. 2010;85(7):1196–202.
Boulet J, Zanten M. Ensuring high quality patient care: the role of accreditation, licensure, specialty certification and revalidation in medicine. Med Educ. 2014;48(1):75–86.
Davis DJ, Ringsted C. Accreditation of undergraduate and graduate medical education: how do the standards contribute to quality? Adv Health Sci Educ. 2006;11(3):305–13.
van Zanten M, Boulet JR, Norcini JJ. Ethical Integration of Internationally Educated Health Professionals: Ethical and Regulatory Contexts. Paper presented at: 13th International Health Workforce Collaborative Conference. Brisbane; 2011. http://rcpsc.medical.org/publicpolicy/conference13.php. Accessed 4 Feb 2019.
Barzansky B, Hunt D, Moineau G, Ahn D, Lai CW, Humphrey H, Peterson L. Continuous quality improvement in an accreditation system for undergraduate medical education: benefits and challenges. Med Teach. 2015;37:11:1032–8. https://doi.org/10.3109/0142159X.2015.1031735.
Norcini JJ, van Zanten M. An overview of accreditation, certification, and licensure processes; 2010.
Djuricich AM, Ciccarelli M, Swigonski NL. A continuous quality improvement curriculum for residents: addressing core competency, improving systems. Acad Med. 2004;79(10):S65–7.
Wong BM, Levinson W, Shojania KG. Quality improvement in medical education: current state and future directions. Med Educ. 2012;46(1):107–19.
Karle H. Global standards and accreditation in medical education: a view from the WFME. Acad Med. 2006;81(12):S43–8.
Al-Shehri AM, Al-Alwan I. Accreditation and culture of quality in medical schools in Saudi Arabia. Med Teach. 2013;35(s1):S8–S14.
Shortell SM, et al. Assessing the impact of continuous quality improvement/total quality management: concept versus implementation. Health Serv Res. 1995;30(2):377.
Berwick DM. Continuous improvement as an ideal in health care. N Engl J Med. 1989;320(1):53.
Laffel G, Blumenthal D. The case for using industrial quality management science in health care organizations. Jama. 1989;262(20):2869–73.
Radawski D. Continuous quality improvement: origins, concepts, problems, and applications. Perspect Physician Assist Educ. 1999;10(1):12–6.
Kritchevsky SB, Simmons BP. Continuous quality improvement: concepts and applications for physician care. Jama. 1991;266(13):1817–23.
Savin-Baden M, Howell-Major C. Qualitative research: the essential guide to theory and practice. Qualitative Research: The Essential Guide to Theory and Practice. Abingdon-on-Thames: Routledge; 2013.
Stringham SH. Does quality management work in the public sector. Public Adm Manag Interact J. 2004;9(3):182–211.
Majone G. Strategy and structure: The political economy of agency independence and accountability. Designing Independent and Accountable Regulatory Agencies for High Quality Regulation; 2005. p. 10–1.
Scientific Council for Government, P., Supervising public interests. Towards a broader perspective on government supervision (Toezien op publieke belangen. Naar een verruimd perspectief op rijkstoezicht). 2013.
Nicolay CR, et al. Systematic review of the application of quality improvement methodologies from the manufacturing industry to surgical healthcare. Br J Surg. 2012;99(3):324–35.
Maani KE, Putterill MS, Sluti DG. Empirical analysis of quality improvement in manufacturing. Int J Qual Reliability Manag. 1994;11(7):19–37.
Kwak YH, Anbari FT. Benefits, obstacles, and future of six sigma approach. Technovation. 2006;26(5):708–15.
Hoonakker P, Carayon P, Loushine T. Barriers and benefits of quality management in the construction industry: an empirical study. Total Qual Manag. 2010;21(9):953–69.
Chase GW. Improving construction methods: a story about quality. J Manag Eng. 1998;14(3):30–3.
Lurie SJ, Mooney CJ, Lyness JM. Measurement of the General Competencies of the Accreditation Council for Graduate Medical Education: A Systematic Review. Acad Med. 2009;84(3):301–9. https://doi.org/10.1097/ACM.0b013e3181971f08.
Prystowsky JB, Bordage G. An outcomes research perspective on medical education: the predominance of trainee assessment and satisfaction. Med Educ. 2001;35(4):331–6.
Marshall M, Ÿvretveit J. Can we save money by improving quality? BMJ Qual Saf. 2011:bmjqs-2010.
Heard JK, et al. An institutional system to monitor and improve the quality of residency education. Acad Med. 2004;79(9):858–64.
Taylor MJ, McNicholas C, Nicolay C, et al. Systematic review of the application of the plan–do–study–act method to improve quality in healthcare. BMJ Qual Saf. 2014;23:290–8.
Akdemir N, et al. Using Gamification to understand accreditation in postgraduate medical education. J Grad Med Educ. 2019;11(4 Suppl):207–10.
Blouin, D. and A. Tekian, Accreditation of Medical Education Programs: Moving From Student Outcomes to Continuous Quality Improvement Measures. Acad Med, 2017.
Eitel F, Steiner S, Tesche A. Quality management: transition to medical making the education. Med Teach. 1998;20(5):445–50.
van Zanten M, et al. Overview of accreditation of undergraduate medical education programmes worldwide. Med Educ. 2008;42(9):930–7.
Acknowledgements
The authors would like to thank the members of the International Health Professions Accreditation Outcomes Consortium, and everyone who reviewed and provided feedback to this project.
About this supplement
This article has been published as part of BMC Medical Education Volume 20 Supplement 1, 2020: Current themes and challenges facing HPE accreditation in the twenty-first century. The full contents of the supplement are available online at https://bmcmedicaleducation.biomedcentral.com/articles/supplements/volume-20-supplement-1.
Funding
No funding was obtained for this study.
Author information
Authors and Affiliations
Contributions
NA and FS drafted the manuscript. LP, CC, and FS authored specific sections of the manuscript and made critical revisions and additions to the manuscript. All authors approved the submission of the final manuscript.
Authors’ information
NA is a PhD researcher in medical education at the OLVG Hospital and VU Medical Center in Amsterdam, the Netherlands. The subject of her PhD is accreditation of residency training. She combines the areas of medical education, quality management, and regulation in her PhD. NA holds a Master’s degree in law and medicine.
LP is the former assistant secretary of the Committee on Accreditation of Canadian Medical Schools (CACMS) and led the Interim Review Process of the Association of Faculties of Medicine of Canada. She was a member of CACMS for 6 years and participated in many CACMS/LCME accreditation surveys in various capacities. LP was the senior evaluation advisor to the Faculty of Medicine at the University of British Columbia and was formerly dean of the Undergraduate Medical Education Program at the Faculty of Medicine, University of Ottawa.
CC is a Specialist in Internal Medicine and associate professor of Medicine at the University of Ottawa, and is Director, Curriculum, UGME Program at the University of Ottawa. CC is a past director of CPD in the Office of Specialty Education at the RCPSC and developed the Royal College’s CPD accreditation system. CC is a founding member of the International Academy for CPD Accreditation and is a Fellow of the Society for Academic Continuing Medical Education.
FS has worked as a gynecologist and dean at the OLVG teaching hospital in Amsterdam, the Netherlands. He is also a professor in Health Systems Innovation and Education at VU University Amsterdam and the Athena Institute for Transdisciplinary Research. FS is the national project leader of the curriculum design for postgraduate training in obstetrics and gynecology, and is currently leading the national reform project for accreditation of residency training and medical practice as chairman of the Legislative College for Accreditation of Residency Training in the Netherlands, where he is closely involved in the regulation of training. FS is the president of the board of the Netherlands Association for Medical Education. He also holds positions in several national and international committees and projects in postgraduate training reform.
Corresponding author
Ethics declarations
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Akdemir, N., Peterson, L.N., Campbell, C.M. et al. Evaluation of continuous quality improvement in accreditation for medical education. BMC Med Educ 20 (Suppl 1), 308 (2020). https://doi.org/10.1186/s12909-020-02124-2
Published:
DOI: https://doi.org/10.1186/s12909-020-02124-2