Skip to main content
Download PDF

Adapting the Donabedian model in undergraduate nursing education: a modified Delphi study

BMC Medical Education volume 24, Article number: 202 (2024) Cite this article

Abstract

Background

Donabedian conceptual and multidimensional framework has been used in several studies in an educational context. In this study, we aimed to adapt the Donabedian three-component model (structure, process, and outcome) in undergraduate nursing education. This conceptual framework provides a comprehensive image of nursing education institutions and can help to evaluate institutions by measuring different aspects of their performance. A comprehensive understanding of the various elements of an educational institution helps to develop a complete, appropriate relevant set of performance indicators.

Methods

This was a modified Delphi study. It had three rounds. The expert panel consisted of nursing faculty members and nursing Ph.D. students. In the first round, a questionnaire was designed based on interviews, focus groups, and a literature review. Experts rated their agreement with each element on a 5-point Likert scale in rounds two and three. The consensus level was set as 75%. The stability between rounds was also determined by calculating kappa coefficients. One Sample T-Test was also calculated for new items in round three.

Results

All 55 items of the questionnaire were confirmed in the second round based on the consensus percentage of 75. Five new items were added to the third round based on comments in round two. Eventually, all elements except one were confirmed according to the consensus level, kappa values, means, and One-Sample T-Test in round three. The structure's key elements include staff (academic and non-academic); equipment; guidelines; resources and facilities; and students’ demographics and characteristics. Process key elements include communication; education; evaluation; cooperation; and consultation. Outcome key elements include knowledge development; nursing image; alumni’s outcome; students’ outcome; related medical centers’ performance; accreditation and evaluation results; and satisfaction.

Conclusions

Different elements of a nursing education institution at the bachelor's level were determined. The results of this study can help related bodies to develop and implement a comprehensive and systematic evaluation. These results can also be a basis for making this model useful in other nursing courses or education in other fields.

Peer Review reports

Background

Nurses contribute to global health [1] and affect patient’s experience of care [2]. The continuum of nursing practice contains a broad range from health promotion, to disease prevention, to coordination of care, to cure, and to palliative care [3]. They even deal with the private and personal aspects of people's lives [4]. Nursing care covers all aspects of health needs containing physical social, mental, and spiritual [3].

To improve health care and health systems, nursing education must be improved [5]. Improved education systems ensure quality care provided by graduated nurses [3]. The production of competent nurses is the critical role of nursing education [5]. For improving the quality of higher education institutions, evidence-based evaluation and audit are needed [6]. Audits, evaluations, and assessments have become increasingly common in higher education [7]. These can be complex, interrelated processes in nursing education [8]. One of the practical challenges in conducting evaluation is the use of appropriate evaluation methods and tools [9].

Most educational institutions use peer review and accreditation as an assessment of their performance [10]. Accreditation is often cited as a barrier to program innovation [11], it can be complex and time-consuming [12], and understanding it’s outcomes is difficult for lay stakeholders [13]. Performance measurement can be used as a powerful tool for evaluating and controlling organizations [14]. Even if accreditation is used, it is useful to have performance indicators to evaluate the accreditation program [15, 16].

Performance measurement includes efforts to monitor, evaluate, and establish the relationship between the organization’s goals, resources, and activities, with the results, outputs, and achievements of goals [17]. Appropriate indicators must be selected to measure the performances [18]. Key performance indicators provide a powerful and effective tool to evaluate, adjust, and motivate employees to work more intensively, thus bringing more benefits to the common cause [19].

During the previous steps of this project, we were looking to develop performance indicators for nursing education institutions. We held several interviews and focus groups with different groups of stakeholders. We also conducted a review of existing indicators on nursing schools’ websites. During the data analysis, we realized that most of the indicators or areas that need to be measured can be classified based on three components: structure, process, and outcome. We were also looking for a proper conceptual framework that is compatible with the study. It is useful to have a conceptual framework to guide the development of a set of indicators [20]. We needed a conceptual framework to help us in developing and organizing the indicators. Existing frameworks were reviewed to find a suitable framework for the study. We also used a published systematic review of existing frameworks [21]. During this review and our data analysis, we found that the Donabedian Model is completely compatible with the collected data and could be used as the conceptual framework in indicator development for nursing education institutions. So we used the Donabedian quality improvement model as a guide to classify data. However, this model was used in educational contexts before, but it has never been adapted comprehensively in an educational context. We assumed maybe adapting this model and providing its elements in nursing education may be a basis for further studies or help in quality improvement studies.

Avedis Donabedian proposed a framework in the 1960s. Based on his framework the quality of health care could be assessed by evaluating three elements—structure, process, and outcome. Its flexibility makes it useful in quality improvement initiatives across clinical settings [22]. The Donabedian model provides a framework for conducting evidence-based service evaluation. With this model, the differences between elements of structure, process, and the outcome can be recognized. This would provide a more complete picture of a service [23]. By following Donabedian's model, the discovery of a beneficial outcome will be of much practical value because one knows what factors are necessary for it to happen (structure) and how it was actually achieved (process). In this manner the evaluation of quality can be more concrete and comprehensive [24]. According to Donabedian’s model, improvements in the structure can improve processes that should in turn improve outcome [25].

Donabedian conceptual and multidimensional framework has been frequently used [26]. Including several studies in the educational context [24, 27,28,29,30]. But we could not find a study that has adopted this model in nursing education. In this Delphi study, we aimed to adapt the Donabedian model as a proper conceptual framework for nursing education. We specified its elements comprehensively and defined each of its three components. Later this conceptual model could be used for performance indicator development in undergraduate nursing education or other goals.

Methods

Design

This is a Modified Delphi study. The Delphi method assumes that group opinion is more valid than individual opinion and the technique aims to achieve agreement among a group of experts on a certain issue [31]. Expert consensus methods are commonly employed in consumer, education, and health services research [32]. This Delphi study had three rounds. Delphi studies are typically carried out in two to three rounds [33]. It is difficult to keep a high response rate within a Delphi study that has many rounds [31]. A schematic view of the Delphi rounds is provided in Fig. 1.

Fig. 1
figure 1

Schematic view of the Delphi rounds. This modified Delphi study had three rounds, in round 1 a questionnaire was designed that were rated by expert panel members in round 2 and 3. 5 New items were added to the questionnaire based on expert panel opinion in round three. Finally, 59 items were confirmed

Full size image

Preparation phase

We considered the main components of the Donabedian model (structure, process, outcome) and tried to identify and confirm the elements of these three components during the Delphi rounds. Like the Donabedian, we assumed that the structure would affect the process and the process would affect the outcome [34]. Also, based on Donabedian's definitions of structure, process, and outcome [35], we considered the following definitions for these three components in nursing education (these definitions were also confirmed by the experts during the Delphi rounds):

  • Structure: Attributes which in and with the nursing education can occur. It includes all factors affecting the conditions of education.

  • Process: All the acts of nursing education institutions.

  • Outcome: All the effects of nursing education institutions.

Expert panel

The expert panel for this study consisted of the nursing faculty members and the nursing Ph.D. students. Regarding the faculty members, we tried to invite experts with experiences in different areas. They consisted of those who had experiences in nursing school management, were members of the board of nursing, or simply were nursing professors or instructors. Regarding the Ph.D. students, besides they were students (as an important group of stakeholders) and familiar with the concept of model adaptation (as they were at the Ph.D. level), we invited those who had experiences in nursing education or practice. The expert is defined as a person with knowledge and experience in a particular subject matter. Experts are used to increase the qualitative strength of recommendations or consensus. There is no standard size of the panel members, but 10–15 experts are considered acceptable [36, 37].

Sampling

Purposive sampling was used in this study. People who were considered "experts" in the field of the study, and also had a variety of experiences, were selected and invited to participate in the study. Purposive sampling is designed to find people who can and are willing to provide information through knowledge or experience [38].

Setting

This study was based in a nursing and midwifery faculty. But experts from colleges across the country were invited to study as the expert panel members.

Main features of Delphi studies

The Delphi method presents characteristics and features including the anonymity of experts, iteration, controlled feedback, and qualitative and quantitative data [39]. In this study, these features were addressed in different rounds:

  • Anonymity of experts: In this study, expert panel members were anonymous to each other. This assures the free expression of opinions and helps to avoid social pressure from dominant individuals [40].

  • Iteration: It is possible for experts to change their opinions or judgments at any point in the study [40]. We prepared an exclusive questionnaire for each of the participants in the third round (Fig. 2). In these questionnaires, there was a section that specified the person's answer to each of the items in the previous round, and an opportunity was created for the expert to change her/his previous answer if she/he wished.

  • Controlled feedback: Experts are informed about the views of other experts who participate in the study [40]. In the third round’s questionnaire, besides each person’s own answer to each of the items in round two, we also provided the overall expert panel members’ agreement percentage for each specific item. So the panel members could know what had been the overall expert panel opinion about the item.

  • Although the Delphi method is qualitative, it can provide quantitative results [40]. In this study, the qualitative data that was retrieved from the first round and also from comments in the second round were converted to items and were rated quantitatively by the expert panel.

Fig. 2
figure 2

The questionnaire that were designed for the third round of Delphi. This questionnaire provides expert previous rate on the elopements, expert panel overall agreement, and a chance for experts to change their previous answers

Full size image

Consensus

The panel of experts was asked to indicate their level of agreement with each of the specified elements in a 5-point Likert scale (5. Strongly agree, 4. Agree, 3. Neither agree nor disagree, 2. Disagree, 1. Strongly disagree). Two "Strongly agree" and "Agree" answers were considered as agreement, the answer "neither agree nor disagree" as neutral and the answers "Disagree" and "Strongly disagree" were considered as disagreement. The level of agreement for achieving consensus was set as 75% [41]. This means that at least 75% of experts had to rate an element 5 or 4, till that item can be maintained. Besides this we also calculated the stability of responses between rounds 2 and 3 to decide on ending the Delphi rounds [41, 42].

Round one

In the modified Delphi technique, the first round can be made up of focus groups or face-to-face interviews or a structured questionnaire may be used, with quantitative questions based on the literature in previous researches [43].

In this round, a questionnaire was designed consisting of 55 items of expected elements for the three components of structure, process, and outcome. These elements were the result of the literature review, the interviews, and the focus groups that were held with stakeholders in previous steps of the main study that this Delphi study is part of. Our aim in the main study was to develop nursing education key performance indicators. We wanted to know about the aspects or indicators that should be measured to represent a nursing education institution’s performance. We tried to gather different groups of stakeholders’ perspectives. Four focus groups were conducted with bachelor's, master's, and Ph.D. students (total number of students = 27). A focus group was conducted with nurses (n = 6). Seven individual interviews were conducted with nursing faculty members and policymakers. Also, an online form was designed on Google Form, which was filled by 91 nurses, nursing students, and nursing faculty members (demographic characteristics of the participants and further information about this part are provided in the supplementary file named “Additional information about the major study”).

In the first round of this modified Delphi study, a questionnaire was designed from the findings of these steps. This questionnaire was later rated in the subsequent rounds by the expert panel’s members. Except for the questionnaire items (elements) we also prepared a section for each of the three components so that the expert panel members could provide further comments about items or possible other elements that we hadn’t mentioned. We also prepared an extra section where the expert panel could leave their comments about the entire structure of the model designed based on the Donabedian Model. At the beginning of the questionnaire we provided some explanation about the study aims, the Donabedian Model, how the items (elements of three components) were developed, and how the questionnaire must be filled. We explained the components of the Donabedian model and their definitions in the main model. Then we provided explanations about the previous steps of the study, including interviews and literature review, and how the elements related to each component were obtained. We also stated some reasons regarding the use of this model in educational fields and the importance of adapting the model in nursing education. Then the participants were asked to express their level of agreement with the fact that each of these mentioned elements can be part of the structure, process or outcome of a nursing education institution. In other words, we wanted to ensure with the help of experts that the elements considered for each component are correct and relevant.

Round two

After designing the questionnaire in the first round, in the second round of this Delphi study, 20 experts were invited to participate in the study. Of these, 15 experts agreed to participate. If possible, the questionnaires were delivered to the experts in person, and if they were not available, questionnaire were delivered to them by email. If necessary, reminder messages were sent to some of the participants one to three times, and finally, 13 members answered the questionnaire in the second round of the study. We put one-month limitation to complete this round.

Round three

In the third round, after editing the questionnaire of the second round based on the panel members’ comments, the edited questionnaire was given to thirteen panel members and they were asked to rate the items. Their previous answer and feedback from other expert panel members’ answers were provided. By sending reminder messages, finally, after the completion of one month considered for the third round, 12 questionnaires were collected. At the beginning of the third round’s questionnaire, we provided some explanation about the study again, also how the agreement percent were calculated, which items (elements) were new and which had been edited.

Data analysis

Response rates were calculated for rounds (Table 1). Besides the fact that no specific guidelines exist for acceptable response rates for Delphi studies and that response rate varies from 8 to 100% in studies. Several authors recommend that a 70% response rate for each round is necessary to maintain accuracy [31].

Table 1 Response rates for two rounds of Delphi. The ratio of the filled questionnaire to the people invited to the study was calculated in each round
Full size table

Qualitative data from round two were analyzed by content analysis. About ratings that were assigned to each element, quantitative data analysis was conducted. SPSS version 23 was used. Mean, Standard deviation, and agreement percent (5. Strongly agree plus 4. Agree) for each of the elements calculated (Table 2).

Table 2 Agreement percent, standard deviation, mean, kappa value, and one sample T-Test p- values for elements of structure, process, and outcome in undergraduate nursing education
Full size table

Weighted Kappa coefficient were also calculated by R software to assess responses stability between rounds 2 and 3. For new item that were added to the round three based on panel members comment, One Sample T-Test were calculated (Table 2).

Results

Demographic characteristics

We had 13 members in round 2 and 12 members in round 3. There were seven faculty members and six Ph.D. student. Six of the faculty members had Ph.D. degree too. The average year of experiences were 19.3. The participants’ experiences included: Scientific nursing associations membership; Evaluation and accreditation committee’s membership; Nursing Board membership; membership of the University Nursing Policy Council; membership of the National Nursing Research Network; Education development office (EDO) director; Nursing research centers membership; Hospital and school management; Nursing; and Nursing education.

Rounds statistics

Round 2

According to the consensus level of 75%, the consensus was reached for all the items in round 2.

Round 3

The consensus level of 75% was reached for all the items in round 3 except item number 52 (Student transfer). This item had been added to round 3 based on the comments in round 2.

Weighted kappa

Cohen suggested the Kappa result be interpreted as follows: values ≤ 0 as indicating no agreement and 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41– 0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement [44]. Forty-six items got kappa greater than 0.6 which is a substantial or perfect agreement.

For items number 5, 7, 26, 41, 42, and 54 the kappa coefficient got 0 value that is because at least one of the variables is constant and has zero variance, because of this the kappa could not be calculated. This doesn’t mean that there is no agreement between two rounds answers for these items. These items got a high percentage of agreement and high means, for this reason, we didn’t omit these items.

For items, number 4, 19, and 37, kappa values are 0.4, 0.39, and 0.44 respectively. Sometimes high agreement and low kappa values are the results of the first kappa paradox [45]. Because they had an acceptable level of agreement percent and means, and kappa was also fair or moderate we kept these items too.

One sample T-test

One Sample T-Test was calculated for 5 new items in the third round. Except for item number 17, which T-Test could not be calculated due to zero variance, the p-value was statistically significant for the remaining 4 items (Table 2). For item number 17, because the agreement percent is 100, and the mean is 5, this element was also maintained. For the other 4 items, the mean is also more than three (the maximum value could be 5(.

Final result

According to the percentage of agreement, kappa values, means, and T-test, all the elements except element number 52 (students transfer) are confirmed. Structure’s, process’, and outcome’s key related elements are presented in Fig. 3 (items related to each of these key elements are available via a Supplementary file named “Model”).

Fig. 3
figure 3

Key elements of structure, process and outcome in undergraduate nursing education. These elements were confirmed during Delphi round for undergraduate nursing education

Full size image

In this study, elements of a nursing education institution performance from the perspective of different stakeholders were confirmed by nursing experts and nursing education experts.

Structure

Five main elements were determined for the structure component. According to the definition of the structure component in this model, all the features that nursing education occurs in or with them can be part of the structure of this model. The employees, whether they are involved in education or those who are responsible for other tasks related to an educational institution, can be one of the key elements of structure and undeniably contribute to the institution performance.

Maintaining the human resources standards, along with the training workshops that are held for them, can help improve their performance and ultimately improve the performance of educational institutions. In exchange for the proper performance, reward and payment systems can be a motivation for employees or even a basis for attracting more skillful and competence employees to the institution, and in turn improve the institution performance.

Equipment, guidelines, resources and facilities, along with staff, can help educational institutions achieve their desired goals. Available facilities and equipment, educational spaces, and educational aids can effect on students and staff performance and as a result the total performance of institution.

Although the outcomes related to the students and graduates of an institution can be the functional consequences of an institution, the basic characteristics of the students who enroll in an educational institution can be its structural components. Because these initial specifications can affect the process and ultimately the achieved outcomes. On the other hand, the performance of the institution itself will be effective in attracting students with special characteristics to the institution.

Process

The processes include all the acts of an educational institution, if they are implemented well, they can lead to the desired outcomes by using the existing good structure. To establish desirable performance or improve it, an organization needs different acts either inside the organization or beyond. Each of these acts can help achieve desired outcomes by attracting resources, producing knowledge, presenting knowledge, or introducing the organization to outside world. In this study, the processes included communication, education, evaluation, collaboration, and consultation.

An educational institution with good performance will usually be able to take a step by establishing communication either at the level of the organization or beyond the organization; national and international cooperation in line with its strategic goals, or even the promotion of the nursing field. The organization must establish proper communication and collaboration with employees, stakeholders, other institutions, or related bodies. The educational organization should transfer their valuable experiences for use in appropriate cooperation with the industry, other schools, or other related centers outside the faculty and also use these organizations information and experiences. The school and these organizations should work together to produce products, services, or knowledge in order to achieve their lofty organizational goals and also benefit the country, and even the national and international community.

Sufficient quality education should be provided to students in connection with all their educational and information needs. When necessary, student should have access to appropriate individual consultation, and finally the organization, staff, and students should be evaluated appropriately.

Outcome

Participants expected outcomes included factors related to stakeholders and diverse audiences. They believed that a nursing education institution should contribute to the production of knowledge related to the field of nursing and also affect the image of nursing in a society. From the perspective of the participants in this study, the performance of nursing education institutions will have consequences for their students and graduates, which include a range of issues such as knowledge, skills, higher education, professional competence, and even their interest in the field in which they are studying.

Also, medical centers related to these educational centers can be affected by the performance of the school. Alumni with appropriate skills, training, and education can help improve the performance of healthcare institutions. School and faculty members’ collaborations with these settings can be helpful in different ways.

The results of external evaluations and accreditation conducted on the school can be the result of the performance of the nursing education institution, also the satisfaction of staff, service recipients, and employers is important as they benefit from the performance of the institution.

Discussion

This is a Modified Delphi study that is part of a greater study that aimed to develop undergraduate nursing education key performance indicators. In this part of the study, we aimed to adapt the Donabedian Model for use in nursing education. We tried to specify its elements comprehensively and define each of its three components.

We needed a conceptual framework to get an overview of nursing education institutions in order to measure their performance. By examining the existing frameworks as well as the findings from the previous steps of the study, we realized that we need a framework that can organize our data into three components: structure, process, and outcome. This framework exists in healthcare with the aim of measuring the quality of their performance (Donabedian model). We decided to adapt the Donabedian Model. Its content can be useful to design performance measurement systems or improve performance in education. Many of the frameworks we reviewed were designed for healthcare, and some of the frameworks designed in education were not appropriate or too complex to organize our findings. Donabedian's framework has defined the main components of an organization with a simple yet almost complete design. The relationships between the components are clear. The emphasis of this model is on performance measurement, which was the main goal of our main project and It was consistent with our data.

The main elements obtained for each of the components of the Donabedian model in nursing education included 5 elements for structure |(staff, equipment, guidelines, resources and facilities, students’ demographic and characteristics), 5 elements for process (communication, education, evaluation, cooperation, consultation), and 7 elements for outcome component (knowledge development, nursing image, alumni’s outcome, students’ outcomes, related clinical settings performance, accreditation and evaluation results, satisfaction.

In this study, we adapted a healthcare model for use in an educational context. In a study by Selvik et al. also a health care model were served well as a conceptual framework for analyzing the benefits of educational efforts [46]. In Custis et al.’s study, through the use of Donabedian’s framework, the healthcare informatics students completed essential learning activities [47]. In another study, the structure and process of surgical oncology patient education within one integrated health system were assessed using the Donabedian framework [48]. Avci in his dissertation utilized Donabedian’s three approaches to formulate quality standards and indicators in bioethics education [30]. The results of these studies confirm the fact that the models designed in certain contexts can be generalized and used in other contexts. Conceptual frameworks can be used for different purposes in education. Considering the purpose we have of using a conceptual framework, we can use the framework designed in other disciplines and there will be no need to design a new framework. We needed a conceptual framework that could clarify all aspects of a nursing education institution and show us a clear and comprehensive picture of this interesting phenomenon. so that we can use this image for evaluation of these institutions. For this reason, the use of Donabedian's framework, which is designed to evaluate the performance of institutions, seems appropriate. It was only necessary to adapt this framework in the new context.

A study by Anderson et al. discusses that Joint Commission on Accreditation of Healthcare Organizations (JCAHO) framework can be adapted to evaluate the quality of nursing programs and it could be applied in an academic setting to improve student and organizational outcomes [49]. In a study by Lippe and Carter, Stufflebeam's Context, Input, Process, and Product model (CIPP) was utilized to evaluate end of life care content in a nursing education program. They concluded that model serves as a valuable guide for curriculum evaluation [50]. As these studies concluded, the conceptual frameworks of other disciplines could be well used in nursing education.

We used the Delphi technique as the means of model adaptation. The Delphi method is widely used to adapt or develop models in different contexts. In Boyer’s study, a modified Delphi method supported the adaptation and validation of a nursing competencies framework [51]. In another study, the Delphi method was used to develop a process model of key elements in the implementation process [52]. In Stockton et al. study, to identify key elements of a framework for the adaptation of specialist community-based child and family health, a Delphi study was conducted [53]. There are numbers of studies that have used different kinds of the Delphi techniques to develop or adapt models and frameworks. It seems that this method can be proper for achieving the goals of such studies. Because the first round of this Delphi study is based on interviews, focus groups, and literature review we used the modified Delphi technique as the study method [43].

We specified elements of each of the Donabedian model’s three components in nursing education. The first component is structure. In our study, staff (academic and non-academic), equipment, guidelines, resources and facilities, and students’ demographics and characteristics are key elements of this component in nursing education. In a study by Cassiani et al., in the designed instrument based on the Donabedian model to assess the situation of nursing education, the items related to structure in the instrument assessed the numbers of students and faculty, school policies or guidelines, classrooms, and laboratories [54]. In another study, structure expectations for critical care nurse education included a standard core curriculum, clinically credible academic staff, and courses compliant with a higher education framework. Published workforce standards and policies were important structures for the practice learning environment [55]. In a study by Jarrar et al., the resources (materials, facilities, and humans) as well as the organizational structure, policies, and procedures reflect the structural quality [56]. The findings of these studies regarding structure is compatible with our study findings. The findings of these studies together with our study show that human resources, equipment, facilities, physical space and guides are among the elements that are considered in relation to an educational institution.

In our study, process key elements include communication; education; evaluation; cooperation; and consultation. In Cassiani et al.’s study, the items related to the process included general professional competencies, curriculum model and teaching/learning strategies, clinical experiences, nursing program evaluation, and student evaluation [54]. This shows that education and evaluation are among key elements in both studies.

Based on our findings, outcome key elements include knowledge development; nursing image; alumni’s outcome; students’ outcome; related medical centers’ performance; accreditation and evaluation results; and satisfaction. In Cassiani et al.’s study, outcomes or results focused on the place of employment of the graduates, and whether they were integrating competencies related to Universal Health in their work [54].

The findings of this study can provide a framework for policymakers or nursing managers to systematically evaluate their educational institutions. This adapted model provides a comprehensive set of nursing education institution elements and helps to design a performance measurement or evaluation system. These results can also be a basis for making this model useful in other nursing courses or other fields. We believe this study may be the first study that has comprehensively adapted the Donabedian Model for use in nursing education.

The faculty can achieve a more comprehensive view of the faculty and its performance by using the elements identified in this study in internal evaluations. During the evaluation, taking into account the important areas from the point of view of the main stakeholders of the college and identifying and fixing the weak points of these areas, it can gain a higher position in today's competitive market.

The findings of the study by presenting in national or international conferences and scientific assemblies can help policy makers in nursing and similar clinical disciplines so that they can use the ideas presented in this study and the improvement model for performance evaluation and performance improvement systems. The quality of Donabedian should be helped and by integrating and completing the findings of this study and other studies, they should take steps to improve the performance of their institution and the nursing profession.

This study was carried out with the help and participation of various groups of nursing education stakeholders, as well as the use of numerous texts and websites of various nursing schools around the world. However, it seems that consultation with other stakeholders and professionals in the international fields of nursing education, or newer fields related to higher education such as sustainable development, should be given more attention.

Also, this study was done with an emphasis on nursing education at the undergraduate level, and probably other elements in graduate level of education may need to be considered.

Study limitations and strengths

  • One of the limitations of our study was that the response rate in the second round was 65%. Although it is better to keep the response rate at least 70% in all rounds, but because the response rate was significant in the third round, this rate can be attributed to the reluctance of people to participate in the study or ignoring the invitation to participate from experts who were invited via email, Because the response rate in the first round for the experts who were invited in person was 100%.

  • Another limitation is that because 5 items were new in round 3 we couldn’t calculate kappa for them. Continuing the Delphi rounds for these 5 items might decrease the response rate, and it also demanded more time.

  • One of the strengths of this study is the using different groups of stakeholders’ opinion in designing the initial questionnaire. Also, while reviewing websites of schools and colleges around the world, their experiences have been used in order to design the questionnaire.

  • Calculating and reporting Kappa in order to check the stability between Delphi rounds and to decide on the completion of Delphi is one of the strong points of the methodology of this study.

Conclusion

Different elements of a nursing education institution at the bachelor's level were determined. The results of this study can help related bodies to develop and implement a comprehensive and systematic evaluation. These results can also be a basis for making this model useful in other nursing courses or other fields.

Availability of data and materials

The datasets used and/or analyzed during the current study are available from the corresponding author on reasonable request.

References

  1. Giménez-Espert MdC, Prado-Gascó V, Soto-Rubio A. Psychosocial Risks, Work Engagement, and Job Satisfaction of Nurses During COVID-19 Pandemic. Front Public Health. 2020;8:566896. https://doi.org/10.3389/fpubh.2020.566896.

  2. Kieft RAMM, de Brouwer BBJM, Francke AL, Delnoij DMJ. How nurses and their work environment affect patient experiences of the quality of care: a qualitative study. BMC Health Serv Res. 2014;14(1):249. https://doi.org/10.1186/1472-6963-14-249.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Institute of Medicine (US) Committee on the Robert Wood Johnson Foundation Initiative on the Future of Nursing, at the Institute of Medicine. The Future of Nursing: Leading Change, Advancing Health. Washington (DC): National Academies Press (US); 2011. p. 700. https://doi.org/10.17226/12956.

  4. Jasemi M, Purteimor S, Zabihi RE, Pak MHM, Eghtedar S. Nurses’ Strategies for Conscience-based Care Delivery: A Qualitative Study. Indian J Palliat Care. 2019;25(4):517–22. https://doi.org/10.4103/IJPC.IJPC_65_19.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Bvumbwe T, Mtshali N. Nursing education challenges and solutions in Sub Saharan Africa: an integrative review. BMC Nurs. 2018;17(1):3. https://doi.org/10.1186/s12912-018-0272-4.

    Article  PubMed  PubMed Central  Google Scholar 

  6. Farahsa S, Tabrizi JS. How evaluation and audit is implemented in educational Organizations? A Systematic Review. Res Dev Med Educ. 2015;4(1):3–16. https://doi.org/10.15171/rdme.2015.002.

    Article  Google Scholar 

  7. Söderlind J, Geschwind L. Making sense of academic work: the influence of performance measurement in Swedish universities. Policy Rev Higher Educ. 2019;3(1):75–93. https://doi.org/10.1080/23322969.2018.1564354.

    Article  Google Scholar 

  8. Matthiesen V, Wilhelm C. Quality Outcomes and Program Evaluation in Nursing Education: An Overview of the Journey. Qual Manag Health Care. 2006;15(4):279–84. https://doi.org/10.1097/00019514-200610000-00010.

    Article  PubMed  Google Scholar 

  9. Fynn JF, Hardeman W, Milton K, Jones AP. A scoping review of evaluation frameworks and their applicability to real-world physical activity and dietary change programme evaluation. BMC Public Health. 2020;20(1):1000. https://doi.org/10.1186/s12889-020-09062-0.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Ahmadi Chenari H, Zakerimoghadam M, Baumann SL. Nursing in Iran: Issues and Challenges. Nurs Sci Q. 2020;33(3):264–7. https://doi.org/10.1177/0894318420920603.

    Article  PubMed  Google Scholar 

  11. Halstead JA. Fostering Innovation in Nursing Education: The Role of Accreditation. Teach Learn Nurs. 2020;15(1):A4–5. https://doi.org/10.1016/j.teln.2019.10.003.

    Article  Google Scholar 

  12. Duarte N, Vardasca R. Literature Review of Accreditation Systems in Higher Education. Education Sciences. 2023;13(6):582. https://doi.org/10.3390/educsci13060582.

    Article  Google Scholar 

  13. Abubakar A, Hilman H, Kaliappen N. New tools for measuring global academic performance. Sage Open. 2018;8(3) https://doi.org/10.1177/2158244018790787

  14. Englund H, Ludvigsen E. Performance measurement systems as management control in R&D organizations: A case study [thesis]. Karlskrona: Blekinge Tekniska Högskola; 2015. p. 62.

  15. Scrivens E. Assessing the value of accreditation systems. Eur J Pub Health. 1997;7(1):4–8. https://doi.org/10.1093/eurpub/7.1.4.

    Article  Google Scholar 

  16. Jaafaripooyan E, Agrizzi D, Akbari-Haghighi F. Healthcare accreditation systems: further perspectives on performance measures. Int J Qual Health Care. 2011;23(6):645–56. https://doi.org/10.1093/intqhc/mzr063.

  17. Jahanmehr N, Rashidian A, Khosravi A, Farzadfar F, Shariati M, Majdzadeh R, et al. A Conceptual Framework for Evaluation of Public Health and Primary Care System Performance in Iran. Glob J Health Sci. 2015;7(4):341–57. https://doi.org/10.5539/gjhs.v7n4p341.

    Article  PubMed  PubMed Central  Google Scholar 

  18. Yalagi MS, Ranjitha K, Krishnan K. Importance of key performance indicators measurement. Int J Adv Res Sci Eng Technol. 2016;2(1):328–36. https://doi.org/10.31695/IJERAT.

    Article  Google Scholar 

  19. Broshkov M, Forostian O, Kichuk Y, Liapa M, Horbashevska M, Kakhiani Y. Management of key performance indicators by heads of higher education institutions. Int J Manag. 2020;11(5):286–98. https://doi.org/10.34218/IJM.11.5.2020.028.

    Article  Google Scholar 

  20. Advisory Committee on Official Statistics. Good practice guidelines for the development and reporting of indicators. In: Third World Forum on ‘Statistics, Knowledge and Policy’Charting Progress, Building Visions, Improving Life; 27-30 October 2009; Busan, Korea. Wellington, New Zealand: Statistics New Zealand; 2009.

  21. Klassen A, Miller A, Anderson N, Shen J, Schiariti V, O’Donnell M. Performance measurement and improvement frameworks in health, education and social services systems: a systematic review. Int J Qual Health Care. 2010;22(1):44–69.

    Article  PubMed  Google Scholar 

  22. Santry HP, Strassels SA, Ingraham AM, Oslock WM, Ricci KB, Paredes AZ, et al. Identifying the fundamental structures and processes of care contributing to emergency general surgery quality using a mixed-methods Donabedian approach. BMC Med Res Methodol. 2020;20(1):247.

    Article  PubMed  PubMed Central  Google Scholar 

  23. Gentry SV, Powers EFJ, Azim N, Maidrag M. Effectiveness of a voluntary family befriending service: a mixed methods evaluation using the Donabedian model. Public Health. 2018;160:87–93.

    Article  CAS  PubMed  Google Scholar 

  24. Lee LYK, Holroyd E. Evaluating the effect of childbirth education class: a mixed-method study. Int Nurs Rev. 2009;56(3):361–8.

    Article  PubMed  Google Scholar 

  25. Moore L, Lavoie A, Bourgeois G, Lapointe J. Donabedian’s structure-process-outcome quality of care model: Validation in an integrated trauma system. J Trauma Acute Care Surg. 2015;78(6):1168–75.

    Article  PubMed  Google Scholar 

  26. Ibn El Haj H, Lamrini M, Rais N. Quality of care between Donabedian model and ISO9001V2008. Int J Qual Res. 2013;7(1):17–30.

  27. Botma Y, Labuschagne M. Application of the Donabedian quality assurance approach in developing an educational programme. Innov Educ Teach Int. 2019;56(3):363–72.

    Article  Google Scholar 

  28. Rowan MS, Rukholm E, Bourque-Bearskin L, Baker C, Voyageur E, Robitaille A. Cultural Competence and Cultural Safety in Canadian Schools of Nursing: A Mixed Methods Study. Int J Nurs Educ Scholarsh. 2013;10(1):1–10.

    Article  Google Scholar 

  29. Jette DU, Nelson L, Palaima M, Wetherbee E. How do we improve quality in clinical education? examination of structures, processes, and outcomes. J Physical Therapy Educ. 2014;28:6–12.

    Article  Google Scholar 

  30. Avci E. Evaluating the effectiveness of bioethics education through quality standards and indicators. Int J Ethics Educ. 2022;7(1):5–19.

    Article  Google Scholar 

  31. Keeney S, McKenna H, Hasson F. How to Get Started with the Delphi Technique. In: The Delphi Technique in Nursing and Health Research. United Kingdom: Wiley-Blackwell; 2011. p. 43–68. https://doi.org/10.1002/9781444392029.

  32. Jr King PR, Beehler GP, Donnelly K, Funderburk JS, Wray LO. A practical guide to applying the delphi technique in mental health treatment adaptation the example of enhanced problem-solving training (E-PST). Prof Psychol Res Prac. 2021;52(4):376–86.

    Article  Google Scholar 

  33. Niederberger M, Spranger J. Delphi technique in health sciences: a map. Front Public Health. 2020;8:457.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Ameh S, Gómez-Olivé FX, Kahn K, Tollman SM, Klipstein-Grobusch K. Relationships between structure, process and outcome to assess quality of integrated chronic disease management in a rural South African setting: applying a structural equation model. BMC Health Serv Res. 2017;17(1):229.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Kajonius PJ, Kazemi A. Structure and process quality as predictors of satisfaction with elderly care. Health Soc Care Community. 2016;24(6):699–707.

    Article  PubMed  Google Scholar 

  36. Alizadeh S, Sedigh Maroufi S, Sohrabi Z, Norouzi A, Dalooei R, Ramezani G. Large or Small Panel in the Delphi Study? Application of Bootstrap Technique. J Evol Med Dent Sci. 2020;9:1267–71.

    Article  Google Scholar 

  37. Latif RB, Dahlan A, Abdul Mulud Z, Nor M. The Delphi Technique as a Method to Obtain Consensus in Health Care Education Research. Educ Med J. 2017;9:89–102.

    Article  Google Scholar 

  38. Etikan I, Musa SA, Alkassim RS. Comparison of convenience sampling and purposive sampling. Am J Theor Appl Stat. 2016;5(1):1–4.

    Article  Google Scholar 

  39. Profillidis VA, Botzoris GN. Executive Judgment, Delphi, Scenario Writing, and Survey Methods. In: Modeling of transport demand – analyzing, calculating, and forecasting transport demand. Amsterdam: Elsevier; 2019. p. 125–61. https://doi.org/10.1080/01441647.2019.1635226.

  40. Nasa P, Jain R, Juneja D. Delphi methodology in healthcare research: How to decide its appropriateness. World J Methodol. 2021;11(4):116–29.

    Article  PubMed  PubMed Central  Google Scholar 

  41. Belton I, MacDonald A, Wright G, Hamlin I. Improving the practical application of the Delphi method in group-based judgment: A six-step prescription for a well-founded and defensible process. Technol Forecast Soc Chang. 2019;147:72–82.

    Article  Google Scholar 

  42. Trevelyan E, Robinson N. Delphi methodology in health research: How to do it? Eur J Integr Med. 2015;7:423–8.

    Article  Google Scholar 

  43. Massaroli A, Martini JG, Lino MM, Spenassato D, Massaroli R. The delphi method as a methodological framework for research in nursing. Texto & Contexto-Enfermagem. 2018;26(4):e1110017. https://doi.org/10.1590/0104-07072017001110017.

  44. McHugh ML. Interrater reliability: the kappa statistic. Biochemia medica. 2012;22(3):276–82.

    Article  MathSciNet  PubMed  PubMed Central  Google Scholar 

  45. Bexkens R, Claessen FM, Kodde IF, Oh LS, Eygendaal D, van den Bekerom MP. The kappa paradox Shoulder elbow. 2018;10(4):308.

  46. Selvik JT, Abrahamsen EB, Moen V. Conceptualization and application of a healthcare systems thinking model for an educational system. Stud High Educ. 2022;47(9):1872–89.

    Article  Google Scholar 

  47. Custis LM, Hawkins SY, Thomason TR. An innovative capstone health care informatics clinical residency: Interprofessional team collaboration. Health Informatics J. 2016;23(1):69–79.

    Article  PubMed  Google Scholar 

  48. Bartholomew CS. An assessment of surgical oncology patient education [thesis]. Bozeman, Montana: Montana State University; 2018. p. 51.

  49. Anderson P, Cuellar N, Rich K. Performance improvement in higher education: adapting a model from health care agencies. J Nurs Educ. 2003;42(9):416–20. https://doi.org/10.3928/0148-4834-20030901-08.

  50. Lippe M, Carter P. Using the CIPP model to assess nursing education program quality and merit. Teach Learn Nurs. 2018;13(1):9–13. https://doi.org/10.1016/j.teln.2017.09.008.

    Article  Google Scholar 

  51. Boyer L, Pepin J, Dubois S, Descôteaux R, Robinette L, Déry J, et al. Adaptation and validation of a nursing competencies framework for clinical practice on a continuum of care from childhood to adulthood: A Delphi study. Nurse Educ Today. 2020;93:104530. https://doi.org/10.1016/j.nedt.2020.10453052.

    Article  PubMed  Google Scholar 

  52. Parker G, Kastner M, Born K, Berta W. Development of an implementation process model a Delphi study. BMC Health Serv Res. 2021;21(1):558. https://doi.org/10.1186/s12913-021-06501-5.

    Article  PubMed  PubMed Central  Google Scholar 

  53. Stockton DA, Fowler C, Debono D, Travaglia J. Adapting community child and family health service models for rural and other diverse settings: A modified Delphi study to identify key elements. Health Soc Care Community. 2022;30(6):e6145–62. https://doi.org/10.1111/hsc.14052.

    Article  PubMed  PubMed Central  Google Scholar 

  54. Cassiani SHDB, Wilson LL, Mikael SDSE, Peña LM, Grajales RAZ, McCreary LL, et al. The situation of nursing education in Latin America and the Caribbean towards universal health. Rev Lat Am Enfermagem. 2017;25:2913. https://doi.org/10.1590/1518-8345.2232.2913.

    Article  Google Scholar 

  55. Gullick J, Lin F, Massey D, Wilson L, Greenwood M, Skylas K, et al. Structures, processes and outcomes of specialist critical care nurse education: An integrative review. Aust Crit Care. 2019;32(4):331–45. https://doi.org/10.1016/j.aucc.2018.09.00756.

    Article  PubMed  Google Scholar 

  56. Jarrar Mt, Mohamed RB, Al-Bsheish M, Albaker W, Alumran A, Alomran AK. Students’ perception of quality of learning experience (structure, process and outcome) discipline versus problem based medical curriculum and the mediation role of process Quality. Healthcare (Basel). 2022;10(8):1584. https://doi.org/10.3390/healthcare10081584. (editors).

Download references

Acknowledgements

The present article was extracted from the Ph.D. thesis written by the first author and approved by the Research Vice-chancellor of Tabriz University of Medical Sciences. Also, the authors wish to acknowledge all the participants of this study.

Funding

This research was funded by Tabriz University of Medical Sciences, grant number 1399/12/27–66063. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript; or in the decision to publish the results.

Author information

Authors and Affiliations

  1. Department of Pediatric Nursing, School of Nursing and Midwifery, Tabriz University of Medical Sciences, Tabriz, Iran

    Marjan Ghofrani

  2. Department of Pediatric Nursing, School of Nursing and Midwifery, Shahid Beheshti University of Medical Sciences, Tehran, Iran

    Leila Valizadeh

  3. Department of Medical Surgical Nursing, School of Nursing and Midwifery, Shahid Beheshti University of Medical Sciences, Tehran, Iran

    Vahid Zamanzadeh

  4. Medical Education Research Center, Health Management and Safety Promotion Research Institute, Tabriz University of Medical Sciences, Tabriz, Iran

    Akram Ghahramanian

  5. Department of Medical Surgical Nursing, School of Nursing and Midwifery, Tabriz University of Medical Sciences, Tabriz, Iran

    Akram Ghahramanian

  6. Department of Health Service Management, School of Management and Medical Informatics, Tabriz University of Medical Sciences, Tabriz, Iran

    Ali Janati

  7. Nursing & Midwifery Care Research Center, School of Nursing & Midwifery, Isfahan University of Medical Sciences, Isfahan, Iran

    Fariba Taleghani

Authors
  1. Marjan Ghofrani
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Leila Valizadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Vahid Zamanzadeh
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Akram Ghahramanian
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Ali Janati
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Fariba Taleghani
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

MG, LV, VZ, and AG designed the study. Data were collected, analyzed, and interpreted by the contribution of MG, LV, VZ, AG, AJ, and FT. Manuscript was drafted by MG and revised by LV.

Corresponding author

Correspondence to Leila Valizadeh.

Ethics declarations

Ethics approval and consent to participate

The study was conducted in accordance with the Declaration of Helsinki, and approved by the Ethics Committee of Tabriz University of Medical Sciences (IR.TBZMED.REC.1399.1128). Informed consent was obtained from all subjects involved in the study.

Consent for publication

Not applicable.

Competing interests

The authors declare no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Ghofrani, M., Valizadeh, L., Zamanzadeh, V. et al. Adapting the Donabedian model in undergraduate nursing education: a modified Delphi study. BMC Med Educ 24, 202 (2024). https://doi.org/10.1186/s12909-024-05187-7

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s12909-024-05187-7

Keywords

BMC Medical Education

ISSN: 1472-6920

Contact us