Volume 10, Issue 1 (Jan-Mar 2021)
JCHR 2021, 10(1): 68-82 |
Back to browse issues page
Download citation:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
BibTeX | RIS | EndNote | Medlars | ProCite | Reference Manager | RefWorks
Send citation to:
Rabee mandejin M R, Janbazi S. Designing a Comprehensive Evaluation Model for Health Reform Plan in Iran. JCHR 2021; 10 (1) :68-82
URL: http://jhr.ssu.ac.ir/article-1-584-en.html
URL: http://jhr.ssu.ac.ir/article-1-584-en.html
1- 1. Department of Public Administration, Faculty of Management, Islamic Azad University, Central Tehran Branch, Tehran, Iran , m.rabiee2012@yahoo.com
2- 2. Department of Public Policy, Islamic Azad University of Hamedan, An independent researcher, Hamedan, Iran
2- 2. Department of Public Policy, Islamic Azad University of Hamedan, An independent researcher, Hamedan, Iran
Full-Text [PDF 981 kb]
(753 Downloads)
| Abstract (HTML) (1565 Views)
Full-Text: (373 Views)
Abstract
Introduction: The present study aimed to design a comprehensive model to evaluate the health system reform plan so the indicators of this model can lead to the health reform plan's effectiveness. Hence, this plan's effective factors, including policymakers and managers, suppliers, and clients, were identified and explained.
Methods: According to the practical purpose and nature of the research, the combined/mixed research method was used, in 2018-2019. For the qualitative phase, the Delphi technique was used in exploratory interviews, and quantitative methods were used to collect data through a questionnaire. In the quantitative phase, 400 employees of 10 hospitals were selected by stratified random sampling; and in the second section, 300 clients were randomly selected in two hospitals with the highest and lowest evaluation scores in the previous phase. Content analysis method and Delphi technique were used for classification and analysis of qualitative data, and SPSS Ver.22, AMOS Ver.24 and Expert Choicev24 were used for data description and dimensions and indices' weight extraction. Dimensions under consideration include the financial, social responsibility, growth and learning, citizens and client, and internal process fields.
Results: The components with the highest and lowest weight and impact on the effectiveness of the plan were identified in the following dimensions: financial field, Social responsibility field, growth and learning field, citizens and client field, internal process field, including weights 0.266, 0.244, 0.202, 0.164, 0.124, respectively.
Conclusion: Given the results, the components, indicators, dimensions, and levels of the model designed, the comprehensive evaluation of the Health System Reform Plan in Iran (CEHSRP-IR) with the normal mean was confirmed by the statistical population, and this model can be used in all organizations implementing this plan.
Keywords: Health care reform, Health plan, Integrated healthcare systems, Performance evaluation
Introduction
The health system reform plan with three approaches of financial protection of people, equity in access to health services, and improving quality of services has been put on the government's agenda to implement health objectives in the 5th Five Year Development Plan. On March 30, 2014, this plan was implemented according to Note 21 of the 2014 Budget Bill. Part of the project's funding was funded by reforming the price of goods, services, and subsidies on bread and electricity (1).
Investigating performance and efficiency as one of the effective factors on the effectiveness of a healthcare reform plan can help policymakers and executives better identify its strengths and weaknesses and strive to improve them (2). After five years of implementing this plan, there is a need for conducting scientific research with a precision tool on the evaluation of the project implementation. Designing a comprehensive model for a healthcare reform plan in the country is considered the first step in this field, and the present study seeks to fill the theoretical vacuum of the lack of a comprehensive model of healthcare reform plan assessment.
According to the studies conducted in this field, unfortunately, no comprehensive approach is found in evaluating this plan. Only one aspect of the issues related to this plan has been addressed by each of the researchers who have taken measures in this field, and the approach used in most studies has been in a simple framework (measuring the attitudes of clients and practitioners in reform plans, etc.). This vacuum is as important as the Deputy Minister of Health, who is also responsible for the implementation of the plan, and has pointed out the need for its implementation in a clear manner (3), that measures and results of the implementation of the health reform plan should be evaluated in the language of statistics and based on recognized international indicators (3). This study aimed to address four major challenges in performance evaluation included: Lack of comprehensiveness in evaluation criteria at different levels, One-dimensional view of evaluation and lack of comprehensiveness of evaluation dimensions, Lack of attention and pay attention to organizations' social responsibility in the evaluation criteria and inconsistency of performance indicators and strategic goals (4). In this regard, designing a comprehensive evaluation model of health reform plan in Iran was selected for the present study in order to cover part of this basic need, using the Extended Balanced Scorecard (BSC), considering the local, climatic, cultural, and structural conditions of this plan (project) and using the Willcocks'theoretical framework. This study aimed to design a comprehensive model of health reform plan evaluation in Iran with an Extended Balanced Scorecard approach so that relevant organizations can implement their strategy based on strategic goals and comprehensively evaluate its results.
Wu et al. during a study conducted a balanced scorecard based on a valid comprehensive measurement system by investigating the employees working in the hospital in western China. In the present study, 216 professional employees received valid answers through questionnaires. According to the results, ranking the importance of the Balanced Scorecard views is related to a serious concern in future planning and management. The establishment of a balanced scorecard measurement system for integrated health care organizations in China (5) has been recommended in this study. Also, Farooq Salman et al. in a study (case study: Oman's Suhar University) investigated the effectiveness of using the Balanced Scorecard (BSC) as a management tool in the performance evaluation and thus the quality of university services. According to their results, there is a strong relationship between the strategic roadmap and the four BSC perspectives. According to the present study results, the concept of BSC can be introduced to an educational institution to fill the gap created between creating a strategy and implementation in the same way (6). Leksono et al., in a study, investigated the Balanced Scorecard (BSC)-based sustainable supply chain and, using expert opinions, extracted 29 indices and examined their effects on BSC dimensions and finally, mapped the strategy map by examining the relationships of indices (7).
Rahmat et al. concluded that environmental and social standards and resource efficiency could prioritize performance indicators and clarify the strategy map in a detailed way based on Extended BSC in the field of Indonesia's manufacturing sector while investigating the environmental and social aspects of the scorecard (8). Gutacker& Street, in a study on public treatment sectors in the UK entitled "multidimensional performance evaluation of public sector organizations using dominant criteria," concluded that we need to find significant correlations between goals. They concluded that ignoring correlations can lead to inaccurate performance evaluation (9). Mehralian et al. in a study entitled "TQM and organizational performance using a balanced scorecard approach" on Iranian pharmaceutical industries, have concluded that the implementation of TQM has a positive and significant effect on the BSC and its four perspectives. They believed that managers should strongly use TQM methods to achieve their strategic goals due to the strong relationship between TQM and all four organizational performance perspectives (BSCs) (10). Anjomshoae et al., in a study entitled "Towards a Dynamic Balanced Scorecard Model for Managing the Performance of Humanitarian Aid Organizations," explained the sustainable, balanced scorecard and thereby introduced a conceptual model with a BSC approach to humanitarian relief organizations (11). Kailash et al., in a study entitled "Development of Balanced Scorecard for Health Care Using Interpretive Structural Modeling and Analytic Network Process," aimed to establish a Balanced Scorecard-based Performance Assessment Framework for Health Care Systems, identified key indicators of performance and their relationship to the health care system. They used an integrated interpretive structural modeling approach and an analytical networking process to develop a balanced scorecard (12).
In a study entitled "Investigating the effect of Health CareReformPlan on Hospital Performance Indicators in Lorestan Province," Dadgar et al. concluded that implementing the above plan has led to making a positive change in performance indicators of the hospital. Therefore, it is recommended to pay attention to the continuous improvement of the current process and the continuation of the plan's implementation (13). A study was carried out to investigate the effect of subsidy targeting and health reform plan on equity index in household health expenditures. The results showed that the incomplete implementation of the first phase of the Subsidy Targeting Law has had no positive effect on justice in the health system's financing due to decreasing health system financing after implementation of subsidy targeting law. On the other hand, no significant change has made in the financing of the health system in terms of the inequality and injustice with implementing the second phase of subsidies targeting (Paragraph b, Article (34) of the 5th Five Year Development Plan and the Health ReformPlan as a financial reform of the health system). So, it seems that the main factors affecting the reduction of direct payments by the people and thus the reduction of income inequalities which must be considered in health system reforms include the implementation of the laws in a precise and correct way, maximizing the protection of laws, timely financing and greater participation of government in financing the health system (14). A study was carried out to examine health insurance organizations' performance in cross-sector collaboration with the Ministry of Health and Medical Education to implement the health system reform plan effectively. The results showed a need for clarification and creating legal requirements on insurance organizations' obligations and subsequently "monitoring and control by the health system administrator, Ministry of Health (15). According to the results of another study that was carried out to examine the importance of patient satisfaction in the health system reform plan, centers' attention to patient satisfaction is relatively favorable. However, there is a need for additional efforts to achieve this goal reform plan. It seems that basic review is required to handle complaints and suggestions on the forms used to satisfy the reform plan (16). The results of a study on analyzing the key indicators of performance in the field of human resources showed that key indicators of performance are often used as a "value" and are linked to organizational strategy in order to measure activities such as the effectiveness of leadership method in the development of the organization, employment, services and satisfaction (17). According to the results of a study that aimed to analyze the way used to choose the policy interventions and decision making of Iranian health system policymakers (1979-2014) and analyze the pattern of decision-making and policy thinking, the decision made to apply at least one or more important and broad policy interventions from a lifelong perspective is considered as an important principle considered during the tenure of the health policymakers involved in the country's health system. Based on this view, a policy is made based on lexical rationalism patterns, garbage-can model, and open decision making (Open decision making is an approach to arriving at actionable agreements through participatory practices.) and decision-making during a crisis. Also, the health implications obtained can be justified by the models used (18). According to the results of a study that aimed to measure the way used to implement health reform plans from experts who monitored the treatment, the indicators evaluated in relation to each plan did not materialize completely, but they were fairly favorable. Given that evaluations have been made at different time intervals from the start of each plan, a small amount of failure to realize can be due to several reasons, including inadequate training, shortage of workforce, lack of preparation of some infrastructures, such as increased numbers of the patients referring to centers, and increasing burden of problems in health centers in the early stages of plans. Finally, the present study has recommended re-evaluating these indicators' utility in the second round of visits to the medical centers compared to the first round of visits (19).
Based on the mentioned experiences and considering the framework of this research that was mentioned in the expression of the problem, we seek the goal of four important challenges in evaluating performance under the headings: lack of comprehensiveness of evaluation criteria at different levels, one-dimensional view of evaluation and lack of comprehensiveness of the dimensions of evaluation, lack of attention and consideration of social responsibility of organizations in evaluation criteria and lack of alignment of performance indicators and strategic goals (Bozorg Haddad, 2018, 13-13). In this direction, to address the challenges of a one-dimensional view of evaluation and the lack of comprehensiveness and imbalance of the evaluation system, as well as the mismatch of performance indicators and strategic goals of a balanced scorecard with four financial dimensions, employees, internal processes and growth and learning and To solve the problem of lack of attention and to consider the social responsibility of the project in the evaluation criteria of the scorecard developed with the fifth dimension under the title of environmental (social responsibilities) and to solve the problem of lack of comprehensiveness of evaluation criteria at different levels, using the framework provided By Wilcox and its generalization in line with the dimensions of the national document of general health system policies, four levels of evaluation (including policy makers and strategic managers, employees, service providers and suppliers) are identified and classified. Therefore, the main purpose of this study was to design a comprehensive model for evaluating the health transformation plan in Iran so that the relevant organizations can implement their strategy based on strategic goals and evaluate the results comprehensively and correct their mistakes.
Methods
This study is considered descriptive and cross-sectional research that was carried out during 2018-2019. A mixed-method (qualitative and quantitative) was used in the present study due to its purpose and nature. To carry out this study, a relatively comprehensive understanding was first achieved in relation to how to evaluate the health system reform plan in Iran by studying previous research records, and a preliminary framework was provided on this basis.
In the qualitative phase, purposive sampling and semi-structured interviews of 15 experts, the Delphi method was used to evaluate and extract 70 indicators of comprehensive health system reform plan evaluation in 5 dimensions of Extended Balanced Scorecard (BSC and there was a consensus on it. Then, weights of dimensions and indicators and their priorities were determined using paired comparisons and SPSS and Expert Choice software. In the quantitative phase, the simple sampling method was used to fit the designed model adaptively, and the views of 400 health care reform plan providers in 10 teaching and non-teaching hospitals (two independent groups) in Shahid Beheshti University of Medical Sciences were obtained.
The sample size in qualitative research is the sample size that adequately answers the research questions. Therefore, after separating the levels, a total of 17 people were selected from among them to the extent of adequacy (saturation) by snowball method from experts at each level in whom the role of performance was more noticeable and open. In the quantitative part of the research, according to the statistical population and using Cochran's formula, 400 employees of 10 hospitals of Shahid Beheshti University of Medical Sciences were selected to evaluate the health transformation plan by stratified random sampling. Also, 300 patients (project recipients) in two hospitals of Shahid Beheshti University of Medical Sciences were selected by simple random sampling method.
Cochran's formula
n = sample size N = statistical population size
z = Percentage of acceptable standard reliability error
p = Percentage of population without definite attribute
q (1-p) = percentage of the population without a definite attribute
d= Degree of probability or probable desired accuracy
In the first stage, the study of research literature, dimensions, components, and indicators derived from theoretical foundations and research conducted in the field of subject and exploratory studies became the basis for identifying the conceptual model. By conducting theoretical studies and using the content analysis method, the initial conceptual model was presented. In other words, after gaining the necessary knowledge of the scientific bases and existing models and classifying and analyzing them using the content analysis method, the main dimensions, components and indicators were identified. In several stages of explorative interviews with 17 experts, the dimensions, components, and indicators were identified and corrected by the Delphi technique. After final corrections and review of validity and confirmation, the basis of investigations and field and experimental studies and analyses was provided.
In the second stage, which was a qualitative study based on the Delphi method, in two parts and the research area -Shahid Beheshti University of Medical Sciences including five educational hospitals (i.e., Imam Hossein Hospital, Loghman Hospital, Tajrish Martyrs Hospital, Modares Hospital, and Taleghani Hospital) and five non-educational hospitals (i.e., Imam Khomeini Hospital in Firoozkooh, 3rd Shaban Hospital in Damavand, Anonymous Martyrs Hospital, Zaeem Hospital in Pakdasht and Pakdasht Martyrs Hospital), the data was used for the comprehensive evaluation of the health reform plan based on comments of the statistical population. After evaluating the plan at the level of 10 hospitals by the servers of the plan, the plan was judged and evaluated in two hospitals (i.e., Modares and Zaeem Pakdasht hospitals) terms of clients' comments, and the relevant statistical analyzes were presented. Therefore, at this stage, following the components of the comprehensive evaluation model of the health reform plan in Iran through in-depth and exploratory interviews with scientific and executive experts who were purposefully selected, the necessary qualitative data was collected. Using content analysis as a research technique, the concepts, categories, and main and secondary factors were identified and analyzed. These concepts, factors, and categories were considered the basis for developing the tool (questionnaire) to achieve effective factors and recognize the dimensions and components, providing a model for presenting a comprehensive evaluation model of Iran's health reform plan. This section resulted in a package of basic dimensions and components and indicators of the comprehensive evaluation model of Iran's health reform plan. In the qualitative part of the research, the researcher-designed semi-structured questionnaire was used. The structured questionnaire was used to collect information in the 2nd to 4th phases of the Delphi technique and the research field phases. Methods of data collection in this research included questionnaires, interviews, observation, and documents. These questionnaires consist of two general and specialized sections. The former includes demographic questions (including age, gender, level of education, and work experience), and the latter includes research questions. Twenty minutes of direct and face-to-face interviews were used for each sample to complete the questionnaire. Before completion, individuals were asked to sign a written consent form at the top of the questionnaire. The validity of the questionnaire was calculated to be 0.46 according to the opinion of 30 experts in this field, and its reliability was calculated using SPSS software version 22 according to Cronbach's alpha of 0.832, 0.835, 0.917, 0.811, 0.788, and 0.976 for financial indicators, social responsibility, citizens and clients, the internal process, the growth and learning, and the questionnaire totally, respectively. Therefore, from the obtained values, it can be concluded that the questionnaire prepared in this study has the necessary content validity and reliability and can properly evaluate the intended purpose of this study. The inclusion criteria for included target groups and stakeholders (individuals and policymaking and implementation centers in the Iranian health system) in the health reform plan encompasses four levels, including Iranian health governance and policymaking (the level of policymakers and strategic managers), the level of suppliers and middle managers, the level of service providers (operational managers of service units) and the level of clients of the two hospitals (i.e., Modares and Zaeem Pakdasht hospitals) and employees of 10 hospitals. The exclusion criteria included inability to complete the questionnaires for personal issues, incomplete completion of the questionnaire, or withdrawal of continuing the research. One of the ethical points of research was to assure the subjects about the confidentiality of the information and the right to withdraw at any research stage.
In the qualitative phase, purposive sampling and semi-structured interviews of 15 experts, the Delphi method was used to evaluate and extract 70 indicators of the comprehensive evaluation of health system reform plan in 5 dimensions of Extended Balanced Scorecard (BSC), and there was a consensus on it.
The experts were selected based on theoretical mastery, practical experience, willingness, and ability to participate in research and access (20).
In a study in which the Delphi technique (The Delphi technique is a quantitative option aimed at generating consensus.) was used, if the participants were homogeneous, 10 to 15 samples would be sufficient to perform Delphi. According to the studies and articles using the Delphi method, it is recommended that the number of experts be between 1 and 2 people (20). Therefore, in the present study, about 50 people were identified; finally, 18 people were selected after filtering and applying the aforementioned indicators. It should be noted that 17 individuals participated in the first round of the Delphi method (The Delphi method, also known as Estimate-Talk-Estimate [ETE]) is a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts.), and 15 questionnaires from the questionnaires distributed were returned, and it was discovered that they could be used. Fifteen individuals participated in the second round and 14 individuals in the third and fourth rounds. The weights of dimensions and indicators, and priorities extracted by the Delphi method were determined using paired comparisons and SPSS and Expert Choice software.
In the third stage, a quantitative study in order to determine the weight and importance of the indicators and dimensions of the model, dimensions, basic components, and indicators obtained in the qualitative stage was exposed to the judgment of experts, determining weight, the importance of the indicators and model dimensions quantitatively. The final model of the comprehensive evaluation of the health reform plan in Iran was presented at this stage.
In the fourth stage, evaluation using the designed model, which was a quantitative phase in the first part to implement the designed model, 400 employees of 10 educational and non-educational hospitals of Shahid Beheshti University of Medical Sciences were selected by stratified random sampling method. In the second part of the quantitative phase, 300 clients in two educational and non-educational hospitals of Shahid Beheshti University of Medical Sciences were selected by simple random method, who had obtained the highest and lowest evaluation scores in the previous phase to evaluate the health reform plan according to the designed model. To describe the data and extraction of dimensions and indicators' weight, SPSS software Ver.22, AMOS software Ver.24, and Expert Choicev24 software were used. Also, the central parameters (i.e., arithmetic mean, geometric mean, harmonic mean, mode, and quartets) and dispersion indicators (range of changes, interquartile range, mean deviation from the mean, variance, standard deviation, and scattering coefficient) were considered as descriptive statistics. As inferential statistics, Kendall and Confirmatory factor analysis were used in line with the experts' consensus and model confirmation in the field tests.
Results
Qualitative part: Measures and findings in the sound done according to the Delphi method
The dimensions adapted from the extended balanced scorecard model in this study are as follows: financial - social responsibility - citizens and client - internal process - growth and learning dimensions (21). In the fourth round, the mean and standard deviation of the total components were equal to 3.844 and 0.823, respectively. According to the comparison of mean and standard deviation values of all components in the fourth round, with similar values in the second round, the components had a higher mean and lower standard deviation than the second round, but no significant change was observed compared to the third round. Kendall's coefficient of concordance was calculated for panel members' responses to the indices of five dimensions, and it was 0.735 for the second round, 0.893 for the third round, and 0.903 for the fourth round. Given that the panel members were more than four people (15 individuals for the second round, 15 individuals for the third round, and 14 individuals for the fourth round), this value is quite significant.
Given that Kendall's coefficient in the fourth round increased by only 0.01 compared to the third round, and no significant increase was found in the members' consensus in the two successive rounds, we can end up repeating the Delphi rounds. On the other hand, according to Kendall's coefficient values, there is a strong consensus in the second round, but there is a strong consensus among the penalists in the third and fourth rounds. The indices extracted in this model are presented in Tables 1 and 6:
Prioritization of the dimensions and indicators of the comprehensive evaluation model of the health care reform plan in Iran
A) Dimensions: 5 dimensions determined based on the mean, geometric mean, and weight values (extracted from Expert Choice software) in the final model were prioritized, and the results are presented in Table (1):
A significant difference was observed between the mean observed for the "social responsibilities field" in teaching and non-teaching hospitals
(t = 10.678, p =0.004) Therefore, the mean value of the social responsibilities field in teaching hospitals is less than this field's mean value in non-teaching hospitals.
A significant difference was observed between the mean observed for the "growth and learning" field in teaching and non-teaching hospitals (t = 15.789, p =0.007) Therefore, the mean value of the growth and learning field in teaching hospitals is higher than this field's mean value in non-teaching hospitals.
Discussion
The decision-making process regarding the health system's high-priority needs and ensuring that such priorities are met, including justice in access to health services and financial justice in health, depends on the functions of the health system. Therefore, after years of governmental efforts to promote public health and finance these commitments, the health reform plan emerged. As a result, making strategic decisions based on the future ahead of the health reform plan is of great importance. In doing so, in line with the main goal of the research, which is to design a comprehensive model for evaluating the health reform plan in Iran in order to achieve effectiveness, in phases 1 to 3 of the research to answer the main question "how is the health reform plan in Iran?": and the sub-question 1 "what are the dimensions, basic components, and prioritization in the comprehensive evaluation model of the health reform plan in Iran?", the initial conceptual model was determined, and then the main dimensions, components, and indicators were extracted after which, during several discussions with professors and experts, the extracted components and indicators were modified and adjusted. Finally, the model and all its components and indicators were reviewed and finally approved. In the next stage, the dimensions, components, and indicators of the plan were designed in the form of a questionnaire, and in two phases, the questionnaire was distributed among 400 employees of 10 hospitals and 300 patients of 2 hospitals. Finally, all questionnaires were collected and analyzed. Prioritization of dimensions and their components at the macro, intermediate, micro, and general levels of service based on the mean values obtained from the final round of the Delphi method and based on the weight extracted from the pairwise comparisons of the indicators shown by EXPERT CHOICE software showed that "Finance" is the priority and the "Internal Process" is the last priority.
To answer sub-question 2, "what are the indicators of the comprehensive evaluation model of the health reform plan in Iran (in 5 dimensions and 4 levels)?" The extracted indicators in each dimension obtained by Delphi's rounds were weighted and ranked. The indicators of each dimension of the model are discussed as follows:
Calculations showed that in the learning and growth dimension, "per capita educational growth rate of personnel" with a weight of 0.049 is in the first rank and "change in the educational system of medical groups based on the health reform plan" is in the last rank with a weight of 0.003.
The results obtained from this section of the study are consistent with the results reported in studies conducted by Piri et al. (16), Wu et al. (5), Farooq Salman et al. (6), Leksono et al. (7), Gutacker & Street (9), Mehralian et al. (10), Kalish and Jitish (12), and Dadgar et al. (13).
Calculations showed that in the financial dimension, "establishment and optimization of performance-based payments for employees working in the plan" is in the first rank with a weight of 0.047 and "reduction of direct payments of patients and other clients" is in the last rank with a weight of 0.003.
The results obtained from this section of the study are consistent with the results reported in studies conducted by Wu et al. (5), Farooq Salman et al. (6), Leksono et al. (7), Gutacker & Street (9), Mehralian et al., (10), Kheyri et al., (19), and OliaeiManesh et al., (15) and Dadgar et al., (13).
Calculations showed that in terms of social responsibility, "creating motivational mechanisms and attracting the participation of suppliers of the health reform plan in providing health care services" is in the first rank with a weight of 0.032 and "supervision of the Ministry of Health on the quality of health, medical and social services to society" is in the last rank with a weight of 0.002.
The results obtained from this of the study are consistent with the results reported in studies conducted by Piri et al. (16), Wu et al. (5), Farooq Salman et al. (6), Gutacker & Street (9), Mehralian et al. (10), OliaeiManesh et al., (15), Rahmat et al., (8), Anjomshoae et al., (11) and Dadgar et al. (13).
Calculations showed that in the indicators extracted in the dimension of citizens and clients, "creating a system for recording patients' complaints about the errors of health care providers in the website of Ministry of Health" is in the first rank with a weight of 0.035 and "increasing patient/clients' satisfaction planners regarding staff performance" ranks last with a weight of 0.004.
The results obtained from this section of the study are consistent with the results reported in studies conducted by Piri et al. (16), Wu et al. (5), Farooq Salman et al. (6), Gutacker & Street (9), Mehralian et al., (10), OliaeiManesh et al., (15) Kheyri et al., (19), and Dadgar et al., (13).
Calculations showed that in the dimension of internal processes, "providing performance feedback system and improving processes and structures" ranks first with a weight of 0.024 and "transfer of responsibility and authority of the Ministry of Health while maintaining management and regulatory controls to local units such as (governorate, municipality and provincial councils, non-governmental organizations, etc.) ranks last with a weighted value of 0.005. The results obtained from this section of the study are consistent with the results reported in studies conducted by Piri et al. (16), Wu et al. (5), Farooq Salman et al. (6), Gutacker & Street (9), Mehralian et al., (10), Kailish and Jitish (12), Doshmangir and Rashidian (18), OliaeiManesh et al., (15), Leksono et al., (7), and Dadgar et al. (13).
In phase 4 of the study, a 900-point questionnaire consisting of 70 items obtained from the components in the final model was prepared through the Delphi technique with 5-choice spectrum responses (score range 1-3-5-7-9) and based on the random stratified sampling for 400 staff members of 5 educational hospitals (including Imam Hossein Hospital, Loghman Hospital, Tajrish Martyrs Hospital, Modarres Hospital, and Taleghani Hospital) and 5 non-educational hospitals (including Imam Khomeini Hospital of Firoozkooh, the 3rd of Shaban hospital of Damavand, the anonymous martyrs' hospital, Zaeem hospital of Pakdasht and the martyrs' hospital of Pakdasht) and then health reform plan evaluated in terms of a statistical population. After this stage and in order to evaluate the model in terms of service recipients' (citizens and clients) comments and using a questionnaire made by the CEHSRP-IR model (with 12 items), the health reform plan was evaluated by the statistical community members, with a total number of 300 people selected by simple random from among patients and clients of Modarres and Zaeem hospitals (in the previous stage, obtained the highest (554.38) and the lowest (364.17) scores by the service providers, respectively). It was shown that 51% of the service providers and 55% of the service recipients had achieved their goals in the health reform plan.
Research limitations include
the lack of available scientific resources, the lack of similar work in this area, insufficient budget to do and advance the work, inadequate cooperation in departments and executive bodies, and officials' low cooperation of some managements and individuals in distributing and collecting questionnaires.
In this study, 5 dimensions of health reform plans in Iran were studied and researched in 4 levels. Based on it, 17 components and 70 indicators for success and effectiveness of the plan implementation were extracted through exploratory interviews with experts and literature review approved by experts. According to the study, none of the country's previous studies on the evaluation of the health reform plan had dealt with the social responsibility dimension. Besides, many researchers have investigated the plan from a single angle or have relied on a certain area/component. Therefore, this research's extracted dimensions are more comprehensive than other previous research and have a more general view of the structure, implementation, and monitoring of the country's health reform plan.
Consequently, the research model is complete and comprehensive based on these dimensions, components, and indicators. On the other hand, very little research has been done to evaluate the health reform plan in Iran. Therefore, this research accordingly has an advantage.
Conclusion
In this study, the need to adapt and fit the model's main dimensions (including finance, social responsibilities, internal processes, citizens and clients, and growth and learning with a comprehensive approach to achieving effectiveness in the organization) was emphasized. Among the components extracted for the dimension social responsibilities, the following components respectively had the highest weight and impact on the effectiveness of the plan's performance and output: the system and the mechanism of participation of all nodes involved in the health reform plan in order to actively monitor the implementation of the plan, the full development of services including insurance and health services in all parts of the country for different social classes, the commitment of the governing bodies to comprehensively implement the plan and attention to the fundamental rights of the people.
In the financial dimension, the following components showed to have the highest weight and impact on the effectiveness of the plan's performance and output, respectively: intelligent and targeted monitoring in the field of supply, distribution, and consumption of pharmaceuticals equipment and medical supplies, financing the plan through governmental and public budgets as well as donors in proportion to progress and effectiveness, optimizing the income, payments and investment structures based on productivity of service levels.
In the dimension of customers (clients), the following components respectively showed the highest impact and weight on the plan's performance and output: maximum satisfaction of the plan clients, community health growth and diseases management, the responsiveness of all levels of the plan to public opinions, and the creation of transparency making platforms.
Acknowledgments
The authors would like to express their gratitude to the honorable Professor, Seyyed Mehdi Alvani, for his advice and guidance in this study. It is noteworthy to mention that the present study has been extracted from dissertation 17121042972001 no.
Authors' contribution
SH.J and M.R conceived of the presented idea.
SH.J developed the theory and performed the computations.
M.R verified the analytical methods.
Supervised the findings of this work. All authors discussed the results and contributed to the final manuscript.
Conflict of interest
The author does not have any conflict of interest.
Introduction: The present study aimed to design a comprehensive model to evaluate the health system reform plan so the indicators of this model can lead to the health reform plan's effectiveness. Hence, this plan's effective factors, including policymakers and managers, suppliers, and clients, were identified and explained.
Methods: According to the practical purpose and nature of the research, the combined/mixed research method was used, in 2018-2019. For the qualitative phase, the Delphi technique was used in exploratory interviews, and quantitative methods were used to collect data through a questionnaire. In the quantitative phase, 400 employees of 10 hospitals were selected by stratified random sampling; and in the second section, 300 clients were randomly selected in two hospitals with the highest and lowest evaluation scores in the previous phase. Content analysis method and Delphi technique were used for classification and analysis of qualitative data, and SPSS Ver.22, AMOS Ver.24 and Expert Choicev24 were used for data description and dimensions and indices' weight extraction. Dimensions under consideration include the financial, social responsibility, growth and learning, citizens and client, and internal process fields.
Results: The components with the highest and lowest weight and impact on the effectiveness of the plan were identified in the following dimensions: financial field, Social responsibility field, growth and learning field, citizens and client field, internal process field, including weights 0.266, 0.244, 0.202, 0.164, 0.124, respectively.
Conclusion: Given the results, the components, indicators, dimensions, and levels of the model designed, the comprehensive evaluation of the Health System Reform Plan in Iran (CEHSRP-IR) with the normal mean was confirmed by the statistical population, and this model can be used in all organizations implementing this plan.
Keywords: Health care reform, Health plan, Integrated healthcare systems, Performance evaluation
Introduction
The health system reform plan with three approaches of financial protection of people, equity in access to health services, and improving quality of services has been put on the government's agenda to implement health objectives in the 5th Five Year Development Plan. On March 30, 2014, this plan was implemented according to Note 21 of the 2014 Budget Bill. Part of the project's funding was funded by reforming the price of goods, services, and subsidies on bread and electricity (1).
Investigating performance and efficiency as one of the effective factors on the effectiveness of a healthcare reform plan can help policymakers and executives better identify its strengths and weaknesses and strive to improve them (2). After five years of implementing this plan, there is a need for conducting scientific research with a precision tool on the evaluation of the project implementation. Designing a comprehensive model for a healthcare reform plan in the country is considered the first step in this field, and the present study seeks to fill the theoretical vacuum of the lack of a comprehensive model of healthcare reform plan assessment.
According to the studies conducted in this field, unfortunately, no comprehensive approach is found in evaluating this plan. Only one aspect of the issues related to this plan has been addressed by each of the researchers who have taken measures in this field, and the approach used in most studies has been in a simple framework (measuring the attitudes of clients and practitioners in reform plans, etc.). This vacuum is as important as the Deputy Minister of Health, who is also responsible for the implementation of the plan, and has pointed out the need for its implementation in a clear manner (3), that measures and results of the implementation of the health reform plan should be evaluated in the language of statistics and based on recognized international indicators (3). This study aimed to address four major challenges in performance evaluation included: Lack of comprehensiveness in evaluation criteria at different levels, One-dimensional view of evaluation and lack of comprehensiveness of evaluation dimensions, Lack of attention and pay attention to organizations' social responsibility in the evaluation criteria and inconsistency of performance indicators and strategic goals (4). In this regard, designing a comprehensive evaluation model of health reform plan in Iran was selected for the present study in order to cover part of this basic need, using the Extended Balanced Scorecard (BSC), considering the local, climatic, cultural, and structural conditions of this plan (project) and using the Willcocks'theoretical framework. This study aimed to design a comprehensive model of health reform plan evaluation in Iran with an Extended Balanced Scorecard approach so that relevant organizations can implement their strategy based on strategic goals and comprehensively evaluate its results.
Wu et al. during a study conducted a balanced scorecard based on a valid comprehensive measurement system by investigating the employees working in the hospital in western China. In the present study, 216 professional employees received valid answers through questionnaires. According to the results, ranking the importance of the Balanced Scorecard views is related to a serious concern in future planning and management. The establishment of a balanced scorecard measurement system for integrated health care organizations in China (5) has been recommended in this study. Also, Farooq Salman et al. in a study (case study: Oman's Suhar University) investigated the effectiveness of using the Balanced Scorecard (BSC) as a management tool in the performance evaluation and thus the quality of university services. According to their results, there is a strong relationship between the strategic roadmap and the four BSC perspectives. According to the present study results, the concept of BSC can be introduced to an educational institution to fill the gap created between creating a strategy and implementation in the same way (6). Leksono et al., in a study, investigated the Balanced Scorecard (BSC)-based sustainable supply chain and, using expert opinions, extracted 29 indices and examined their effects on BSC dimensions and finally, mapped the strategy map by examining the relationships of indices (7).
Rahmat et al. concluded that environmental and social standards and resource efficiency could prioritize performance indicators and clarify the strategy map in a detailed way based on Extended BSC in the field of Indonesia's manufacturing sector while investigating the environmental and social aspects of the scorecard (8). Gutacker& Street, in a study on public treatment sectors in the UK entitled "multidimensional performance evaluation of public sector organizations using dominant criteria," concluded that we need to find significant correlations between goals. They concluded that ignoring correlations can lead to inaccurate performance evaluation (9). Mehralian et al. in a study entitled "TQM and organizational performance using a balanced scorecard approach" on Iranian pharmaceutical industries, have concluded that the implementation of TQM has a positive and significant effect on the BSC and its four perspectives. They believed that managers should strongly use TQM methods to achieve their strategic goals due to the strong relationship between TQM and all four organizational performance perspectives (BSCs) (10). Anjomshoae et al., in a study entitled "Towards a Dynamic Balanced Scorecard Model for Managing the Performance of Humanitarian Aid Organizations," explained the sustainable, balanced scorecard and thereby introduced a conceptual model with a BSC approach to humanitarian relief organizations (11). Kailash et al., in a study entitled "Development of Balanced Scorecard for Health Care Using Interpretive Structural Modeling and Analytic Network Process," aimed to establish a Balanced Scorecard-based Performance Assessment Framework for Health Care Systems, identified key indicators of performance and their relationship to the health care system. They used an integrated interpretive structural modeling approach and an analytical networking process to develop a balanced scorecard (12).
In a study entitled "Investigating the effect of Health CareReformPlan on Hospital Performance Indicators in Lorestan Province," Dadgar et al. concluded that implementing the above plan has led to making a positive change in performance indicators of the hospital. Therefore, it is recommended to pay attention to the continuous improvement of the current process and the continuation of the plan's implementation (13). A study was carried out to investigate the effect of subsidy targeting and health reform plan on equity index in household health expenditures. The results showed that the incomplete implementation of the first phase of the Subsidy Targeting Law has had no positive effect on justice in the health system's financing due to decreasing health system financing after implementation of subsidy targeting law. On the other hand, no significant change has made in the financing of the health system in terms of the inequality and injustice with implementing the second phase of subsidies targeting (Paragraph b, Article (34) of the 5th Five Year Development Plan and the Health ReformPlan as a financial reform of the health system). So, it seems that the main factors affecting the reduction of direct payments by the people and thus the reduction of income inequalities which must be considered in health system reforms include the implementation of the laws in a precise and correct way, maximizing the protection of laws, timely financing and greater participation of government in financing the health system (14). A study was carried out to examine health insurance organizations' performance in cross-sector collaboration with the Ministry of Health and Medical Education to implement the health system reform plan effectively. The results showed a need for clarification and creating legal requirements on insurance organizations' obligations and subsequently "monitoring and control by the health system administrator, Ministry of Health (15). According to the results of another study that was carried out to examine the importance of patient satisfaction in the health system reform plan, centers' attention to patient satisfaction is relatively favorable. However, there is a need for additional efforts to achieve this goal reform plan. It seems that basic review is required to handle complaints and suggestions on the forms used to satisfy the reform plan (16). The results of a study on analyzing the key indicators of performance in the field of human resources showed that key indicators of performance are often used as a "value" and are linked to organizational strategy in order to measure activities such as the effectiveness of leadership method in the development of the organization, employment, services and satisfaction (17). According to the results of a study that aimed to analyze the way used to choose the policy interventions and decision making of Iranian health system policymakers (1979-2014) and analyze the pattern of decision-making and policy thinking, the decision made to apply at least one or more important and broad policy interventions from a lifelong perspective is considered as an important principle considered during the tenure of the health policymakers involved in the country's health system. Based on this view, a policy is made based on lexical rationalism patterns, garbage-can model, and open decision making (Open decision making is an approach to arriving at actionable agreements through participatory practices.) and decision-making during a crisis. Also, the health implications obtained can be justified by the models used (18). According to the results of a study that aimed to measure the way used to implement health reform plans from experts who monitored the treatment, the indicators evaluated in relation to each plan did not materialize completely, but they were fairly favorable. Given that evaluations have been made at different time intervals from the start of each plan, a small amount of failure to realize can be due to several reasons, including inadequate training, shortage of workforce, lack of preparation of some infrastructures, such as increased numbers of the patients referring to centers, and increasing burden of problems in health centers in the early stages of plans. Finally, the present study has recommended re-evaluating these indicators' utility in the second round of visits to the medical centers compared to the first round of visits (19).
Based on the mentioned experiences and considering the framework of this research that was mentioned in the expression of the problem, we seek the goal of four important challenges in evaluating performance under the headings: lack of comprehensiveness of evaluation criteria at different levels, one-dimensional view of evaluation and lack of comprehensiveness of the dimensions of evaluation, lack of attention and consideration of social responsibility of organizations in evaluation criteria and lack of alignment of performance indicators and strategic goals (Bozorg Haddad, 2018, 13-13). In this direction, to address the challenges of a one-dimensional view of evaluation and the lack of comprehensiveness and imbalance of the evaluation system, as well as the mismatch of performance indicators and strategic goals of a balanced scorecard with four financial dimensions, employees, internal processes and growth and learning and To solve the problem of lack of attention and to consider the social responsibility of the project in the evaluation criteria of the scorecard developed with the fifth dimension under the title of environmental (social responsibilities) and to solve the problem of lack of comprehensiveness of evaluation criteria at different levels, using the framework provided By Wilcox and its generalization in line with the dimensions of the national document of general health system policies, four levels of evaluation (including policy makers and strategic managers, employees, service providers and suppliers) are identified and classified. Therefore, the main purpose of this study was to design a comprehensive model for evaluating the health transformation plan in Iran so that the relevant organizations can implement their strategy based on strategic goals and evaluate the results comprehensively and correct their mistakes.
Methods
This study is considered descriptive and cross-sectional research that was carried out during 2018-2019. A mixed-method (qualitative and quantitative) was used in the present study due to its purpose and nature. To carry out this study, a relatively comprehensive understanding was first achieved in relation to how to evaluate the health system reform plan in Iran by studying previous research records, and a preliminary framework was provided on this basis.
In the qualitative phase, purposive sampling and semi-structured interviews of 15 experts, the Delphi method was used to evaluate and extract 70 indicators of comprehensive health system reform plan evaluation in 5 dimensions of Extended Balanced Scorecard (BSC and there was a consensus on it. Then, weights of dimensions and indicators and their priorities were determined using paired comparisons and SPSS and Expert Choice software. In the quantitative phase, the simple sampling method was used to fit the designed model adaptively, and the views of 400 health care reform plan providers in 10 teaching and non-teaching hospitals (two independent groups) in Shahid Beheshti University of Medical Sciences were obtained.
The sample size in qualitative research is the sample size that adequately answers the research questions. Therefore, after separating the levels, a total of 17 people were selected from among them to the extent of adequacy (saturation) by snowball method from experts at each level in whom the role of performance was more noticeable and open. In the quantitative part of the research, according to the statistical population and using Cochran's formula, 400 employees of 10 hospitals of Shahid Beheshti University of Medical Sciences were selected to evaluate the health transformation plan by stratified random sampling. Also, 300 patients (project recipients) in two hospitals of Shahid Beheshti University of Medical Sciences were selected by simple random sampling method.
Cochran's formula
n = sample size N = statistical population size
z = Percentage of acceptable standard reliability error
p = Percentage of population without definite attribute
q (1-p) = percentage of the population without a definite attribute
d= Degree of probability or probable desired accuracy
In the first stage, the study of research literature, dimensions, components, and indicators derived from theoretical foundations and research conducted in the field of subject and exploratory studies became the basis for identifying the conceptual model. By conducting theoretical studies and using the content analysis method, the initial conceptual model was presented. In other words, after gaining the necessary knowledge of the scientific bases and existing models and classifying and analyzing them using the content analysis method, the main dimensions, components and indicators were identified. In several stages of explorative interviews with 17 experts, the dimensions, components, and indicators were identified and corrected by the Delphi technique. After final corrections and review of validity and confirmation, the basis of investigations and field and experimental studies and analyses was provided.
In the second stage, which was a qualitative study based on the Delphi method, in two parts and the research area -Shahid Beheshti University of Medical Sciences including five educational hospitals (i.e., Imam Hossein Hospital, Loghman Hospital, Tajrish Martyrs Hospital, Modares Hospital, and Taleghani Hospital) and five non-educational hospitals (i.e., Imam Khomeini Hospital in Firoozkooh, 3rd Shaban Hospital in Damavand, Anonymous Martyrs Hospital, Zaeem Hospital in Pakdasht and Pakdasht Martyrs Hospital), the data was used for the comprehensive evaluation of the health reform plan based on comments of the statistical population. After evaluating the plan at the level of 10 hospitals by the servers of the plan, the plan was judged and evaluated in two hospitals (i.e., Modares and Zaeem Pakdasht hospitals) terms of clients' comments, and the relevant statistical analyzes were presented. Therefore, at this stage, following the components of the comprehensive evaluation model of the health reform plan in Iran through in-depth and exploratory interviews with scientific and executive experts who were purposefully selected, the necessary qualitative data was collected. Using content analysis as a research technique, the concepts, categories, and main and secondary factors were identified and analyzed. These concepts, factors, and categories were considered the basis for developing the tool (questionnaire) to achieve effective factors and recognize the dimensions and components, providing a model for presenting a comprehensive evaluation model of Iran's health reform plan. This section resulted in a package of basic dimensions and components and indicators of the comprehensive evaluation model of Iran's health reform plan. In the qualitative part of the research, the researcher-designed semi-structured questionnaire was used. The structured questionnaire was used to collect information in the 2nd to 4th phases of the Delphi technique and the research field phases. Methods of data collection in this research included questionnaires, interviews, observation, and documents. These questionnaires consist of two general and specialized sections. The former includes demographic questions (including age, gender, level of education, and work experience), and the latter includes research questions. Twenty minutes of direct and face-to-face interviews were used for each sample to complete the questionnaire. Before completion, individuals were asked to sign a written consent form at the top of the questionnaire. The validity of the questionnaire was calculated to be 0.46 according to the opinion of 30 experts in this field, and its reliability was calculated using SPSS software version 22 according to Cronbach's alpha of 0.832, 0.835, 0.917, 0.811, 0.788, and 0.976 for financial indicators, social responsibility, citizens and clients, the internal process, the growth and learning, and the questionnaire totally, respectively. Therefore, from the obtained values, it can be concluded that the questionnaire prepared in this study has the necessary content validity and reliability and can properly evaluate the intended purpose of this study. The inclusion criteria for included target groups and stakeholders (individuals and policymaking and implementation centers in the Iranian health system) in the health reform plan encompasses four levels, including Iranian health governance and policymaking (the level of policymakers and strategic managers), the level of suppliers and middle managers, the level of service providers (operational managers of service units) and the level of clients of the two hospitals (i.e., Modares and Zaeem Pakdasht hospitals) and employees of 10 hospitals. The exclusion criteria included inability to complete the questionnaires for personal issues, incomplete completion of the questionnaire, or withdrawal of continuing the research. One of the ethical points of research was to assure the subjects about the confidentiality of the information and the right to withdraw at any research stage.
In the qualitative phase, purposive sampling and semi-structured interviews of 15 experts, the Delphi method was used to evaluate and extract 70 indicators of the comprehensive evaluation of health system reform plan in 5 dimensions of Extended Balanced Scorecard (BSC), and there was a consensus on it.
The experts were selected based on theoretical mastery, practical experience, willingness, and ability to participate in research and access (20).
In a study in which the Delphi technique (The Delphi technique is a quantitative option aimed at generating consensus.) was used, if the participants were homogeneous, 10 to 15 samples would be sufficient to perform Delphi. According to the studies and articles using the Delphi method, it is recommended that the number of experts be between 1 and 2 people (20). Therefore, in the present study, about 50 people were identified; finally, 18 people were selected after filtering and applying the aforementioned indicators. It should be noted that 17 individuals participated in the first round of the Delphi method (The Delphi method, also known as Estimate-Talk-Estimate [ETE]) is a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts.), and 15 questionnaires from the questionnaires distributed were returned, and it was discovered that they could be used. Fifteen individuals participated in the second round and 14 individuals in the third and fourth rounds. The weights of dimensions and indicators, and priorities extracted by the Delphi method were determined using paired comparisons and SPSS and Expert Choice software.
In the third stage, a quantitative study in order to determine the weight and importance of the indicators and dimensions of the model, dimensions, basic components, and indicators obtained in the qualitative stage was exposed to the judgment of experts, determining weight, the importance of the indicators and model dimensions quantitatively. The final model of the comprehensive evaluation of the health reform plan in Iran was presented at this stage.
In the fourth stage, evaluation using the designed model, which was a quantitative phase in the first part to implement the designed model, 400 employees of 10 educational and non-educational hospitals of Shahid Beheshti University of Medical Sciences were selected by stratified random sampling method. In the second part of the quantitative phase, 300 clients in two educational and non-educational hospitals of Shahid Beheshti University of Medical Sciences were selected by simple random method, who had obtained the highest and lowest evaluation scores in the previous phase to evaluate the health reform plan according to the designed model. To describe the data and extraction of dimensions and indicators' weight, SPSS software Ver.22, AMOS software Ver.24, and Expert Choicev24 software were used. Also, the central parameters (i.e., arithmetic mean, geometric mean, harmonic mean, mode, and quartets) and dispersion indicators (range of changes, interquartile range, mean deviation from the mean, variance, standard deviation, and scattering coefficient) were considered as descriptive statistics. As inferential statistics, Kendall and Confirmatory factor analysis were used in line with the experts' consensus and model confirmation in the field tests.
Results
Qualitative part: Measures and findings in the sound done according to the Delphi method
The dimensions adapted from the extended balanced scorecard model in this study are as follows: financial - social responsibility - citizens and client - internal process - growth and learning dimensions (21). In the fourth round, the mean and standard deviation of the total components were equal to 3.844 and 0.823, respectively. According to the comparison of mean and standard deviation values of all components in the fourth round, with similar values in the second round, the components had a higher mean and lower standard deviation than the second round, but no significant change was observed compared to the third round. Kendall's coefficient of concordance was calculated for panel members' responses to the indices of five dimensions, and it was 0.735 for the second round, 0.893 for the third round, and 0.903 for the fourth round. Given that the panel members were more than four people (15 individuals for the second round, 15 individuals for the third round, and 14 individuals for the fourth round), this value is quite significant.
Given that Kendall's coefficient in the fourth round increased by only 0.01 compared to the third round, and no significant increase was found in the members' consensus in the two successive rounds, we can end up repeating the Delphi rounds. On the other hand, according to Kendall's coefficient values, there is a strong consensus in the second round, but there is a strong consensus among the penalists in the third and fourth rounds. The indices extracted in this model are presented in Tables 1 and 6:
Prioritization of the dimensions and indicators of the comprehensive evaluation model of the health care reform plan in Iran
A) Dimensions: 5 dimensions determined based on the mean, geometric mean, and weight values (extracted from Expert Choice software) in the final model were prioritized, and the results are presented in Table (1):
B) Indicators: In this model, in addition to indicators, strategies, key indicators of performance, and missions in each field are specified.
Growth and learning dimension
According to the values of mean, geometric mean, and weight of the indices (extracted from Expert Choice software), prioritization of growth and learning indices are presented according to Table (2):
Growth and learning dimension
According to the values of mean, geometric mean, and weight of the indices (extracted from Expert Choice software), prioritization of growth and learning indices are presented according to Table (2):
Financial dimension
According to the values of mean, geometric mean, and weight of the indices (extracted
from Expert Choice software), prioritization of financial indices are presented according to
Table (3):
According to the values of mean, geometric mean, and weight of the indices (extracted
from Expert Choice software), prioritization of financial indices are presented according to
Table (3):
Social responsibilities dimension
According to the values of mean, geometric mean, and weight of the indices (extracted from Expert Choice software), prioritization of financial indices are presented according toTable (4) :
According to the values of mean, geometric mean, and weight of the indices (extracted from Expert Choice software), prioritization of financial indices are presented according toTable (4) :
Citizens and clients dimension
According to the values of mean, geometric mean, and weight of the indices (extracted from Expert Choice software), prioritization of financial indices are presented according to Table (5):
According to the values of mean, geometric mean, and weight of the indices (extracted from Expert Choice software), prioritization of financial indices are presented according to Table (5):
Internal processes dimension
According to the values of mean, geometric mean, and weight of the indices (extracted from Expert Choice software), prioritization of financial indices are presented according toTable (6) :
According to the values of mean, geometric mean, and weight of the indices (extracted from Expert Choice software), prioritization of financial indices are presented according toTable (6) :
Quantitative section: Fitting the model
According to the results of the t-test, the statistical population agrees with a high Mean on the designed indices and models. However, priorities were variable due to the teaching and non-teaching nature of the centers in some dimensions.
According to the results of the t-test, the statistical population agrees with a high Mean on the designed indices and models. However, priorities were variable due to the teaching and non-teaching nature of the centers in some dimensions.
A significant difference was observed between the mean observed for the "social responsibilities field" in teaching and non-teaching hospitals
(t = 10.678, p =0.004) Therefore, the mean value of the social responsibilities field in teaching hospitals is less than this field's mean value in non-teaching hospitals.
A significant difference was observed between the mean observed for the "growth and learning" field in teaching and non-teaching hospitals (t = 15.789, p =0.007) Therefore, the mean value of the growth and learning field in teaching hospitals is higher than this field's mean value in non-teaching hospitals.
Discussion
The decision-making process regarding the health system's high-priority needs and ensuring that such priorities are met, including justice in access to health services and financial justice in health, depends on the functions of the health system. Therefore, after years of governmental efforts to promote public health and finance these commitments, the health reform plan emerged. As a result, making strategic decisions based on the future ahead of the health reform plan is of great importance. In doing so, in line with the main goal of the research, which is to design a comprehensive model for evaluating the health reform plan in Iran in order to achieve effectiveness, in phases 1 to 3 of the research to answer the main question "how is the health reform plan in Iran?": and the sub-question 1 "what are the dimensions, basic components, and prioritization in the comprehensive evaluation model of the health reform plan in Iran?", the initial conceptual model was determined, and then the main dimensions, components, and indicators were extracted after which, during several discussions with professors and experts, the extracted components and indicators were modified and adjusted. Finally, the model and all its components and indicators were reviewed and finally approved. In the next stage, the dimensions, components, and indicators of the plan were designed in the form of a questionnaire, and in two phases, the questionnaire was distributed among 400 employees of 10 hospitals and 300 patients of 2 hospitals. Finally, all questionnaires were collected and analyzed. Prioritization of dimensions and their components at the macro, intermediate, micro, and general levels of service based on the mean values obtained from the final round of the Delphi method and based on the weight extracted from the pairwise comparisons of the indicators shown by EXPERT CHOICE software showed that "Finance" is the priority and the "Internal Process" is the last priority.
To answer sub-question 2, "what are the indicators of the comprehensive evaluation model of the health reform plan in Iran (in 5 dimensions and 4 levels)?" The extracted indicators in each dimension obtained by Delphi's rounds were weighted and ranked. The indicators of each dimension of the model are discussed as follows:
Calculations showed that in the learning and growth dimension, "per capita educational growth rate of personnel" with a weight of 0.049 is in the first rank and "change in the educational system of medical groups based on the health reform plan" is in the last rank with a weight of 0.003.
The results obtained from this section of the study are consistent with the results reported in studies conducted by Piri et al. (16), Wu et al. (5), Farooq Salman et al. (6), Leksono et al. (7), Gutacker & Street (9), Mehralian et al. (10), Kalish and Jitish (12), and Dadgar et al. (13).
Calculations showed that in the financial dimension, "establishment and optimization of performance-based payments for employees working in the plan" is in the first rank with a weight of 0.047 and "reduction of direct payments of patients and other clients" is in the last rank with a weight of 0.003.
The results obtained from this section of the study are consistent with the results reported in studies conducted by Wu et al. (5), Farooq Salman et al. (6), Leksono et al. (7), Gutacker & Street (9), Mehralian et al., (10), Kheyri et al., (19), and OliaeiManesh et al., (15) and Dadgar et al., (13).
Calculations showed that in terms of social responsibility, "creating motivational mechanisms and attracting the participation of suppliers of the health reform plan in providing health care services" is in the first rank with a weight of 0.032 and "supervision of the Ministry of Health on the quality of health, medical and social services to society" is in the last rank with a weight of 0.002.
The results obtained from this of the study are consistent with the results reported in studies conducted by Piri et al. (16), Wu et al. (5), Farooq Salman et al. (6), Gutacker & Street (9), Mehralian et al. (10), OliaeiManesh et al., (15), Rahmat et al., (8), Anjomshoae et al., (11) and Dadgar et al. (13).
Calculations showed that in the indicators extracted in the dimension of citizens and clients, "creating a system for recording patients' complaints about the errors of health care providers in the website of Ministry of Health" is in the first rank with a weight of 0.035 and "increasing patient/clients' satisfaction planners regarding staff performance" ranks last with a weight of 0.004.
The results obtained from this section of the study are consistent with the results reported in studies conducted by Piri et al. (16), Wu et al. (5), Farooq Salman et al. (6), Gutacker & Street (9), Mehralian et al., (10), OliaeiManesh et al., (15) Kheyri et al., (19), and Dadgar et al., (13).
Calculations showed that in the dimension of internal processes, "providing performance feedback system and improving processes and structures" ranks first with a weight of 0.024 and "transfer of responsibility and authority of the Ministry of Health while maintaining management and regulatory controls to local units such as (governorate, municipality and provincial councils, non-governmental organizations, etc.) ranks last with a weighted value of 0.005. The results obtained from this section of the study are consistent with the results reported in studies conducted by Piri et al. (16), Wu et al. (5), Farooq Salman et al. (6), Gutacker & Street (9), Mehralian et al., (10), Kailish and Jitish (12), Doshmangir and Rashidian (18), OliaeiManesh et al., (15), Leksono et al., (7), and Dadgar et al. (13).
In phase 4 of the study, a 900-point questionnaire consisting of 70 items obtained from the components in the final model was prepared through the Delphi technique with 5-choice spectrum responses (score range 1-3-5-7-9) and based on the random stratified sampling for 400 staff members of 5 educational hospitals (including Imam Hossein Hospital, Loghman Hospital, Tajrish Martyrs Hospital, Modarres Hospital, and Taleghani Hospital) and 5 non-educational hospitals (including Imam Khomeini Hospital of Firoozkooh, the 3rd of Shaban hospital of Damavand, the anonymous martyrs' hospital, Zaeem hospital of Pakdasht and the martyrs' hospital of Pakdasht) and then health reform plan evaluated in terms of a statistical population. After this stage and in order to evaluate the model in terms of service recipients' (citizens and clients) comments and using a questionnaire made by the CEHSRP-IR model (with 12 items), the health reform plan was evaluated by the statistical community members, with a total number of 300 people selected by simple random from among patients and clients of Modarres and Zaeem hospitals (in the previous stage, obtained the highest (554.38) and the lowest (364.17) scores by the service providers, respectively). It was shown that 51% of the service providers and 55% of the service recipients had achieved their goals in the health reform plan.
Research limitations include
the lack of available scientific resources, the lack of similar work in this area, insufficient budget to do and advance the work, inadequate cooperation in departments and executive bodies, and officials' low cooperation of some managements and individuals in distributing and collecting questionnaires.
In this study, 5 dimensions of health reform plans in Iran were studied and researched in 4 levels. Based on it, 17 components and 70 indicators for success and effectiveness of the plan implementation were extracted through exploratory interviews with experts and literature review approved by experts. According to the study, none of the country's previous studies on the evaluation of the health reform plan had dealt with the social responsibility dimension. Besides, many researchers have investigated the plan from a single angle or have relied on a certain area/component. Therefore, this research's extracted dimensions are more comprehensive than other previous research and have a more general view of the structure, implementation, and monitoring of the country's health reform plan.
Consequently, the research model is complete and comprehensive based on these dimensions, components, and indicators. On the other hand, very little research has been done to evaluate the health reform plan in Iran. Therefore, this research accordingly has an advantage.
Conclusion
In this study, the need to adapt and fit the model's main dimensions (including finance, social responsibilities, internal processes, citizens and clients, and growth and learning with a comprehensive approach to achieving effectiveness in the organization) was emphasized. Among the components extracted for the dimension social responsibilities, the following components respectively had the highest weight and impact on the effectiveness of the plan's performance and output: the system and the mechanism of participation of all nodes involved in the health reform plan in order to actively monitor the implementation of the plan, the full development of services including insurance and health services in all parts of the country for different social classes, the commitment of the governing bodies to comprehensively implement the plan and attention to the fundamental rights of the people.
In the financial dimension, the following components showed to have the highest weight and impact on the effectiveness of the plan's performance and output, respectively: intelligent and targeted monitoring in the field of supply, distribution, and consumption of pharmaceuticals equipment and medical supplies, financing the plan through governmental and public budgets as well as donors in proportion to progress and effectiveness, optimizing the income, payments and investment structures based on productivity of service levels.
In the dimension of customers (clients), the following components respectively showed the highest impact and weight on the plan's performance and output: maximum satisfaction of the plan clients, community health growth and diseases management, the responsiveness of all levels of the plan to public opinions, and the creation of transparency making platforms.
Acknowledgments
The authors would like to express their gratitude to the honorable Professor, Seyyed Mehdi Alvani, for his advice and guidance in this study. It is noteworthy to mention that the present study has been extracted from dissertation 17121042972001 no.
Authors' contribution
SH.J and M.R conceived of the presented idea.
SH.J developed the theory and performed the computations.
M.R verified the analytical methods.
Supervised the findings of this work. All authors discussed the results and contributed to the final manuscript.
Conflict of interest
The author does not have any conflict of interest.
Review: Research |
Subject:
General
Received: 2019/11/9 | Accepted: 2021/03/20 | Published: 2021/03/29
Received: 2019/11/9 | Accepted: 2021/03/20 | Published: 2021/03/29
References
1. 1. Gholipour R, Netagh F. The analysis of health system reform plan- challenges and consequences, The first national conference on public administration in Iran, Tehran, Faculty of Management, University of Tehran, 2015. Recovered from https://www.civilica.com/Paper-CIPA01-CIPA01_046.html
2. OliaeiManesh A. et al., Performance of health insurance organizations in intersectional collaboration with the ministry of health and education in effective implementation of health system reform plan, Proceedings of 11th National Conference on Critique of Government Performance in Health Field, Tehran, National Institute of Health Research. 2015; (1):17-18. [Persian]
3. Harirchi I. The need for evaluation of health system reform plan according to international indicators [performance report], 2016. Available from: http://dolat.ir/detail/281042
4. Bozorg Haddad E. Multidimensional model design for performance evaluation of National Iranian Gas Company, PhD thesis, University of Tehran, Campus of Qom, 2018.
5. Wu X, Li S, Xu N, et al. Establishing a balanced scorecard measurement system for integrated care organizations in China. The International Journal of Health Planning And Management. 2019; 34(2): 672-92. [DOI:10.1002/hpm.2726]
6. Farooq SAM, Firdouse R, Diana FM. University performance evaluation and strategic mapping using balanced scorecard (BSC): Case study - Sohar University, Oman. International Journal of Educational Management. 2018; 32(4): 689-700. [DOI:10.1108/IJEM-05-2017-0107]
7. Leksono EB, Suparno S, Vanany I. Development of Performance Indicators Relationships on Sustainable Healthcare Supply Chain Performance Measurement Using Balanced Scorecard and DEMATEL. International Journal on Advanced Science Engineering Information Technology. 2018; 8 (1):115-122. [DOI:10.18517/ijaseit.8.1.3852]
8. Rahmat N, Saripuji P, Djoko SG. Extended Balanced Scorecard. International Journal of Engineering & Technology. 2018; 7 (2): 48-51.
9. Gutacker N, Street A. Multidimensional performance assessment of public sector organizations using dominance criteria. Health Economics. 2018; 27(2): 13-27. [DOI:10.1002/hec.3554]
10. Mehralian Gh, Jamal A, Nazari N, et al. TQM and organizational performance using the balanced scorecard approach. International Journal of Productivity and Performance Management. 2017; 66(1): 111-125. [DOI:10.1108/IJPPM-08-2015-0114]
11. Anjomshoae A, Hassan A, Kunz N, et al. Toward a dynamic balanced scorecard model for humanitarian relief organizations' performance management. Journal of Humanitarian Logistics and Supply Chain Management. 2017;7(2):194-218. [DOI:10.1108/JHLSCM-01-2017-0001]
12. Kailash M, Jitesh T. Development of Balanced Scorecard for healthcare using Interpretive Structural Modeling and Analytic Network Process. Journal of Advances in Management Research. 2014; 11(3):232-256. [DOI:10.1108/JAMR-12-2012-0051]
13. Dadgar, R. The examination of effect of health system reform plan on hospital performance indicators in Lorestan province. Journal of scientific research of Lorestan University. 2017; 19(2):10-15. [Persian]
14. OliaeiManesh A. The impact of the implementation of the targeted subsidies law and health system reform plan on equity index on household health expenditure, Proceedings of 11th National Conference on Critique of Government Performance in Health Field, Tehran, National Institute of Health Research. 2015; 1:33-34.
15. OliaeiManesh A. Performance of health insurance organizations in intersectional collaboration with the ministry of health and education in effective implementation of health system reform plan, Proceedings of 11th National Conference on Critique of Government Performance in Health Field, Tehran, National Institute of Health Research. 2015; 1:17-18.
16. Piri, Z. The importance of patient satisfaction in health system reform plan: experience, Tabriz University of Medical Sciences, Proceedings of 11th National Conference on Critique of Government Performance in Health Field, Tehran, National Institute of Health Research. 2015; 1:23-24.
17. Riyazi SR. The analysis of key performance indicators in the field of human resources. 3rd National and International Conference on Management and Accounting in Iran, Tehran, 2016. Available from: from http://www.tpbin.com/ article/53709.
18. Doshmangir L, Rashidiyan A. The analysis of the choice of policy interventions and the decisions of policy makers in health system (and analysis of thinking pattern about decision-making and policy- making), 1979-2014, Iran, Proceedings of 11th National Conference on Critique of Government Performance in Health Field, Tehran, National Institute of Health Research, 2015; 1:48.
19. Kheyri, F. Patients pay reduction program in hospitals, Proceedings of 11th National Conference on Critique of Government Performance in Health Field, Tehran, National Institute of Health Research. 2015; 1:12-13.
20. Stone Fish L, Busby DM. The Delphi method. In D. M Sprenkle and F. P. Piercy, Research Methods in Family Therapy. 2015; 1(1):238-253.
21. Kaplan R, Norton TD. The Balanced scorecard Translating Strategy into Action. Business School Press. 2013; 48: 217-225.
Send email to the article author
| Rights and permissions | |
 |
This work is licensed under a Creative Commons Attribution 4.0 International License. |
