Many Moving Parts: Evaluating the Implementation of the Kin KeeperSM Cancer Prevention Intervention

1. Department of Obstetrics, Gynecology & Reproductive Biology, College of Human Medicine,  Michigan State University, 965 Fee Road, Room 629A, East Lansing, MI 48824
2.  School of Nursing, Florida A&M University, 334 W. Palmer Ave, Tallahassee, FL 32307
3. Department of Obstetrics, Gynecology & Reproductive Biology, College of Human Medicine, Michigan State University, East Fee Hall, 965 Fee Road, Suite 632 Lansing, MI 48824
4. Center for Women, Children & Youth, College of Nursing ,The Ohio State University, 1585 Neil Avenue, 362 Newton Hall, Columbus, OH 43210

ARTICLE INFO		ABSTRACT
Original Article Received: 13 Nov 2017 Accepted: 19 Feb 2018		Introduction: This evaluation of the implementation of a cancer prevention intervention includes; intervention fidelity, participant satisfaction, participant retention, and the cost of program implementation. The main purpose of this paper is to disseminate findings in order to inform and promote similar community health programming. Methods: The evaluation framework integrates several models, including the Centers for Disease Control and Prevention (CDC) Evaluation Framework and the University of Wisconsin-Extension (UW-EX) Evaluation Model. This evaluation process is complex in that it: (1) is community-based participatory research (CBPR); (2) is a Randomized Controlled Trial (RCT); (3) uses an ecological approach; and (4) is multicultural. Results: For intervention fidelity, we found that 90% of participants reported high fidelity in the delivery of the intervention as well as strong participant satisfaction. Likewise, 90% of CHWs also report high fidelity in training provided and intervention integrity. Over 36 months of follow-up, the participant retention rate was 79% or more with no significant differences between the intervention and control groups. The cost to implement and deliver the program to each intervention participant was $151. Conclusion: These outcomes can inform policymakers in order to disseminate and implement quality health communication programming.   Keywords: Program Evaluation, Cancer Prevention, Kin Keeper SM, Black, Latina, Arab, Women, Health Disparities, Health Communication.
Corresponding Author: Karen Patricia Williams

Breast and cervical cancer are leading causes of cancer morbidity and cancer-related mortality among women in the United States. Although morbidity and mortality rates have declined, disparities persist among racial/ethnic minorities and low-income women. ⁽¹⁾ African Americans (herein referred to as Black) have a higher mortality for all cancers combined and Black women are nearly 50% more likely to die from breast and cervical cancer than White women. ^{(2, 3)} Breast cancer is the leading cause of death among Hispanic women. ⁽⁴⁾ Likewise, many other medically underserved ethnic minority women, including Arab women, have very low breast and cervical cancer screening rates. Yet, evidence suggests that breast and cervical cancer mortality rates can be greatly decreased with appropriate screening and follow-up. ^{(5, 6)}
Studies also suggest that community-based participatory research (CBPR) is an effective approach to build capacity by delivering effective treatment to minorities and medically underserved populations to address health disparities. ^{(7, 8)} CBPR is also a valuable strategy to address cancer disparities through cancer education. CBPR is particularly useful with hard-to-reach populations, including medically underserved ethnic minorities. The Kin Keeper ^SM Cancer Prevention Intervention was used here in a randomized controlled trial (RCT) incorporating CBPR methods to deliver breast and cervical cancer education to Black, Latina, and Arab women to increase screening behavior. ⁽⁹⁾ The cancer education curriculum was delivered by racially concordant community health workers (CHWs). Kin Keeper ^SM is an intricate intervention in which CHWs deliver the intervention. Intervention participants can invite their direct-line female relatives to also participate in the intervention. ⁽⁹⁾
Evaluation of prevention and intervention programs are often limited to efficacy and effectiveness. Yet, it is important to consider all elements of conducting a process evaluation of a prevention study.
A comprehensive process evaluation of a complex intervention requires examination of specific components that assess contextual aspects of the intervention. ⁽¹⁰⁾ The purpose of this report is to describe the evaluation of implementation process including four main components of a breast and cervical cancer intervention entitled Kin Keeper ^SM: (1) intervention fidelity; (2) patient satisfaction; (3) participant retention; and (4) cost. The study is important because explaining the process of evaluating the implementation of the Kin Keeper ^SM program provides insight into health promotion program design.
Methods
Kin Keeper ^SM delivered breast and cervical cancer education to Black, Latina, and Arab women in order to increase screening (mammograms and Pap tests). CHWs of the corresponding ethnicity with current clients from community health centers in the Greater Detroit area recruited participants to the intervention. Those clients that agreed to participate were randomized into the intervention group or the control group using a propensity scoring method derived from a detailed demographic questionnaire. ⁽⁹⁾ After meeting inclusion criteria for the study, participants in both arms of the study were offered the opportunity to invite two to three female family members (i.e., mother, daughter, sister, aunt, grandmother) to participate with them.
The total number of participants in this study is 516. Women in the intervention group (N = 306) received breast and cervical education over two sessions and had the information read to them with demonstrations using breast and cervical anatomical models. Those in the control group (N = 210) received only reading materials read on their own. The sampling procedure is fully described in a previous publication. ⁽⁸⁾ All women were followed for over three years to compare screening behaviors and retention of the breast and cervical education between the intervention and the control group.
Intervention Fidelity
Participant fidelity was guided by the National Institutes of Health (NIH) Treatment domains. ⁽¹¹⁾ For the purpose of this report, we measured Training Provided, Receipt of Treatment, and Treatment Integrity domains. Since the intention was to assess the fidelity of intervention, only intervention participants were provided with the survey. Intervention participants received a 12-item survey to assess Treatment Integrity and Receipt of Treatment after each of the breast and cervical cancer education sessions (Table 1).

Race	Arab	Latina	African American	Total
N (%)	125 (40.98)	33 (10.82)	147 (48.20)	305
Median Age in years (SD)	44.80 (12.24)	41.06 (10.58)	42.89 (14.06)	43.79 (13.02)
First Survey-Breast Cancer Items
I completed a form that asked me questions about my education, my health, and other similar things.	95.77	100	99.17	97.64
I heard a talk about breast cancer. (TI)	99.31	100	100	99.67
Before the talk, the community health worker asked me questions about what I knew about breast cancer. (TI)	98.60	100	99.19	99.00
After the talk, the community health worker asked me questions about what I knew about breast cancer. (TI)	99.30	100	99.19	99.33
The community health worker used flip charts during the talk. (TI)	98.61	100	100	99.33
During the talk, the community health worker showed some models of women’s breast. (TI)	99.31	100	99.19	99.33
The community health worker answered any questions about breast cancer. (TI)	98.59	100	100	99.33
I enjoyed the talk about breast cancer.	99.31	100	99.19	99.33
I understand what the CHW said about breast cancer. (TR)	99.30	96.97	100	99.33
I learned a great deal from what the CHW said about breast cancer. (TR)	97.92	96.97	99.18	98.33
The community health worker gave out gift bags.	97.67	100	91.23	93.04
We rescheduled our next home visit.	98.40	100	100	99.15
Second Survey-Cervical Cancer Items
I completed a form that asked me questions about how my family and I talk to one another. (TI)	97.14	85.85	98.39	96.30
I heard a talk about cervical cancer. (TI)	99.31	100	100	99.67
Before the talk, the community health worker asked me questions about what I knew about cervical cancer. (TI)	95.86	100	97.58	97.01
After the talk, the community health worker asked me questions about what I knew about cervical cancer. (TI)	98.61	96.97	100	99.00
The community health worker used a flip chart during the talk. (TI)	99.31	100	99.19	99.33
During the talk, the community health worker showed what cervical cancer looks like. (TI)	97.87	100	100	98.99
The community health worker answered any questions about cervical cancer. (TI)	99.31	100	100	99.67
I enjoyed the talk about cervical cancer.	99.29	100	99.19	99.32
I understood what the community health worker said about cervical cancer. (TR)	97.93	100	100	99.00
I learned a great deal from what the CHW said about cervical cancer. (TR)	99.31	100	100	99.67
The community health worker gave out gift bags. (Not a fidelity item)	92.08	58.62	99.16	91.57
I was told that I would get a postcard in the mail the next time the community health worker contacts me.(Not a fidelity item.)	99.27	96.88	100	99.31

CHWs were also given a survey adapted from the Michigan Department of Community Health CHW Survey that included 18 items pertaining to Training Provided and Treatment Integrity. Surveys were delivered individually to the CHW sand were anonymous. Surveys were translated in Spanish and Arab for non-English speaking Latina and Arab participants, respectively.
Participant Satisfaction
Assessment of participant satisfaction was provided to ascertain that not only was the intervention delivered but that it was acceptable as well. After all of the 516 participants completed the intervention and control education, a five-item survey was given to all participants to assess satisfaction. Items included: (1) Is this your first time in a research project? ; (2) Did you like participating in this project? ; (3) If yes, what is the reason? : (a) the cancer topic; (b) the incentives; (c) the CHW; or (d) all of the above; (4) Can we contact you to tell you the results?; and (5) Can we contact you for other research projects? Not only do these items intend to capture likability of the program, but whether or not they have a positive attitude towards CBPR. Chi-Square was used to examine differences in satisfaction between the intervention group and the control group. All fidelity and satisfaction questions are displayed in Table 1.
Participant Retention
At the first recruitment meeting, each participant was asked to give detailed contact information, including where they could be reached for follow up, and informed consent was signed. After families enrolled in the study, they completed demographic questionnaires and received educational sessions. The CHWs were responsible for follow-up with their families at 12, 18, 24, 30 and 36 months. Once done the intervention group participants agreed to be contacted and assisted the CHW to locate family as necessary for future follow-ups. Tracking and follow-up data were kept in hard-copy and electronic format. The CHWs maintained their own confidential hard-copy records with visit information. The electronic format, a Microsoft Excel workbook (Microsoft Office Excel, 2010), included data collected from the CHW hard-copy records and was formatted into worksheets for each CHW listing her client and the follow-up due dates as well as recording when the follow-ups had been completed for each family member. All data was merged into an IBM SPSS database ^{(12, 13)}. This database was maintained by the study coordinator to check the CHWs’ records and the Microsoft Excel workbook. We also used other organizational techniques to maximize retention, such utilizing a continually updated master list of contact information for all participants, sending reminder postcards, and making follow-up calls after the last appointment. The contact information was kept separately from identification numbers and other identifying data to ensure confidentiality.

Follow-up	All N = 516 (%)	Treatment n = 306 Kin keepers (%)	Control = 210 Participant-client (%)	Odds Ratio	95% CI
					lower	Upper
12-month	428 (82.95)	250 (81.70)	176 (83.81)	1.16	0.73	1.85
18-month	423 (81.99)	250 (81.70)	173 (82.38)	1.27	0.79	2.05
24-month	423 (81.99)	242 (79.08)*	181 (86.19)	1.65	1.02	2.67
30-month	365 (70.74)	210 (68.62)	155 (73.81)	1.29	0.87	1.90
36-month	399 (77.33)	241 (78.76)	158 (75.24)	0.82	0.54	1.24
Mean Total	408 (79.07)	239 (77.97)	169 (80.29)	1.15	0.75	1.29

Cost Assessment
The intervention cost assessment is described in detail elsewhere. ⁽¹⁴⁾ Few evaluations are published demonstrating community-based health promotion intervention costs, and of those, many are conducted after the end of the program. The evaluation of the cost of Kin Keeper ^SM was performed from the perspective of a health organization adding the Kin Keeper ^SM intervention to an existing CHW program. The cost of delivering the Kin Keeper ^SM intervention was assessed during a 12-month steady-state period of operation. The study period represented a typical operating year, during which the health agencies implemented the intervention at a relatively constant level in terms of participants visited, with no major organizational changes. Cost categories included CHW training, CHW labor (recruitment and home visits), CHW supervision, and material costs. CHW time was allocated separately to training, recruitment, and delivering the intervention. CHW training costs assumed reinforcement-training after approximately 100 visits to participants. A ten percent full-time equivalent (FTE) supervisor position was allocated for CHW supervision considering the trial’s purposes and was considered an appropriate share for the needs of the intervention. Transportation costs for a home visit were calculated based on a fixed transportation stipend. The CHWs were provided a stipend for two intervention visits. Materials costs were separated between consumable goods for each home visit (e.g., health brochures) and durable goods that could be used for several visits (e.g., breast model including in a breast cancer educational kit). For durable goods, we conservatively assumed that each item had a useful life of approximately 100 visited participants before needing replacement. The average cost of delivering the intervention to a family was reported.
Statistical Analysis
Statistical analyses included descriptive outcomes including means, standard deviations, and chi-square to detect significance. Frequency distributions provided descriptive characteristics at baseline. To explore bivariate associations between each independent variable and the outcomes of interest, we used logistic regressions and reported unadjusted odds ratios. Cost assessments were calculated and reported using the average cost of delivering the intervention to a family by summing all the cost components of the intervention, including the two home visits.
Results
The intervention group (n = 305) was made up of 48% Black, 11% Latina, and 41% Arab women. Almost all (> 90%) women agreed that they received the treatment in the way that it was intended, that is, Treatment Received and Treatment Integrity elements. No differences were found between race/ethnicity. A total of 16 CHWs responded to the survey. Fifty percent were Black, 13% Latina and 38% Arab. The majority of CHWs (> 90%) reported they felt they delivered the treatment as it was intended and in the same way other CHWs did. Both participants and CHWs determined there was an impact on their lives. CHWs consider serving their community as important and they felt empowered by informing a research project. CHWs and intervention participants noted that the program was mutually rewarding, indicating that there was “cross-fertilization and cross benefit” of working with each other. ⁽¹⁵⁾
Results for participant satisfaction are reported here for each question between the intervention and control groups. Question 1: Is this your first time in a research project? 49.2% of the intervention group and 52.4% of the control group reported “yes”. This outcome was not significant. Question 2: Do you like participating in this project? 91.4% of intervention group reported “yes” versus 100% of the control group reported “yes”. Question 3: For those that answered “yes” we queried: What was the reason? Possible responses included: (1) the topic was breast and cervical cancer; (2) the incentives; (3) the relationship with their CHW or (4) all of the above. The majority of intervention group participants and control group participants, 52.5% vs. 53% respectively, gave the reason as “all of the above”. The difference between the groups was not significant. Question 4: Can we contact you with the research results? 85% of the intervention group and 82.4% of control group responded “yes”. Question 5: Can we contact you for other research projects? 86% of the intervention group and 78.8% of the control group responded “yes”.
        At the 12-month follow-up interview, the overall retention rate was 83% with the intervention group having an 82% retention rate and 84% for the control group. This difference was not significant. Through 18-month follow-up, the overall retention rate was 81% with81% for the intervention group and 80% for the control group.
        The total cost of delivering the Kin Keeper ^SM intervention was $151/family. Family-level intervention costs included CHW training costs estimated at approximately $22, CHW costs of approximately $65 with recruitment and two home visits (breast cancer education home visit and cervical cancer education home visit), CHW supervision costs of $25, transportation costs of $25 associated with the two intervention home visits, cost associated with the breast and cervical cancer educational kits estimated at $12, and other material goods costing approximati
 

Discussion
Quantitative and qualitative/contextual factors in the design
Evaluating a multifaceted public health program should include not only quantitative but also qualitative/contextual factors in the design and should incorporate a quality intervention with guided implementation and dissemination; one such intervention is the Kin Keeper ^SM Cancer Prevention Intervention. Including a cost, the component is essential to inform key stakeholders of affordability when deciding to implement prevention programs. We utilized several evaluation methods to capture the nuances and tease apart multilevel components of an implementation of the intervention. Components included intervention fidelity as well as participant satisfaction, retention of a hard-to-follow population and cost assessment.
Importance of fidelity
We found sound intervention fidelity in the delivery of the intervention, integrity of the design, and consistency in training of personnel. A remarkable finding was how similar CHW responses were with open-ended questions, despite anonymity, phrases such as “found it empowering” and “should be implemented in churches, etc.” These benefits served to endorse and confirm that CHWs are a very important component to increase health literacy in the community and to refer underserved individuals to health providers. Likewise, CHWs are an excellent way to build capacity within the community and community health centers. Furthermore, with strong treatment fidelity of the Kin Keeper ^SM program, there is the potential for replication in a number of diseases and in a variety of venues, especially for those facing health disparities.
High participant satisfaction was not particularly surprising because participants working with CHWs whom they trust as peers often report high satisfaction. ^{(16, 17)}  The relationship between clients and CHWs is a powerful way to impart preventative medical information that can lead to higher rates of prevention behavior and screening. The strong relationship between clients and CHWs is an important pathway to allow underserved ethnic women to access health information and be guided to health services, where they are often intimidated by the barriers of access to a provider in a white coat. The role of the CHW and participant satisfaction also speaks to our strong retention rate.
Our findings of follow-up retention of participants can tell us how retention strategies can be tailored to vulnerable demographic populations as part of the research design. There is no “one size fits all” strategies to increase retention, but a careful examination of the culture of a community should be done to ensure strong retention. This can be accomplished by gathering information from stakeholders, health providers, and members of the community, and deliberately incorporating informed strategies into the study design. Likewise, racial/ethnic concordance between the participant (s) and CHW(s) allows for the familiarity of culture and,
In turn, trust. ⁽¹⁸⁾ For example, Blacks or Latinas may subscribe to a very different approach when interfacing with health promotion efforts and healthcare professionals from culturally and geographically different parts of a community. ^{(19, 20)}
Assessment of costs
Assessment of the costs to deliver the Kin Keeper ^SM intervention confirms the usefulness of reporting costs of interventions and providing information for replication and dissemination for medically underserved populations. ⁽²¹⁾ We found Kin Keeper ^SM is a very inexpensive educational intervention to promote breast and cervical cancer screening. In this study, the intervention was delivered to 2-4 family members, but could easily be delivered to a larger number of family members—making it, even more, cost-effective. The U.S. spends more than $15 billion on breast cancer treatment.²² Likewise, 12% of all women will develop invasive breast cancer at some point during their lifetime, with the costs of treatment between $20,000 and $100,000 over a lifetime. ⁽²²⁾ Next steps will assess Kin Keeper ^SM intervention costs in other cultural populations and different public healthcare settings to examine how portable and transferable it is in other locales and venues without added cost. Kin Keeper ^SM is a relatively small investment that could prove effective if it increases screening and in turn, reduces cancer treatment costs and mortality. ⁽¹⁴⁾
The initial recruitment design sought women linked to an ethnically concordant CHW from her community, whom she trusted, and included her close female family member (s). This was intended to achieve strong retention rates as a secondary benefit of the project. A well-thought-out evaluation design can demonstrate to stakeholders and funders what sustainable public health programming may look like. We have shown that CHWs are an excellent and inexpensive way to promote health and wellness by delivering strong prevention interventions to those who need it the most. Using a mixed-methods multilevel design, and synthesizing several components of the evaluation, allows for the information gathered to be used to enhance programming. Thus, we were able to deliver, not only intervention which met the needs of the research participants, but one that also had high fidelity and participant satisfaction, and allowed for high retention and good economic value. The intervention was designed to be attached to existing public health delivery systems that already employ CHWs who have a caseload of underserved clients. When this study is complete, future studies will assess the effectiveness of the program to increase breast and cervical cancer screening and cancer literacy, and assess the study design itself to inform improvements to the model.
Conclusion
Assessing implementation using guided conceptual frameworks is necessary to take the next steps to validate usefulness and feasibility of health promotion programs. We outlined
this evaluation process utilizing a design with “many moving parts”. Reporting on several implementation factors including fidelity, patient acceptability, retention, and cost, allows us to respond to the call for reporting information
about the replicability of health prevention programming. Over the course of the study, we continually met with the key stakeholders. The Kin Keeper ^SM project held bi-annual meetings with CHWs and their supervisors to gather observations from the field, troubleshoot errors, and disseminate current findings. These meetings have allowed for rich dialog by which to guide the follow-up retention and observations from the field that can inform future studies. For instance, CHWs have reported that their relationships with their clients and family members are very impactful in increasing screening behavior and cancer awareness (CHWs, personal communication, June 7, 2014). One limitation of the study was that we did not capture the directors of the public health centers and other key stakeholders research participants to inform the evaluation. The outcomes of this evaluation have the important implications in the context of the tenuous status of the Patient Protection and Affordable Care Act (PPACA) because it is an inexpensive way to deliver prevention programming. With the potential “scale-up” and dissemination, the intervention can be used in hard-to-reach urban and rural populations who do not have access to health prevention programs. Thus, we conclude that evaluating multiple components of a prevention program can inform procedures in replication and dissemination.
Acknowledgments
This project was supported by NIH NINR 1R01011323
Conflict of interest
The authors have no conflict of interest.
 

References