Partnering on the Evaluation of the Montana Tribal Tobacco Program: The Story of a Successful State-Tribal Collaboration

Introduction

Many models and frameworks for collaboratively designing culturally-relevant program evaluations exist. However, examples describing how these approaches have been applied through state-tribal collaboration are limited in the published literature. This paper shares the unique development and implementation story of an evaluation that involved collaboration between a state agency, a research and evaluation firm, eight tribal governments, and two urban Indian centers. The purpose of the paper is not focused on sharing results from the study, but rather sharing the process and offering lessons for state agencies and tribal communities seeking to design and implement tribally-engaged program evaluations. Representatives from all parties served as co-authors to tell a comprehensive story and acknowledge all perspectives of the journey.

Background

Montana is home to eight tribal governments representing eleven tribes: Assiniboine, Blackfeet, Chippewa-Cree, Crow, Gros Ventre, Kootenai, Little Shell Chippewa, Northern Cheyenne, Pend d’Oreille, Salish and Sioux (Montana Legislative Services Division & Margery Hunter Brown Indian Law Clinic, 2016). There are also five urban Indian centers servicing American Indian clients in the state. Approximately 90,259 people identify as American Indian and Alaska Native (AI/AN) alone or in combination with one or more other races, which is 8.4% percent of the state’s total population (U. S. Census Bureau, 2019).

Montana’s tribes are diverse but share a common connection to tobacco through traditional ties. The term “traditional tobacco” is used to define the diverse set of medicines contained in its mixture. Although AI/AN tribes use different types of medicines to make their own traditional tobacco, one common worldview is that the medicines are used for healing and protection (Boudreau et al., 2016; Great Lakes Inter-Tribal Epidemiology Center, 2013). The power or positive energy comes from both the plant and the person who utilizes the mixture in a reciprocal way. The understanding and acceptance of sharing power equally and intentionally between the plant and person has prevented traditional tobacco from being abused (Margalit et al., 2013; Wilson et al., 2019). Traditional tobacco use continues to have an important role in AI/AN culture (Lempert & Glantz, 2019; Unger et al., 2020). However, a long history of assimilation, colonization, and targeted strategies by tobacco companies has prompted the use of commercial tobacco in its place. The tobacco industry took this opportunity to develop commercial tobacco that targets American Indians, such as American Spirit and Red Man. These products inappropriately promote features, symbols, and names that have special and significant meanings to the American Indian population, while causing addiction to harmful substances (Centers for Disease Control and Prevention, 2019). Tobacco companies have also sponsored powwows and awarded grants to influence sales (Lempert & Glantz, 2019). This has resulted in disproportionately high rates of commercial tobacco-related disease (Boudreau et al., 2016; Lempert & Glantz, 2019; Odani et al., 2017).

Commercial tobacco use is a major risk factor for several chronic diseases and contributes to severe health disparities among AI/ANs living in Montana compared to the general population (U. S. Department of Health and Human Services, 2014). In 2019, 42% of AI/AN adults in Montana were current users of conventional cigarettes — almost three times higher than White Montana adults (15%). The high smoking prevalence of AI/ANs in Montana directly contributes to the rate of tobacco-associated cancers among this population, which is 60% higher than the rate among White Montanans (287 cases versus 177 cases per 100,000 people, respectively) (Montana Department of Health and Human Services, 2020). The risk of developing type 2 diabetes is also significantly higher (30% to 40%) for smokers compared to nonsmokers and can lead to additional serious health complications (Centers for Disease Control and Prevention, 2020a). The negative influence of commercial tobacco on American Indians in Montana is a key contributing factor to the shortened average lifespan (16 and 19 years shorter than their White male and White female counterparts, respectively) (Montana Department of Health and Human Services, 2019).

To reduce commercial tobacco use and improve health outcomes among AI/ANs in Montana, the Montana Tobacco Use Prevention Program (MTUPP) provides funding, education, and prevention resources to tribes and tribal organizations. MTUPP is housed under the Chronic Disease and Health Promotion Bureau within the Montana Department of Public Health and Human Services (DPHHS). DPHHS contracts with eight tribal governments and two urban Indian centers to share this work. The tribal contractors, or American Indian Tobacco Prevention Specialists (AI TPS), follow the guidelines of the Centers for Disease Control and Prevention’s (CDC) Best Practices for Comprehensive Tobacco Control Programs (2014) and provide culturally appropriate interventions to educate community members about the dangers of commercial tobacco use. They also promote the return to cultural practices and respect of the traditional use of tobacco. AI TPS use their community and cultural knowledge to create commercial tobacco prevention activities and messaging that promotes a positive relationship with medicinal and ceremonial uses of tobacco that can be passed down to younger generations. AI TPS emphasize the importance of traditional tobacco for ceremony, healing, and giving thanks while simultaneously teaching about the concerns of abusing commercial tobacco products. For the communities involved, they understand that teaching about traditional tobacco simultaneously means not abusing commercial tobacco products. Researchers have found that culturally-tailored programs which incorporate traditional stories, language, and values — including information on traditional tobacco — are most effective in preventing commercial tobacco use (Burgess et al., 2007; D’Silva et al., 2011; Filippi et al., 2013; Margalit et al., 2013). In addition, studies have shown that commercial tobacco prevention programs led by trained AI community members are more likely to increase trust and participant satisfaction with the program (Bosma et al., 2014; Fu et al., 2014; Nadeau et al., 2012).

Project Development

In 2016, AI TPS expressed the need to assess and understand the extent to which MTUPP efforts and activities were making an impact with AI youth, adults, and elders. In response, DPHHS contracted with James Bell Associates (JBA) to develop a culturally responsive evaluation that was led and supported by AI TPS. Tribes have the inherent right as sovereign nations to govern their people, lands, and resources (Cobb, 2005; James et al., 2014). In the context of research and evaluation, through something known as Indigenous data sovereignty, tribes also have the authority to govern the collection, ownership, and application of their data (Kukutai & Taylor, 2016; Tsosie, 2019). As a state government agency requesting an evaluation with tribal nations and communities, DPHHS wanted to ensure that the evaluation of MTUPP was done collaboratively, transparently, and responsively. The timeline in Figure 1 shares the steps taken from September 2016 through June 2020.

Figure 1.Project Timeline and Steps Taken

Initiate Project

Rigorous evaluation can provide meaningful information to support program improvement, future or sustained funding, and the sharing of program effectiveness and value to stakeholders, tribal leadership, and community members. Additionally, data gathered through program evaluation can be shared with program participants to support their ability to track progress, identify challenges, and celebrate successes (Tribal Evaluation Tribal Evaluation Workgroup, 2013). However, AI/AN communities sometimes hesitate to participate in evaluation studies given the negative history of research and evaluation by external investigators being imposed on this population (Christopher et al., 2008).

AI/ANs have always been researchers and have cultivated ways for evaluating and understanding the world through indigenous knowledge (Tribal Evaluation Workgroup, 2013). However, throughout history, AI/AN communities have been subject to unethical studies conducted by outside researchers and evaluators who have used AI/AN data in ways not approved by the tribe(s). Investigators frequently took data from a community, provided no benefit back to the members, and presented inaccurate and damaging findings to the broader population (Cochran et al., 2008; Ferreira & Gendron, 2011; National Congress of American Indians, 2006). As a result, approaches have been developed with an emphasis on building trust and creating evaluations that are meaningful and useful to tribal communities with the goal of supporting successful collaborations between these communities and external researchers and evaluators. These approaches include culturally-responsive evaluation (Bowman & Francis, 2015), community-based participatory research (Ferreira & Gendron, 2011), the Indigenous Evaluation Framework (LaFrance & Nichols, 2010), and the Roadmap for Collaborative and Effective Evaluation in Tribal Communities (Tribal Evaluation Workgroup, 2013). Key concepts in these approaches include respect for tribal sovereignty, the support and inclusion of indigenous ways of knowing, recognition and acknowledgement of historical context, relationship building, bidirectional learning between the community and investigators, and the inclusion of tribal members in research and evaluation planning, implementation, analyses, and interpretation (James et al., 2014; Tribal Evaluation Workgroup, 2013).

Compounding the damaging history of research in tribal communities, politics between state and tribal governments have long been the source of debate and power struggles over social, cultural, political, and economic matters impacting tribes’ abilities to fully exercise their own sovereignty and self-determination (Blumm, 2017; Hanna et al., 2012; Mackey, 2017; Taylor, 2008). This weary relationship has caused tribes to be skeptical about working with and imparting trust to state governments. Although recent efforts have attempted to strengthen and improve collaboration between states and tribes, it remains an ongoing issue (Crepelle, 2020; National Congress of American Indians., 2017; Tweedy, 2020). DPHHS recognized the importance of contracting with evaluators who possess extensive experience working with tribal communities in rural and urban settings and who understand and respect these complex historical contexts.

To that end, in November 2016 DPHHS created a Request for Proposals (RFP) to find culturally-sensitive evaluators to support the planning and implementation of an evaluation study. The RFP specifically called for evaluators who had extensive experience working with tribes and could ensure that the evaluation project used an approach that respected indigenous knowledge and included community involvement. Background knowledge of commercial tobacco prevention and familiarity with Montana tribes were also considered. DPHHS selected JBA, a cross-disciplinary evaluation firm with extensive experience conducting public health-focused evaluations in partnership with local, state, national, and tribal clients. JBA understood the importance of using a strengths-based approach and recognized the impacts of historical misuse of research in diverse tribal communities. With staff who identified as American Indian, including a member from a tribal community within Montana, they possessed both personal and professional experiences that provided the necessary contextual understandings to lead the effort. All of these qualifications were important and valued by AI TPS and positively affected how JBA’s team was received and their ability to foster trusting relationships.

Build Foundation

From the beginning, to support the success of the evaluation planning process, it was critical that DPHHS, JBA, and the AI TPS work collaboratively together. The three entities used an approach to evaluation planning that valued and incorporated the skills and perspectives of each group, prioritized relationship building as well as knowledge and skill building, established meaningful community engagement, and pursued the appropriate tribal approvals and protocols (Figure 2). The priority was to establish a relationship and trust between DPHHS, JBA, and the local programs. Prior to beginning work on the evaluation plan, JBA felt it was imperative to listen and learn more about how MTUPP work was being implemented by individual AI programs. To effectively connect with each individual AI TPS and to better understand their unique geographic locations, JBA and DPHHS hosted three small regional introductory meetings. The regional settings also allowed easier access for AI TPS and their tribal health colleagues to attend.

Figure 2.Strengths and Perspectives of Each Partner

DPHHS recognized the importance of state-tribal relations and how meeting environments can influence this relationship. Providing an equal playing field, as well as a culturally-relevant space for all parties was important. Therefore, JBA and DPHHS conducted the introductory meeting, as well as future meetings, with cultural customs in mind: prayer, a shared meal, comfortable seating arrangements, and opportunities to work and laugh together. Attendees received an overview of Indigenous research compared with Western evaluation studies, information regarding local data collection, and the overall approval processes. AI TPS, individually and collectively, communicated how they interpret the MTUPP workplan activities and identified their goals and priorities for the evaluation project. They also shared their tribal beliefs and practices around traditional tobacco uses and how these were being incorporated into program content. JBA collated and categorized the information gathered from the meetings to find commonality among the various programs, understand where needs existed, and identify areas where an evaluation might be focused.

Inspire Partnership

A second face-to-face meeting took place when JBA and DPHHS held a workshop to guide AI TPS in developing a cross-site logic model (Figure 3). Logic models are often used as tools to articulate a program’s underlying theory of change for program development, management, and evaluation purposes (McLaughlin & Jordan, 1999). They also serve a purpose in community-based programs to enhance communication across stakeholder groups and strengthen synergy between researchers and the community when developed collaboratively (Fielden et al., 2007; Kaplan & Garrett, 2005). The workshop again took place in an inviting location, with the opportunity to build relationships, have open discussion, encourage relaxing and fun activities, and share food. As a group-facilitated process, each AI TPS began first by designing their own program logic model based on how commercial tobacco prevention services were being implemented within their community. The group then built a comprehensive cross-site logic model by posting their own inputs, activities, and outcomes on a sticky wall located on one side of the meeting room. The work done to develop the logic model provided a road map for AI TPS to visualize relationships between resources, key activities implemented across tribal communities, outputs associated with those activities, and their desired short-, intermediate-, and long-term outcomes. The process itself was informative and encouraged multiple perspectives to be represented, particularly the perspectives of AI TPS as local implementers. The logic model ultimately gave DPHHS and JBA a tool that directed the evaluation planning project.

Figure 3.MTUPP Cross-Site Program Logic Model

With the logic model as a guide, JBA worked with the AI TPS to understand the connections between the activities AI TPS were implementing and the short-/intermediate-term outcomes of the program (Figure 4). The process of developing the logic model and seeing the connections allowed AI TPS to identify and better understand how their work impacts community change. AI TPS and DPHHS identified the activities and outcomes that they and other stakeholders in the community were most interested in evaluating. AI TPS were given three different colored sticky dots and asked to place (1) a colored dot on the outcomes they would be most interested in exploring through the evaluation, (2) a different colored dot on the outcomes they thought their Tribal Health leadership would be most interested in, and (3) the last colored dot on the outcomes they thought community members would be most interested in. Ultimately, this process allowed the group to discuss the various priorities of stakeholder groups across communities and come to consensus on the primary goal of the evaluation: to understand which activities were most effective in educating adults and which were most effective in educating youth about commercial and traditional tobacco use.

Figure 4.Linkages between Activities and Short-/Intermediate-Outcomes

Design Tools

Based on the linkages and outcomes that were prioritized by AI TPS and DPHHS, JBA drafted process and outcome evaluation questions focused on the local work being done by AI TPS with adults and youth in their communities. Through a series of emails, group webinars, and in-person meetings, JBA encouraged additional feedback on the priorities of the evaluation and evaluation questions, and proposed data collection methods, which included two surveys: one for adults and one for youth. JBA reviewed existing relevant instruments such as the CDC National Health Interview Survey, the CDC Youth Tobacco Survey, the Global Youth Tobacco Survey, and the Cultural Connectedness Scale to inform the development of survey questions and response options appropriate to AI adults and youth in Montana (Centers for Disease Control and Prevention, 2020b, 2021; Snowshoe et al., 2015; World Health Organization, 2021). Surveys used a retrospective pre/post design to capture self-reported changes in knowledge, attitudes, and beliefs among participants related to traditional and commercial tobacco, exposures to commercial and traditional tobacco use, and intent to use commercial tobacco in the future. Adult participants were additionally asked about current commercial tobacco use and readiness to quit. The adult survey had 21 items and the youth survey had 18 items; both were 2 pages long. The youth survey was pre-tested with a small group of AI high school students in two of the AI communities to increase instrument validity. The surveys went through several rounds of AI TPS feedback and edits before the final versions were completed and accepted. This process ensured transparency and inclusion of the people and communities that were to be served by the evaluation project.

Data Sharing and IRB Approvals

Four types of entities exist to review, approve, and provide oversight for research conducted with AI/AN individuals and communities: Tribal Nations, Tribal Colleges, Tribally-Based or Focused Organizations/Departments, and the Indian Health Service (IHS) (Around Him et al., 2019). As stated by Harding et al., “Only tribal nations themselves can identify potential adverse outcomes, and they can do this only if they understand the assumptions and methods of the proposed research” (2012, p. 6). These review bodies provide enhanced protection and benefits to the community, legitimize tribal research, and support bidirectional education for both tribal members and scientists (Morton et al., 2013). Many tribal nations have their own Institutional Review Boards (IRB) or Research Review Boards (RRB) to review and approve research and evaluation conducted with their members.

DPHHS and JBA worked with the AI TPS to identify the appropriate review and approval body for their community or urban Indian center. Three Montana tribal governments had tribe-specific Institutional Review Boards at the time of the evaluation. The other tribal communities identified a regional tribal Institutional Review Board located in Montana to act on their behalf. The tribe-specific review boards supersede the regional review board. The team followed the specific guidelines and recommendations of each IRB in order to conduct the evaluation ethically and to respect tribal sovereignty. DPHHS and JBA also received letters of support from all participating tribal communities and urban Indian centers and developed data sharing agreements (DSA) with each site.

Deemed best practice for collecting data from AI/AN communities, DSAs are formal contracts that protect against the misuse of data by detailing who has ownership and will have access to the data, and how the data will be collected, stored, analyzed, and disseminated (Urban Indian Health Institute, 2020). For this project, the content of the DSA was decided upon by DPHHS, AI TPS, and Tribal Health Directors. The DSAs specified that the tribe/tribal organization would maintain sole ownership and control of the data and it would only be shared with DPHHS for data analysis. The AI TPS and Tribal Health Directors provided feedback, and all were given an opportunity to make edits to the DSA before it was officially signed by both parties. In conjunction with the DSA document, the regional IRB also requested an official letter of support from their Tribal President/Chair. Once the DSA and letters of support were signed, applications to four Tribal IRBs were submitted (three Tribal Nations and one Tribally-Based Organization).

Collect Data

IRB approval required all data collectors to be trained in research ethics. As known and trusted members of their community, the AI TPS were most appropriate and most qualified to collect the evaluation data from participants. JBA and DPHHS worked together to create a third and final in-person training for AI TPS to help prepare them for their role as data collectors. This last training supported local capacity building and included relevant sessions on human subject research. While several commonly-used human subject research training curriculums exist, AI/AN community partners have raised concerns about the lack of cultural and contextual relevance and the absence of discussion about community risks and benefits (Pearson et al., 2019). They have also questioned their own ability to apply the concepts to the research for which they provide oversight (Hatcher & Schoenberg, 2007; Pearson et al., 2014). Since this lack of culturally relevant research guidelines can be an obstacle to good scientific research, JBA and DPHHS decided to use the rETHICS: Research Ethics Training for Health in Indigenous Communities curriculum for this project. rETHICS is a culturally-tailored research ethics training grounded in Indigenous knowledge and values and discusses concerns as identified by AI/ANs related to conducting ethical research with their communities. The training curriculum has proven effective in increasing AI/AN participants’ knowledge and trust in the research and IRB review process and has been shown to significantly increase participants’ self-efficacy in putting their research ethics knowledge to use (Pearson et al., 2019). The JBA consultants became certified rETHICS trainers and facilitated the rETHICS training for AI TPS.

The goal of the rETHICS training was to provide AI TPS with the opportunity to gain research knowledge and to engage in important skill building to support their participation as data collectors in research involving their communities. Completing the culturally-tailored training curricula prepared data collectors to gather valuable information from community members during this and future evaluation projects. It prompted discussion and problem-solving techniques about issues that might arise during survey administration, and it increased individual knowledge of research and capacity to participate in the research process. In addition to the rETHICS curricula, this final in-person training included sessions on proper administration of the survey tool, data entry, data storage, and the protocol for gathering informed consent.

Reflections and Lessons Learned

Together DPHHS, AI TPS, and JBA were successful in their collaboration to develop and implement an evaluation plan that sought to answer questions important and relevant to the AI TPS. The evaluation was done in a way that was culturally appropriate, received local support and approvals, and provided technical and practical skills to tribal members. In the following section, we share our reflections and recommendations for state agencies, tribes, and urban Indian organizations seeking to evaluate a tribal public health program:

1. Recognize the value and role that each entity brings to the evaluation planning process. DPHHS recognized the important role of the AI TPS in this process. The AI TPS were experts in how MTUPP activities were being implemented, the goals and priorities for the evaluation, the local data collection approval processes, and traditional tobacco uses, beliefs, and practices in each community. DPHHS recognized the value of contracting with an external evaluator who had substantive experience in supporting evaluations in tribal communities. For this project, JBA served as content experts in evaluation, were able to provide appropriate trainings, and supported DPHHS and the AI TPS in obtaining necessary approvals to engage in the evaluation study.

2. Recognize the unique relationship between the state and tribes, and the importance of tribal sovereignty in this context. Engaging tribal leadership early in the planning process and fostering communication and relationship-building between state and tribal governments is imperative to support transparency and ownership of the project by both parties. In hindsight, our group should have allotted more time up front for communication and dialogue with tribal councils, not just with tribal health officials. Asking more questions early on to determine individual systems of authority and to find out who should be brought into the discussion at which points of the project would have benefited our endeavors. Presenting the project to tribal leadership is one way that state governments can partner with local program staff. Doing so makes it possible to gather the support and critical feedback necessary to improve the project and ensure findings are meaningful.

3. Seek to understand the program/intervention together. Often, the way a program/intervention is delivered in the real world is different than the way it was designed and intended to be delivered. JBA initially spent time gathering information and feedback on the goals and focus of the evaluation study from both DPHHS and the AI TPS. It was incredibly valuable to start the evaluation planning process by reviewing the overarching logic model for MTUPP together and then building it out to include the specific activities AI TPS were conducting and the short- and long-term outcomes of the program in their communities. These conversations led to a greater understanding of the program by both groups and informed the evaluation goals, questions, and measures to ensure they were appropriate, scientifically rigorous, and locally meaningful. Ultimately, the evaluation questions and plan aligned with the goals and interest of the tribal and state entities.

4. Build in time and resources for properly completing the tribal IRB process and establishing Data Sharing Agreements with all participating tribes and urban Indian organizations. Due to the historical misuse of data and evaluation practices, completing the IRB application is arguably the most important aspect of any evaluation process. Be cognizant of the time it takes to go through this step and be thoughtful in understanding the IRB requirements as they might vary by IRB. IRB program administrators and committee members often have heavy workloads and committee meetings may only occur on a monthly or quarterly basis. Building in adequate time to account for these various factors will help the evaluation stay on track.

5. Use evaluation planning as an opportunity for skill building and to support self-determination. Ensuring that the evaluation was tribally-engaged and utilizing a community-based participatory research (CBPR) framework, meant that emphasis was placed on engaging the community in all phases of the evaluation, including data collection. It was critical that AI TPS participated as true partners in the data collection process. Ferreira & Gendron (2011) explain that using CBPR as an orientation to research in indigenous communities means that “trained community members participate in the research process in an equitable fashion as full collaborators, not just as ‘research participants’” (p. #) However, many AI TPS had no formal background or experience in research or data collection practices, requiring additional training in this area. AI TPS served as data collectors, which aligned with the community-engaged and culturally-driven process used and also supported capacity building for local staff. One AI TPS shared that through this process, “[I] learned the intricacies of evaluations, how you talk with participants, how you collect and store data. This is what many federal grants require, so it not only helped with tobacco but other funding sources that [we are] a part of.” AI TPS agreed that participating in the evaluation planning and data collection allowed them to grow professionally and to show their communities that they were moving to a standard of implementing locally informed efforts through evaluative processes.

6. Allow time at the onset and throughout the process for relationship building between tribal, state, and evaluation staff. DPHHS and JBA devoted time throughout the process to meet with AI TPS in person, through webinars, and email communication to listen and learn, articulate next steps, and ensure transparency and inclusion. Adopting cultural ways of respect, including a gracious space, making time to laugh and tell stories, sharing food, and establishing an equal playing field were important elements for the success of this evaluation.

This project incorporated an evaluation process at the local level with non-academic, on-the-ground employees (AI TPS and DPHHS) leading the way. Ultimately, and for the purpose of this paper, the survey tools and data outcomes were less important than the process that was taken. The approach led to outcomes of increased cultural knowledge for DPHHS staff, improved research knowledge for AI TPS, and a stronger relationship between the two groups. The project went beyond developing an evaluation plan to establishing a lasting relationship, built on trust and open dialogue, that facilitates future partnerships.