Introduction
Patient-centric engagement research seeks input from patients and various stakeholder groups in the full range of research activities from conceptualization to dissemination (Patient-Centered Outcomes Research Institute, 2018). This approach is aligned with community-based participatory research; both share an aim of integrating patient needs, expertise that comes with lived experience, and empirical research-based knowledge to inform and direct future inquiries (Frank et al., 2014; Israel et al., 1998). O’Haire et al. (2011) highlight the importance of engaging stakeholders to not only generate new research questions meaningful to those most impacted healthcare delivery mechanisms, but also to prioritize those questions based upon the critical lens of lived experience. Through scaffolded engagement activities, stakeholders are supported in collaborative question generation and activities that inform research with the goal of making discoveries that inform health care specific to patient-identified needs.
Patient-centered engagement research can be challenging, however, due to authority bias — cultural norms prioritizing the knowledge and experiences of medical professionals over those of patients, their families, and other stakeholders (Rogers et al., 2020). Authority bias creates the risk of stakeholder self-censorship, negating the benefits of patient-centric engagement projects. Identifying methods and procedures to reduce the risk of authority bias and ensure the meaningful participation of patients and other stakeholders is vital to achieving the aims of engagement research.
Our interdisciplinary core project team has completed several patient-centric quality improvement projects aimed at improving educational experiences and academic attainment for children with cancer and survivors of childhood cancer (Ruble et al., 2023; Thornton et al., 2022). These projects sought to improve the ways in which pediatric oncology care supports the return to school after cancer treatment as well as the long-term impacts of cancer and treatment. Prior projects have resulted in the creation of schooling guides for parents of children treated for cancer as well as the development of a hospital-school liaison program.
Prior projects pointed to a need for quality improvement targeting medical provider communication regarding the neurocognitive impacts of cancer and treatment and the resulting difficulties with thinking and learning. Changing hospital procedures, policies, practices, and culture requires evidence that these changes will be meaningful — thus, our team seeks to conduct comparative effectiveness research to determine and demonstrate the benefits of changes in practice to improve quality of care. We therefore focused on engaging stakeholders in a project to: 1) evaluate current research regarding communication of neurocognitive late effects of childhood cancer and treatment; and 2) establish research priorities to improve communication of neurocognitive late effects.
The core project team included researchers who are also practitioners in the fields of oncology and education. Two PhD-prepared pediatric oncology nurse practitioners and one pediatric neuropsychologist specializing in pediatric oncology care provided the interdisciplinary core project team with expertise in the medical care of children treated for cancer. A doctoral level education specialist directing a hospital-school liaison program and a professor of education offered the interdisciplinary core project team expertise in educational practice and policy. The core project team was further supported by a research assistant with a background in the neuropsychological care of children treated for cancer.
The interdisciplinary core project team recruited stakeholder groups of parents of children treated for cancer, K–12 educators (including teachers and administrators), and pediatric cancer clinicians (including oncology nurse practitioners, psychologists/neuropsychologists, and oncologists) (see Figure 1). Our core team worked with stakeholders in their areas of expertise during discussions and group exercises. We sought methodologies that would increase group cohesion and decrease the risk that authority bias would cause the parent and educator stakeholders to defer to the medical providers’ input. We chose to modify the Stakeholder Engagement in quEstion Development and prioritization (SEED) method (Zimmerman & Cook, 2017) by incorporating the Delphi method, a semi-anonymous qualitative methodology to address concerns of authority bias disrupting potential findings of SEED method engagement and to reduce the time participants spent engaged in synchronous activities. This manuscript provides a brief introduction to both methodologies and shares our approach for integrating them to successfully set research priorities with a diverse group of stakeholders.
The SEED Method
The SEED method is a participatory research approach that engages communities in identifying stakeholder priorities and creating research agendas (Zimmerman & Cook, 2017). Consisting of six steps (see Figure 2), the SEED method outlines how to engage stakeholders with various levels of research expertise in the development and prioritization of patient-centered research inquiries and comparative effectiveness research questions.
As noted in Figure 2, the SEED Method takes all stakeholders through a guided process of conceptualizing factors involved in a topic as well as the generation and prioritization of research questions. This method fits well with the aims of our project, but we wanted to be sensitive to the potential risk of authority bias interfering with full stakeholder participation. We had additional concerns regarding the ability of our large and geographically dispersed stakeholder group to participant in multiple synchronous activities. To address these concerns, we integrated the Delphi method into steps 2 and 5 of the SEED method.
The Delphi Method
Delphi is a method for determining the collective opinion of panel members, typically used to establish and measure consensus among experts (Nasa et al., 2021). This method offers the benefit of a reduction in authority bias by allowing for anonymity in the process of agreeing or disagreeing with positions or statements presented to the group (Dalkey & Helmer, 1963). Survey items are generated through participant input. Participants are then asked to individually rate the survey items by importance to the topic. The consensus around items is calculated, lower-rated items are dropped, and the survey is redistributed to participants for another round of data collection. This process is continued until the desired level of consensus is reached. The Delphi method allows participants to rate items’ importance based on personal perspectives without the influence of stakeholders who may hold greater societal authority, thus reducing the risk of authority bias.
The Delphi method also allowed us to address issues related to time and geography. In many cases, SEED method participants are part of a geographically based community, such as addressing the healthcare needs of a particular rural community (Zimmerman et al., 2020). Our project, however, focused on the community formed around childhood cancer — a low-incidence disease. To ensure our findings were generalizable, we recruited stakeholders from various regions of the U.S., therefore necessitating methodology changes to deal with barriers of time and distance. Integrating the Delphi method and using a hybrid virtual format allowed us to fully include stakeholders who could not travel to our site.
In the following section, we outline our implementation and modification of the SEED method, including how we added the Delphi survey methodology to address concerns about authority bias and replaced synchronous focus groups with asynchronous Delphi surveys to accommodate a geographically dispersed stakeholder group.
Adapted Seed Method Process
Step 1: Identifying and Engaging Stakeholders
We used the SEED method stakeholder matrices to identify stakeholder groups (Zimmerman & Cook, 2017): parents of children treated for cancer, oncology clinicians, and K–12 educators. Stakeholders from previous, related projects were contacted and re-engaged. In some cases, previous stakeholders were unable to continue. To ensure that each group had the necessary number of participants, we used purposive sampling and identified new stakeholders through relationships with the core project team or previous stakeholders. Criteria for selection included regional and demographic diversity. Each stakeholder agreed to participate in synchronous, online stakeholder meetings; the Delphi survey process; a hybrid convening to establish research questions; and a final survey to establish research priorities. Stakeholders were financially compensated for their time.
Step 2: Consult
The SEED method recommends beginning with focus groups that include additional external stakeholders to generate broad topics for potential research questions and prioritization. These focus groups were replaced with online, synchronous group meetings and Delphi surveys.
Synchronous Online Group Meetings
Synchronous online group meetings were held via Zoom. This allowed all stakeholders to meet while avoiding barriers to participation such as travel and access to childcare. During the first synchronous Zoom meeting, stakeholders introduced themselves and their connection to or interest in the project. The findings of a systematic literature review addressing communication of neurocognitive late effects (previously conducted by two core project team members) were then shared using a multimedia presentation to support group understanding of the state of the science. Finally, stakeholders collaborated to generate a list of broad, potential topics for a comparative research agenda based on the state of the science.
Delphi Surveys and Analyses
The core project team created the first iteration of the Delphi survey using the list of topics generated by the stakeholders during the synchronous online meeting. This survey consisted of 16 items. Stakeholders were asked to rate the importance of each topic on a 7-point Likert scale. The digital survey was sent to stakeholders with individualized links via email directed from Qualtrics. Each stakeholders’ “role” (e.g., parent, clinician, or educator) was recorded in place of individual names.
Survey results were analyzed by establishing means for each item (a grand mean), followed by establishing means for each item by stakeholder role (i.e. parent stakeholder group, clinician stakeholder group, and educator stakeholder group). We then compared the grand means to the subgroup item means to establish consensus between the stakeholder groups before removing items due to low grand means.
In cases where item importance was rated differently by the stakeholder groups, parent ratings were given preference to ensure a focus on patient experiences over clinician or educator opinion. Of note, the clinician stakeholders rated the importance of clinician training higher than the parent or educator stakeholders. This may point toward authority bias in which the parent and educators assumed the clinical team was already well-trained. Based on the results of the first round of the survey, a second round was created, as per the Delphi method. Items rated lowest by the three stakeholder groups were removed (see Table 1).
The second round of the Delphi survey was emailed to stakeholders using the same procedure as before. Stakeholders were given access to the results of the first round (including their personal rating of each item) to consider when rating the items in the second round. Using the same analysis method as the first round, the analysis of the second round established a strong consensus. The findings from the Delphi survey became the scaffolding for the SEED method convening.
Step 3: Conceptual Modeling
The SEED Method recommends walking all stakeholders through a series of conceptual modeling activities to support participant understanding of the research question generation process. To accomplish this, we held an all-day, synchronous, hybrid convening. A hybrid model allowed for most stakeholders to meet in-person with four members participating remotely, but synchronously. A website was created to share meeting resources and information for all participants regardless of method of participation (see Figure 3).
During the first part of this meeting, one member of the core project team led the stakeholders through a whole-group, direct-instruction activity to demonstrate and explain how to engage in conceptual modeling. This activity was based on SEED method materials (Zimmerman & Cook, 2017). The stakeholders engaged in a multi-media lesson using an example topic of vaccine hesitancy, which assisted stakeholders without research experience in understanding types of factors and variables considered in a conceptual model.
Following this brief didactic period, stakeholders broke into three groups. Each team included a mix of stakeholders representing parents, clinicians, and educators, and was facilitated by a core project member. To accommodate our hybrid meeting model, two of the teams were comprised of in-person attendees who met in a round table discussion and used materials such as markers, sticky notes, and poster paper to facilitate group work. Those participating virtually met via teleconferencing platform (Zoom) and used Google Jamboard (a virtual whiteboard that allows for synchronous collaboration) to create shared visualizations of their work. This group was facilitated by a core project team member who was physically at the convening and could serve as a bridge between in-person and virtual activities. An additional core project team member observed the three groups, documented conversations, and captured images of work samples.
The three groups were first asked to brainstorm factors that influence healthcare communication. Each factor was written on a sticky note, either physically (see Figure 4) or virtually using Google Jamboard (see Figure 5). Next, teams were asked to discuss the potential relationships between factors and the overall topic of neurocognitive late effects communication. Factors directly influencing communication were placed closer to the “communication” note, while factors indirectly influencing communication were placed farther away.
Step 4: Generating Research Questions
During the second half of the convening, a core project team member used SEED method materials to deliver a lesson about the process of writing research questions. Resources to aid in writing research questions were provided on paper and digital copies were available on the dedicated website. The core project team then reviewed the findings of the Delphi surveys and consensus topics with the stakeholders.
Stakeholders were asked to rejoin their teams and each team was assigned a topic from the Delphi study to develop potential research questions. Because there were three teams and four topics, one team was given two topics to address. Core project team members facilitated the use of the shared resources (peer-reviewed journal articles related to each topic, summaries of the journal articles for quick reference, and research question generation flow-charts for question development). After creating potential patient-centered research questions and comparative effectiveness research questions, each team presented their work to the rest of the group.
Step 5: Prioritizing Questions
Following the full-day, synchronous, hybrid convening, the core project team developed a new survey with the eight stakeholder-generated comparative effectiveness research questions. Stakeholders were asked to rate these questions in order of importance using a 7-point Likert scale (with one being the most important). We included definitions of key terms and interventions mentioned in the questions for stakeholders to review before they ranked the questions. After all stakeholders completed the survey, we analyzed the mean ratings of each item (see Table 2). Again, the use of anonymous rating allowed stakeholders to consider the importance of the proposed research questions without risk of authority bias influencing their responses.
Step 6: Disseminate Research Agenda
Stakeholders were engaged within the dissemination process in several ways: 1) a finalized list of the prioritized research questions was sent to the stakeholders to participate in writing up the findings for relevant publications; and 2) stakeholders were asked for recommendations of organizations that might assist in sharing the project’s findings. Currently, the core project team is collaborating with several organizations to share information relevant to the project and research agenda as well as present that information to relevant research stakeholder audiences. Additionally, the core project team is planning projects to investigate the stakeholder-set research priorities.
Conclusion and Discussion
The aims of our patient-centered engagement project were to: 1) evaluate the current empirical literature on communication of neurocognitive impacts of childhood cancer between parents, educators, and medical providers; and 2) develop a prioritized list of comparative effectiveness research questions related to this topic. By modifying the SEED method through the inclusion of Delphi surveys, we were able to accomplish these goals while also limiting the risk of authority bias and facilitating collaboration among stakeholders, regardless of location.
Value of Stakeholder-Driven Work
Our stakeholders provided valuable insights into the challenges related to the communication of cognitive late effects from cancer and treatment. Their involvement was critical for identifying key topics and prioritizing a research agenda. Including three distinct groups of stakeholders (based upon their role and experiences with childhood cancer) as well as purposive recruitment led to collaboration with stakeholders who offered diverse perspectives and facilitated nuanced understandings of the challenges and potential interventions. While we were foremost interested in the perspective of the patients’ parents, it was clear that the educator and clinician groups offered balanced perspectives — which aided in identifying critical topics. For example, the parent group rated training of clinicians much lower than that clinical group, possibly because of an assumption that medical personnel were already trained to discuss neurocognitive late effects of childhood cancer.
It is important to note that our prior development of an experienced stakeholder cadre was critical in our ability to accomplish so many tasks within the one day convening. Most stakeholders recruited for this project had worked with our core project team on other quality improvement initiatives, thus already establishing trust, a common understanding, and familiarity with quality improvement project processes. The newly recruited stakeholders were quickly integrated into the group as the group culture and norms had already been established.
The benefit of an interdisciplinary core project team should not be overlooked. Starting with a core project team that aligned with both the clinician and educator stakeholder groups assisted in facilitating conversation. Our core project team modeled interdisciplinary conversation and collaboration at each stakeholder meeting, demonstrating that one need not be an expert in each field to contribute to the group’s understanding of the topic.
Funding, Logistics, and Focus
This project was funded by the Patient-Centered Outcomes Research Institute (PCORI). Funding allowed us to compensate stakeholders, purchase necessary materials for the in-person convening, and compensate time and effort for core project team members. Timelines were driven by our grant. Our convening developed both patient-centered outcomes research questions and comparative effectiveness research questions, with our funding emphasizing the importance of developing and disseminating the set of comparative effectiveness research questions.
Challenges
Not all members of our core project team were familiar with both the SEED and Delphi methods. We spent time speaking with experts in both methods before determining the best way to combine the two for our project, which allowed for the entire core project team to feel confident with the two methodologies before engaging in interdisciplinary.
As mentioned, given that childhood cancer is a low incidence disease, we sought out stakeholders from across the U.S. This led to stakeholders being separated by distance, time zones, and competing responsibilities. Our core project team leveraged our expertise in remote collaboration and distance education to build digital supports (i.e. the website, Google Jamboard, digital resources) to address these logistical challenges caused by distance and time.
Suggestions for Future Work
As mentioned, we worked with a cadre of experienced stakeholders who we’d engaged in various prior projects. Those interested in this type of work should consider the importance of the development of such a cadre. Without stakeholders experienced in working with our core project team and who felt comfortable with the group at large, our project would most likely have taken much longer to complete.
Our project was intentionally designed to accommodate stakeholders who could not attend in person. Yet, which stakeholders could convene in-person and who needed to participate remotely changed in an unpredictable manner due to fluctuating personal circumstances. Thus, it is critical that any design intended to provide access to stakeholders participating at a distance be designed in a flexible manner to allow any project participant to suddenly switch to remote participation.
Final Thoughts
The success of the current project suggests that a modified version of the SEED method combined with Delphi surveys can be a useful mechanism for engaging heterogeneous communities in research. This unique approach can be especially beneficial for communities that are not confined to a specific geographic region, but rather, are formed around shared lived experiences, such as a rare disease diagnosis. The integration of Delphi with the SEED method offered additional safeguards against authority bias and reduced the need for synchronous meetings.