Learnings From a Developmental Evaluation of a Dynamic, Arts-Based Death and Grief Awareness Initiative: The Reflection Room Project

Background

Many innovative projects start as pilots to test new practices and/or procedures without a significant investment of resources (Zbrodoff, 2012). Innovative pilot projects generally need flexibility to evolve, with outcome goals being broad, especially at the beginning of the work (Gamble, 2008). We engaged in one such example of an innovative pilot within our Reflection Room project. The pilot led to further phases of inquiry, eventually coalescing into a scalable dying, death, and grief resource ready to spread. After an initial “spark” arising unexpectedly out of a knowledge mobilization activity, we had broad ideas about piloting an arts-based installation to increase visitors’ comfort in thinking and talking about dying and death. We began designing the installation yet did not have clearly defined processes or outputs to measure, which made it challenging to engage in traditional methods of evaluation. This paper outlines our process of engaging experts-by-experience in developmental evaluation (Gamble, 2008; Patton, 2006) as a means of cyclical learning, adaptation, and development through the multiple phases of the Reflection Room project. We provide details about how we conducted this method of inquiry as well as recommendations and lessons learned along the way.

The beginnings of the Reflection Room

The Reflection Room was first conceived as a method of mobilizing knowledge at the Canadian Hospice and Palliative Care Association conference in Ontario, Canada, in 2016. Drs. Paul Holyoke and Barry Stephenson were wanting to share emerging findings of their study of organization-level best practices for supporting high-quality spiritual care at end-of-life (Holyoke & Stephenson, 2017). Rather than delivering an oral presentation, they intended to provide a space in which conference attendees could experience elements of four best practices including: 1) cultivating sacred spaces; 2) allowing questions to emerge; 3) enabling processing of experiences; and, 4) witnessing. This first version of the Reflection Room occupied a meeting room in the conference hotel. Endeavoring to create a sacred space (i.e., a space that felt special), the Reflection Room included images of nature, a colour scheme featuring predominantly red and white, with gray and black as secondary colours, and quiet classical music. Within the room there were large comfortable chairs and a written invitation to participants to read stories and/or write on blank reflection cards (based on the invitation to ‘reflect on your past year of caring for people who were dying and their families’), which facilitated allowing questions to emerge and the processing of experiences. Finally, the reflection cards could be shared by visitors on a “ribbon wall” (see Image 1), which supported acts of witnessing. Drawing on inspiration from Candy Chang’s “Before I Die” installations (Chang, n.d.), the Reflection Room was set up as an unfacilitated experience.

Image 1.Ribbon Wall

Over the course of the three-day conference, many attendees, predominantly healthcare providers working in hospice and palliative care, visited the room and wrote reflections. A notice was posted that reflections would be collected and made public to support future ideas for operationalizing these best practices. After the conference, the reflection cards were scanned, transcribed, and shared with the hospice palliative care community who attended the conference. Feedback from visitors and observations of engagement indicated visitors appreciated having a space to reflect and remember. Many visitors reflected not just as healthcare providers, but as people who had personally experienced dying, death, and grief. It seemed attendees welcomed more opportunities for reflection within both community and healthcare spaces to support more widespread thinking and discussions about dying and death. The impetus for a larger project came from the enthusiastic engagement by conference attendees; one reflection card gave a specific recommendation: “There should be one of these in every hospital.”

Based on this reception and with funding from Saint Elizabeth Health Care through its annual unrestricted grant to the SE Research Centre, our research team decided to turn this one-time experiential knowledge mobilization effort into an innovative arts-based pilot project with a transportable Reflection Room installation designed for healthcare and community spaces to increase visitors’ comfort in thinking and talking about dying and death. However, as a new and evolving idea, there was uncertainty about many aspects of the pilot project: what design elements would support an adaptable installation for complex and diverse settings that would be acceptable to visitors, what processes would be important to potential outcomes, and what would be the best way to evaluate this innovative pilot?

Designing and Evaluating Pilot Projects

Pilot projects are often implemented on a small scale and limited basis to learn more about how and whether they address a particular problem or meet a particular goal prior to potential large-scale implementation (Martin, 2005; World Health Organization, 2011). The World Health Organization recommends pilot projects in healthcare be designed to support evaluation with 1) user friendly components that are straightforward to implement; 2) a context-relevant testing environment; 3) readily available resources; and 4) rigorous documentation to support process and outcome measurement (World Health Organization, 2011). Typically, project evaluations are guided by logic models that outline how activities and processes are related to the project’s objectives, depicting linear causal relationships between essential parts of the program and outcomes (Gamble, 2008). Often, health researchers first engage in formative evaluation, done early in the project, to understand and address any issues related to the context or design of the project (Fagen et al., 2011; Guijt et al., 2012; Martin, 2005). This may also include process evaluation to guide implementation improvement (Saunders et al., 2005). Once addressed, researchers usually conduct a summative evaluation to measure outcomes (Fagen et al., 2011; Guijt et al., 2012; Martin, 2005). Evaluation is necessary to determine replicability and predictability of results (Patton, 2006) and to know if a project should be scaled up, adapted, or discontinued (Guijt et al., 2012).

Applying traditional project evaluation approaches in the case of the Reflection Room project and other innovative pilots like it, would mean there would be little room for flexibility and adaptability in design and approach (Fagen et al., 2011). Innovative projects often require refinement, particularly in the early stages where development is constant to meet different needs of people, dynamic interactions, and complexity (Fagen et al., 2011; Patton, 2006). In fact, a stable, static initiative is sometimes not the purpose of innovative pilots (Patton, 2006; Patton et al., 2015). If projects contain some degree of uncertainty in the goals, outcomes, or context, evaluations based on standardization, consistency, and concreteness can hinder development and innovation (Gamble, 2008; Patton et al., 2015).

A linear approach to project design and evaluation work well when problems are understood and the solutions to the problem are defined within specific boundaries (Gamble, 2008). However, this was not the case for the Reflection Room project pilot due to uncertainty about 1) the design elements for diverse settings and people, 2) the full scope of solutions to increasing comfort thinking and talking about dying and death, and 3) the best way to evaluate this project in different complex healthcare and community environments. We needed a method of evaluation that could support rigorous adaptation and development. One such approach is developmental evaluation.

Developmental Evaluation

Developmental evaluation was first defined by Michael Patton (1994) to explain a process that supports teams to “… conceptualize, design, and test new approaches in a long-term, on-going process of continuous improvement, adaptation, and intentional change….” (Fagen et al., 2011, p. 317) Developmental evaluation combines critical thinking from evaluation work and creativity from development work to create an evaluation approach that is non-linear, emergent, and focused on further developing dynamic programs and initiatives in complex environments (Gamble, 2008; Patton, 1994, 2006, 2011). Rather than working within clearly defined goals and objectives, this methodology accounts for the way processes and outcomes need to evolve as projects progress (Gamble, 2008; Patton, 2006). Developmental evaluation is particularly useful for evaluating and adapting early-stage programs or initiatives that require further exploration and development (Gamble, 2008). Engaging in developmental evaluation can provide useful information about the current state of an initiative, tracking the trajectory of an initiative, gathering timely feedback to inform decision-making about adaptations, and generating ideas about what else might be tried (Gamble, 2008; Guijt et al., 2012). While there are no prescribed methods for data collection or analysis, Table 1 describes common cyclical practices that are part of conducting this type of evaluation (Dozois et al., 2010; Gamble, 2008; Guijt et al., 2012).

The process of engaging in developmental evaluation involves conceptualizing what is getting developed and why, designing the best approach to evaluation based on what is currently known, and generating information from reality-testing to support developmental decisions and adaptation (Patton, 2006). The approach itself is agnostic and draws on common methods of evaluation (Guijt et al., 2012). Teams engaged in developmental evaluation learn to be aware of what is unfolding in the development of a program or initiative (from the bottom-up) and how to respond strategically as the program is being delivered (Patton, 2006). To learn from participants, partners, and the context, developmental evaluation projects might involve data collection such as the use of surveys, interviews, and observations (Gamble, 2008). Typical points of focus for data collection and analysis include three questions: 1) What is getting developed?; 2) What are the implications or anticipated consequences?; and 3) What actions should be taken? (Gamble, 2008; Guijt et al., 2012).

The Case for Developmental Evaluation for the Reflection Room Project

To develop and evaluate the Reflection Room project pilot we first considered conducting a mixed-methods multi-phase study, but realized the project required a methodology that aligned with the dynamic nature of the work. We identified an initial goal to increase peoples’ comfort thinking and talking about dying and death, yet we were open to this evolving with the design and public engagement. The variety of potential settings for installations was also developing (e.g., conferences, libraries, outdoor fundraising events for hospices) so the context could not be specified or held constant. This was a positive development, since it required and enabled the Reflection Room project pilot to spread organically and evolve to address various contexts and needs. Finally, being an interactive installation and a supportive resource, the Reflection Room project involves unfacilitated co-creation by visitors, making it less possible to have concrete processes and outcomes from the outset (Patton, 2011).

We chose developmental evaluation as a methodology because it would support and guide the Reflection Room project using data to drive reconceptualization and adaptations (Gamble, 2008; Patton, 2006). What follows is a description of our process of engaging in developmental evaluation of the Reflection Room project, including the original pilot and two adaptations (see Table 2 for overall schematic of the project). We also discuss the viability and stabilization of the Reflection Room project in its final form. The focus of our description is to elucidate the methodological application and process, including the ways participant engagement evolved (International Association of Public Participation, 2014), with detailed outcomes of evaluations forthcoming and published elsewhere (Carter et al., 2025).

This study was approved by the Southlake Health Research Ethics Board (REB#0077-1718) and the Newfoundland and Labrador Health Research Ethics Board (REB#2016.139). Notices of information about the study and implied consent were posted in the Reflection Room.

Pilot of the Reflection Room Project

The pilot version of the Reflection Room project ran from August 2016 to September 2017. The opportunity to host a Reflection Room was advertised through social media, notices at research and hospice palliative care conferences, and word of mouth. When a potential host organization contacted the research team, they were provided with information about the logistics of installing a Reflection Room. The research team also applied to install the Reflection Room in two settings: an art gallery and a museum. In total, the Reflection Room was installed at 25 sites in healthcare settings (e.g., hospitals, hospices) and community settings (e.g., libraries, museums, universities) across Canada. At this stage, participant engagement occurred through consultation with hosts who provided their ideas about who the potential visitors would be, and where, when and how best to install the Reflection Room. Visitors to the Room were also consulted through surveys.

Conceptualizing

For the Reflection Room project pilot, based on participant recommendations, we wanted to develop an innovative, transportable, arts-based installation to implement in various healthcare and community spaces (formative objective). Reflecting on the reactions to the installation at the conference, we hypothesized that creating this contemplative space for reflection and sharing stories about dying and death would increase comfort thinking and talking about dying and death and maybe motivate people to plan for their own death (outcome objective). In conceptualizing what the installation would entail, we reflected on the response at the hospice palliative care conference. There, visitors said that they benefited from having a special place that invited anonymous stories in an unfacilitated way to allow choice regarding level of engagement. However, we sought to better understand the role of writing and sharing reflections and why these might be supportive as components of the project.

Returning to research literature, we incorporated a model by Schenker et al. (2015) that elucidated the healing mechanisms of storytelling for bereaved family members after a death in a hospital intensive care unit. Their model suggests storytelling engages individuals in cognitive processing (e.g., thinking through events to find closure, order, and a sense of control), disclosing emotions (e.g., sharing emotional trauma), and making social connections (e.g., storytelling is an opportunity for social connection that counteracts loneliness and social isolation through shared experiences), all of which help to reduce mental distress for grieving families (Schenker et al., 2015). While the model was applied to a bereaved population in a specific setting, and was developed as a structured facilitated intervention, we found commonalities between the three processes and the spiritual care principles, which provided further rationale for our conceptualization and potential outcomes. For example, both emphasized the importance of processing and sharing thoughts and feelings, and both framed social connections as important through sharing or witnessing.

Drawing on our conceptual framework that incorporates three models/concepts/theories including spiritual care principles (Holyoke & Stephenson, 2017), the concept of unfacilitated participatory art (Chang, n.d.), and Schenker et al.'s (2015) storytelling model, the Reflection Room project focused on creating a scalable, mobile model amenable to installation in various settings. As each host site installed the project, they were consulted as to how to install the Reflection Room in their context. In this pilot version, the installation of the Reflection Room was completed by the research team using materials that moved from one site to another (see Image 2). An online digital space was also created.

Image 2.The pilot version of the Reflection Room project

Designing

In designing our evaluation, it was important to collect data about use and visitors’ preferences as unobtrusively as possible, balancing the need for data with the principle of the Reflection Room as an anonymous, unfacilitated space. We believed that it could be overly disruptive if visitors’ reactions were actively monitored, and that it would not be feasible for our research team to attend all installations for their entire duration of the project. We decided to start evaluating the Reflection Room project pilot by collecting data to tell us about both the process and outcomes of the developing project. Data included: 1) photographs of the installations to support recall of the context and specific features of each installation, as each installation had to be adapted to fit in a unique space; 2) when possible, observations of the installations recorded in fieldnotes by a member of the research team documenting their impressions of the flow of participants, interactions (if any) between participants, interactions with the installation, and any comments to the research team; 3) the reflection cards posted publicly on “ribbon walls” (see Image 1); and, 4) two short surveys voluntarily completed by participants at the time they visited and again three months later to learn about peoples’ interaction with, and reaction to the installation. At this early stage we did not evaluate the online version as our focus was primarily on understanding what elements of the standard installation materials could be conveniently delivered to and installed in the various sites, and supporting the hosts of the installations.

Reality Testing

After several installations of the Reflection Room pilot, we engaged in reality testing by analyzing the research team’s fieldnotes and photos to determine what was being developed and what actions needed to be taken. Several members of the team collaboratively read over the data to describe what could be learned about what was developed, the implications, and response needed (Sandelowski, 2000). They then discussed iterative findings with the larger team and through discussion came to the following consensus on adaptations needed. These suggestions were shared with current and future host sites:

1. Create a sign with easy-to-understand language to be placed at the entrance to the Reflection Room, “We invite you to pause, reflect and share your experiences with dying and death”. This action item was advised because in the early versions of the pilot in community settings, there was no invitation, and some potential participants did not know what they were being invited to do or why the room existed. Additionally, it was suggested hosts include information sheets provided by our team about the project and place them within the Reflection Room.

2. Install the Reflection Room in quiet indoor areas and not in the centre of busy activities. This action item was advised because we received an invitation to install the Reflection Room outdoors as a complement to a physical activity-based fundraising event for a facility-based hospice. However, there was little engagement with the Reflection Room. Windy conditions compromised the installation features, and we observed that our installation did not create the quiet space for reflection that was a tenet of our pilot.

3. Do not equate the number of reflection cards or surveys with the level of engagement with the Reflection Room and consider additional ways to examine engagement. This action item was advised because at certain sites there were few reflections cards completed and left behind or hung, yet many people shared stories about dying and death with the member of the research team who was observing while replenishing blank reflection cards. These conversations demonstrated that the space was stimulating thinking and talking about dying and death, despite lack of documentation.

4. Future installations should have a longer installation timeline and use an ‘art gallery’ aesthetic to further stimulate engagement. This action item was advised because two installations were installed for several weeks rather than the more usual three to five days in other settings, with the length being dictated by the hosts. Both sites had many visitors to the Reflection Room and many written reflections, and both used an adapted ‘art gallery’ aesthetic.

From analysis of reflection cards (n=463) and surveys completed by participants upon exiting the room (n = 271) and 3 months later (n = 50), outcomes from what was being developed were learned (for a detailed description of the data, analysis, and findings please refer to work by Carter et al. (2025)). Overall, the findings indicated the Reflection Room allowed people to share stories, which included disclosure of emotions and processing thoughts. Additionally, visiting the Reflection Room increased many participants’ comfort in thinking about dying and death, increased the likelihood they would talk to others about dying or death, and increased the likelihood they would plan for their own dying and death. Furthermore, Carter et al. (2025) clarified the way people engaged with the Reflection Room, with about half of survey respondents indicating they visited the Reflection Room but did not leave a reflection. Drawing on these findings, new ideas were generated for reconceptualizing the installation, which led to the development of a subsequent version of the project, the Art Gallery version.

The Art Gallery Version

The Art Gallery version of the Reflection Room project ran from April 2018 to March 2020 with the Reflection Room installed at 25 sites in healthcare settings (e.g., hospitals, hospices) and community settings (e.g., museums, universities) across Canada. At this stage, participant engagement continued to occur through consultation with hosts who provided their ideas about who the potential visitors would be, and where, when and how best to install the Reflection Room. Visitors to the Room were consulted through surveys.

Conceptualizing

As noted, during the pilot version, the Reflection Room had been set up in an art gallery and museum. As suggested by the host, the ‘opening’ of the Reflection Room was to coincide with the opening of another show in the art gallery. Therefore, the aesthetic of the Reflection Room was adapted to reflect the typical aesthetic of an art gallery, including more neutral colours and a few randomly selected reflection cards from previous installations framed and hung on walls. Because we notice increased engagement with this aesthetic, we adapted the Reflection Room into the Art Gallery version. We maintained the same conceptual framework and hypothesis from the pilot as well as most of the design elements (e.g., ribbon wall, Reflection Cards, music, etc.). We adapted the colour scheme to feature more predominantly a neutral white colour; eliminated images of nature except for, where possible, a single bouquet of flowers; included randomly selected framed reflection cards from previous installations; and planned to install the Reflection Room for a minimum of one month (see Image 3). Installation sites continued to be recruited through word of mouth, social media releases, and email lists.

Image 3.The Art Gallery version of the Reflection Room project

Designing

The design of our evaluation stayed constant from the pilot version with one exception of adding informal consultations with Reflection Room hosts to learn about their experience hosting and visitor engagement with the Reflection Room. This decision was based on reality testing from the pilot version where we had learned that our evaluation was not capturing the extent of engagement.

Reality Testing

After several installations of the Art Gallery version, we engaged in reality testing by analyzing research team member fieldnotes and photos, reviewing the Reflection Cards, and informally consulting with Reflection Room hosts. Members of the research team engaged in reading and descriptively summarizing the data (Sandelowski, 2000) and then presented findings to the larger team who engaged in consensus discussions to come to agreement about what the various sources of visitor reactions could mean in terms of adapting and changing what we had developed to date. Below is a summary of recommended adaptations and rationale from the data:

1. Conceptualize a ‘travelling kit’ of the Art Gallery version. This action item was advised because hosts in remote or faraway places (e.g., sites in the Northwest Territories) requested an installation, which our team lacked the resources to be able to install. Through discussions with these hosts, it was decided to try creating a kit that incorporated shippable components, a packing list, detailed and clear instructions, easy to use materials (e.g., the hardware used to hang the ribbon wall was cumbersome) and return shipping once the installation was complete.

2. Adapt our data collection methods to scan the reflection cards, leaving the originals with the site. This adaptation was requested by a host, a newly constructed facility-based hospice, who wanted the Reflection Room as a permanent installation. Their community felt ownership over the reflection cards, with people revisiting the room and re-reading specific cards.

3. Continue to recommend sites to install the Reflection Room for a minimum of one month. This action item was advised because hosts observed that when installations are available for longer periods of time, there is opportunity for multiple visits, writing a reflection when ready, and re-reading others’ reflections. Hosts felt that the longer an installation was up, the more opportunities for flexible unstructured engagement there would be.

To evaluate outcomes, our plan was to focus on comparing differences between the pilot version and the Art Gallery version of the Reflection Room in terms of content of reflection cards (n=479) and comfort with dying and death as examined in the surveys completed by visitors upon exiting the room (n =113) and 3 months later (n =18). Analysis of results was interrupted and put on hold when the COVID-19 public health emergency was declared, but the project restarted as the Reflection Room project was pivoted to address growing needs related to the pandemic.

In the Time of COVID-19 Version

In the Time of COVID-19 version of the Reflection Room project ran from June 2021 to March 2023 with the Reflection Room installed in 32 long-term care homes in Ontario, Canada. At this stage, participation increased through collaboration and empowerment of hosts and their communities as they took ownership of how to install, use, and organize the Reflection Room installation. Visitors to the room were consulted through surveys.

Conceptualizing

Our research team was approached by the Ontario Health Central Region Long-Term Care Homes’ Grief and Bereavement Working Group to discuss using the Reflection Room project as a response to grief and loss in long-term care (LTC) homes in Ontario, Canada. People in LTC homes in Ontario and around the world experienced tremendous loss and distress due to the COVID-19 pandemic with high levels of illness and death (Stall et al., 2021), physical isolation and disruption to meaningful activities (Chu et al., 2022), and many staff experiencing burnout and fear of spreading COVID-19 to residents, colleagues, and their families (Yau et al., 2021). A subcommittee consisting of representatives of people living in LTC homes, their families, the regional branch of Ontario Health, education proponents, and our research team came together to collaborate and co-conceptualize ways of adapting the Reflection Room project to support pandemic-related grief and loss.

Based on the findings reported in Carter et al. (2025), the pilot version of the Reflection Room project increased participants’ comfort talking about dying and death and was an outlet for people to share strong emotions including grief. Additionally, the ‘travelling kit’ from the Art Gallery version showed that it would be feasible to send kits to LTC homes, which were closed to outsiders during the pandemic. The subcommittee decided to experiment with pivoting the focus of the Reflection Room project from dying and death exclusively to inviting reflections on experiences of loss during the pandemic. This broadening of the goals and outcomes aligns with Schenker et al.'s (2015) storytelling model for a bereaved population as well as other research that agrees that if people are given the opportunity to name their experiences with grief and loss, they are often better able to process and cope with their grief (Ratcliffe & Byrne, 2022).

The subcommittee reconceptualized this version of the Reflection Room project with hosts and communities making the following recommendations: 1) installations needed to be quick and easy to set up by LTC home hosts with minimal components and no assistance from the research team; 2) the Reflection Room needed to be adaptable to meet the needs of each community; 3) the project could not be a burden to communities already experiencing exhaustion; and 4) due to limited space, circumstances regarding pandemic-related protections, funding limitations (the available funding was $125 CAD per installation), and the urgency of the moment, the kit needed to be smaller and more versatile than what was previously conceptualized.

Together with the sub-committee, our research team drew on the learnings from the pilot and the Art Gallery versions and collaboratively decided the contents of the Reflection Room kits that could be sent to LTC homes for installation. The sub-committee also recommended essential components for the kits (see Table 3) that would maintain fidelity to the project design and be feasible in cost as well as easy to install (for example, substituting a lightweight, flexible red curtain for the ribbon wall) (see Image 4). A project manager was hired to offer support to LTC home hosts via email and phone calls to offer suggestions as they decided how they wanted to install the Reflection Room in their space for their community. LTC homes were asked to keep the Reflection Room up for a minimum of one month and they were able to keep the Reflection Room kits indefinitely.

Image 4.In the Time of COVID-19 version of the Reflection Room project

Invitations to host the Reflection Room were issued through members of the Long-Term Care Homes’ Grief and Bereavement Working Group using social media releases, and emails to Family Council and Residents’ Council members and administrators of LTC homes in the region.

Designing

With the agreement of the sub-committee our evaluation design included collecting data to understand how the process worked by collecting, 1) an online survey for Reflection Room hosts who installed (or organized the installation of) the Reflection Room; and 2) semi-structured interviews with these hosts where they could share their perceptions of the impact of the Reflection Room, and recommendations of how the Reflection Room might be scaled and spread. The host survey included questions aimed at learning more about what components of the installation support people’s grief. In previous versions of our evaluation this was not asked, so while we were able to identity that visiting the Reflection Room increased comfort talking and thinking about death, we did not know why people believed that to be the case. To evaluate the outcomes of the installations we collected three additional pieces of data, 1) characteristics of the LTC homes (e.g., size, the total number of people infected by Covid-19); 2) copies of reflection cards from each LTC home; and 3) paper-based surveys voluntarily completed by participant visitors to the Reflection Room asking about their experience visiting the Reflection Room.

Reality Testing

We engaged in reality testing drawing on data from the host surveys (n=23) and interviews (n=16), which were collected on a rolling basis as LTC homes installed a Reflection Room. One member of the team engaged in descriptive analysis of survey data with fixed response options and qualitative content analysis (Elo & Kyngäs, 2008) of open-ended survey responses and interview data. The team then engaged in consensus discussions of preliminary results to summarize hosts’ recommendations, which included:

1. Continue with the conceptualization of the ‘essential component kits’. This action item was advised by hosts who explained that the kits were adaptable to being set up in many different locations within LTC homes (e.g., chapel, sunroom, recreation room), easy to set up (e.g., 1-2 hours to set up) and required few resources to maintain (e.g., 1-2 hours a week to check that the Room had the needed supplies).

2. To increase engagement, installations can be conceptualized and adapted for the space with people from the community. This action item was advised by hosts who indicated that engagement with installing the Reflection Room differed across LTC homes, and that more community participation in installing the Reflection Room appeared to strengthen use, connection, and meaningfulness to visitors.

3. Installations can be integrated with existing services. This recommendation was advised by many hosts who indicated that the purpose of the Reflection Room aligned with and enhanced existing programs such as spiritual care counseling, memorials honouring people who had died, and support for bereaved families. They observed that integrating the Reflection Room with existing services was perceived as helpful to the community because it offered an additional resource for grief and loss.

4. Invite communities to choose a meaningful topic, perhaps not only focused on loss, dying, death and grief, for reflection. This action item was advised because as the pandemic went on and appeared to attenuate, some hosts indicated that the focus of this version of the Reflection Room needed to be pivoted to residents’ experience beyond the pandemic.

5. Support accessibility and diverse engagement with the Reflection Room. This action item was advised by hosts who indicated that it was important that future installations prioritize accessibility in terms of location as well as supporting participants with diverse cognitive and physical abilities. This adaptation might include inviting participants to draw on the reflection cards, read the cards aloud, or follow the example of one LTC home that created a ‘Reflection Room Cart’ to take the Reflection Room components to people with reduced mobility.

To support analysis of the COVID-19 reflection cards (n=169) about pandemic-related grief and loss, we drew on the Dual Process Model – Additional Effects of Overload (Stroebe & Schut, 2016). This model explains that grieving involves oscillating between attending to loss-oriented challenges and restoration-oriented challenges. However, the experience of overload (e.g., feeling like you have more than you can manage such as experiencing a global pandemic) can lead to chronic grief or absent grief (Stroebe & Schut, 2016). This theory was not added to our conceptual framework as it was used in analysis, not the conceptualization of the Reflection Room.

The Reflection Cards expressed people’s experiences of overload, loss-oriented stressors, restoration-oriented stressors, learning through loss, and gratitude despite loss. Analysis of survey responses from staff, residents, family caregivers, and others (n=75) and LTC home hosts (n=23) as well as data from interviews with LTC home hosts (n=16) was used to identify several features of the Reflection Room believed to support grieving: 1) offering a quiet restful place; 2) allowing people to reflect and process their emotions and experiences; and, 3) providing an outlet for thoughts and emotions. Staff, residents, caregivers, and LTC home hosts shared they found these elements effective because visiting the Reflection Room 1) provided additional grief support to what currently exists, 2) provided a sense of well-being, and 3) supported community building and connection through sharing stories.

Key to the Reflection Room project entering a stabilization phase, these findings provided rationale about the essential features of the Reflection Room to be sustained and spread across settings. The three essential features of the Reflection Room identified in this analysis, 1) offering a quiet restful place; 2) allowing people to reflect and process their emotions and experiences; and, 3) providing an outlet for thoughts and emotions are rooted in our conceptual framework, involving spiritual care principles (Holyoke & Stephenson, 2017), the concept of unfacilitated participatory art (Chang, n.d.), and Schenker et al.'s (2015) storytelling model, which was consistently applied across conceptualizations of the project.

The ‘Essential Component Kits’: Stabilizing and Spread

The developmental evaluation process allowed us, with increasing levels of engagement with representatives of installing sites, to iterate, learn, test, and adapt over enough years to be able to reach a place of sustainability of the Reflection Room project. The ‘essential component kit’ developed to adapt to COVID-19 appeared to be the most viable version for spread and scale. In collaboration, our research team transitioned the project and its evolution to the Saint Elizabeth Foundation, a recognized leader advocating for and providing access to palliative care particularly for people experiencing structural vulnerability (Saint Elizabeth Foundation, 2025), as a continuing initiative to increase access to dying, death, and grief support. The designed kits were scaled and spread based on the long-term findings of the project being feasible, adaptable to different settings and contexts, and supportive of those experiencing dying, death, and grief. The first initiative of the Saint Elizabeth Foundation was to spread Reflection Room kits to 230 hospices across Canada to be installed where, when, and how the hospices (and residents and families) wanted the Reflection Room. In addition, twenty other organizations (e.g., community-based organizations, a university, a LTC home, including one in the USA) received kits, and a permanent National Reflection Room was installed at The National Cemetery of Canada.

Discussion

When it is not possible to have clearly defined intervention processes, objectives, and outcomes at the start of pilot projects, developmental evaluation offers a methodology that supports evaluation and creative innovation driven by data as projects progress (Gamble, 2008; Guijt et al., 2012; Patton, 2006, 2011). With this evaluation approach, our team was able to gather data and, based on participant engagement, iteratively conceptualize, evaluate, and adapt our Reflection Room project in a cyclical process that continued for seven years. Below we discuss learnings and recommendations from each developmental evaluation activity to support others wanting to engage in similar methodological work.

Conceptualizing

At the start of the Reflection Room project, conceptualization of the installation was theory-informed based on organizational spiritual care principles (Holyoke & Stephenson, 2017), unfacilitated participatory art (Chang, n.d.), and storytelling during bereavement (Schenker et al., 2015). Conceptualization also had to be flexible as shaped by conditions in each setting such as size of room and length of time allowed at each site. The two objectives we thought would be of importance in this version were to 1) create a scalable, mobile installation amenable to various setting (formative objective), and 2) increase participants’ comfort thinking and talking about dying and death (outcome objective).

As the project progressed, we learned from reality testing that a more neutral aesthetic and increased length of installation time supported more options for engagement. Additionally, we learned that the Reflection Room could be packaged into a travelling kit in a way that was acceptable and feasible for hosts and participating communities. In the last version of the Reflection Room project, conceptualization pivoted to address an emergent need and was co-conceptualized with community members. This meant that the outcome objective changed to supporting people experiencing grief, and the formative outcome became even more important. From the outcome evaluation we know the conceptualization of the project did improve comfort with thinking and talking about dying and death and did support people who were grieving (Carter et al., 2025).

As conceptualization was adapted (e.g., the aesthetic, length of installations, and topic of reflections), the theoretically informed process of pausing, reflecting, and writing or reading reflection cards remained constant, giving us confidence in the continuity of key features of the project, which made it successful across contexts. We recognize that as projects adapt, some may find the underpinning theories, models, and/or concepts need to evolve. Tracking these changes and linking them to data helps to develop context specific knowledge based on real world investigations. Grounding developmental evaluation with a conceptual framework can allow researchers to test out theories in real world contexts and recommend useful adaptations when needed. As such, we recommend others wanting to employ developmental evaluation use a conceptual framework to provide evidence-informed grounding and continuity for the conceptualization as the project evolves.

Conceptualization in our project became increasingly shaped by communities who co-developed adaptations, which makes projects better able to respond to the needs and suggestions of community members (Boaz et al., 2018; Patton, 2006; Svensson et al., 2018). In the pilot version participant engagement was embedded in the research-informed theories and from the attendees at the hospice palliative care conference. In the Art Gallery version, participant engagement was in the form of consultations with hosts at each installation site who directed the length and aesthetic of the project. The most viable version of the Reflection Room project was conceptualized and implemented by people embedded in LTC home communities, including residents and families at some sites, who acted as “decision-makers” in the conceptualization of this version of the Reflection Room (Smits et al., 2020).

Co-creation is one of the essential elements of developmental evaluation, however, it often pertains to innovators and evaluators, not necessarily focused on end-users (Goertzen et al., 2020). The participation of community members (e.g., patients, caregivers, clinicians, hosts) throughout developmental evaluations can be tailored to the goals of the project, participant preferences, resources, and the context (Giosa et al., 2025). Co-creation might involve end-users as listeners, co-thinkers, advisors, partners, or decision-makers (Smits et al., 2020). We recommend fostering engagement with people in research that concerns them from the beginning of conceptualization as this produces higher quality knowledge and outputs (Boaz et al., 2018; Smits et al., 2020). However, as with the Reflection Room project, the level of engagement of end-users can and perhaps should evolve over time. In our case, the level of participant engagement moved along the International Association of Public Participation’s Spectrum of Public Participation (International Association of Public Participation, 2014) over time, from “inform” in the initial conference setting, through “consult,” “involve” and “collaborate” with many installations, and finally to “empower” at the time of the LTC and hospice installations when the decisions about the Reflection Room were entirely in the hands of the users, and the function of the research team was primarily organizing the delivery of installation kits.

Designing

In the pilot version, we designed our evaluation in such a way as to be aligned with our conceptual framework, specifically being an unfacilitated installation with no prescribed way of interacting. This meant data collection would be voluntary and as unobtrusive as possible. In the first version, the focus of our evaluation was on variations in engagement with the installation, the content of the reflection cards, comfort thinking and/or talking about dying and death, and planning for one’s own death. This meant some factors would remain unknown, such as reasons why participants experienced changes and how outcomes were influenced by demographic characteristics. In the Art Gallery version, the design of our evaluation stayed constant with one exception of adding consultations with Reflection Room hosts due to learning that our evaluation was not capturing the extent of engagement with the installation. By the time we designed the evaluation for the COVID-19 version of the project, we had already evaluated the pilot version and there were additional questions the subcommittee wanted to be able to answer. Much of the evaluation methods remained the same (e.g., analyzing content of reflection cards, using anonymous surveys), however, to learn more about sustainability, we interviewed the hosts to understand what they learned and recommended about feasibility and impact. Survey questions for visitors included questions about what components of the installation support people’s grief.

Our project was evaluated over a seven-year period, which gave us the ability to continue evaluating adaptations of the project, culminating in a viable version of the Reflection Room. Other developmental evaluation projects appear to occur over a year or less, with little discussion of stabilization, scale, or spread (Goertzen et al., 2020; Wu et al., 2018). Through our evaluation, we came to realize that our first version of the Reflection Room did not use the type of resources available in the context as suggested in WHO’s guidelines (World Health Organization, 2011). It was only in our COVID-19 adaptation of the Reflection Room project this occurred. We recommend others evaluate the feasibility for stabilization, scale, or spread of their pilot projects in the design of their evaluation methods (Goertzen et al., 2020; Wu et al., 2018).

For others designing evaluation methods, we recommend prioritizing congruence between the research question(s), conceptual framework, and overall objectives as the driving rationale for design of the evaluation. We also recommend paying attention to the questions of the communities with whom research is being carried out and adapting the design as needed. This may require changing the research questions, data collection methods, or even the objective as we did in the last version of the project. One challenge with this more iterative and increasingly more participatory approach is the requirement for multiple Research Ethics Board reviews as the evaluation data collection plan evolves. This can slow the evaluation but will provide more meaningful findings that will benefit the communities engaged in the project.

Reality testing and adapting

Reality testing and adapting what is being produced is a key feature of conducting developmental evaluations. In our pilot version, reality testing based on observations and host suggestions provided important information regarding how to enhance visitor experience and engagement with the Reflection Room through providing clear messaging, lengthening the installation time, installing in contexts that are conducive to reflection, and changing the aesthetic to the Art Gallery version with example framed cards. These learnings were possible due to variations between installations as dictated by the installation sites themselves. The benefit of developmental evaluation is the ability to be less rigid with perspectives of conceptualization so that it is more possible to innovate based on what was getting developed organically (Patton et al., 2015). We recommend others also allow room for organic variations in participants, settings, and adaptations when possible.

Learning from our participatory, community-led innovations continued through the subsequent versions of the Reflection Room project. During the Art Gallery version, we adapted the installation to create travelling kits and learned about the feasibility of doing this. We also evaluated engagement of the 30-day installation period and learned how this created more opportunity for flexible unstructured engagement. This 30-day installation period became a recommendation for stabilizing and spreading the Reflection Room project. In the COVID-19 version, we adapted the travelling kits to contain essential components based on community members’ recommendations and learned about the increased feasibility and acceptability of this design.

A key benefit of conducting our developmental evaluation was that methodologically the research process allowed our team to prioritize the needs of visitors, hosts, and communities, and respond quickly in a global crisis (Fagen et al., 2011). Guided by LTC community representatives, the Reflection Room was quickly adapted to meet unanticipated needs from the COVID-19 pandemic. As a methodology, we found this approach of cyclical learning, adapting, and developing strongly aligned with the core principles of ethical research – respect for persons, concern for welfare, and justice (Canadian Institutes for Health Research et al., 2022). Other health researchers have also found value in conducting developmental evaluation due to the ability to quickly pivot to better support participants’ welfare as needs emerge (Harper & Dickson, 2019; Lys et al., 2018).

Limitations

Due to decisions we made about evaluation (e.g., the commitment not to actively engage each Reflection Room visitor but rather to let them decide their level of participation without intrusion), we are unable to comment on demographic characteristics of participants and therefore unable to conclude if a diversity of perspectives guided this evaluation. Due to the diversity of locations in which we installed a Reflection Room, we do know there was some variation in geography across Canada. Furthermore, data analysis from the Art Gallery version was stalled due to the pandemic, which means we missed drawing on these findings to inform the final version of the project. Lastly, due to a focus on methodological learnings from operationalizing the developmental evaluation, the details of data collection and analysis are reported elsewhere (Carter et al., 2025) and forthcoming.

Conclusion

Using developmental evaluation allowed our team to combine the rigour of evaluation with the creativity of design work to adapt our Reflection Room project based on emerging visitor, host, and community needs, leading to the creation of the most viable version of the Reflection Room project. Rather than prioritizing standardization of process and methods, this methodology promotes flexibility in a way that emphasizes core ethical research principles making it more possible to promote participants’ welfare and equity during the development and evaluation of innovative pilot projects. For those who wish to conduct developmental evaluation, we recommend engaging end-users early on and throughout the evaluation and facilitating increasing control over decision-making by users. From a research perspective, we recommend grounding the evaluation with a conceptual framework; evaluating the stabilization, scale, or spread of projects; and adapting the evaluation and design to support organic variations in participants, settings, and emergent questions.