Applying a Virtual World Café Method for Participatory Design of AI Systems

1. Introduction

Artificial intelligence (AI) broadly refers to smart systems that can predict emerging patterns or categories or both from real-time data (e.g. of images, video clips, sounds, or text) by being trained on datasets featuring similar examples (Samoili et al., 2021). AI is becoming increasingly omnipresent in every life and shows great potential in improving our quality of life. For example, AI enabled smart home technology can alert us to abnormal activity in the home such as break-ins (Guo et al., 2019) and AI powered smart wearables can support personal health monitoring (Zhu et al., 2022) as well as provide remote diagnostic tools for healthcare practitioners (Abdollahi et al., 2020). However, AI can also have adverse impacts on society. AI systems trained on biased data can result in machine biases^[1] where people with certain demographics (i.e. gender, race) can be discriminated against when the AI systems are used in significant societal decision making, including hiring and prison sentencing, (Buolamwini & Gebru, 2017) and can spread misinformation (Shin, 2024). Cases of machine biases have highlighted the need to actively engage end-users in AI design to help anticipate and mitigate against them. There is also a growing consensus that allowing stakeholders such as end-users to shape the design of AI systems can lead to AI systems that better encompass user needs, values and preferences (Delgado et al., 2023).

Participatory design (PD) methods show promise in supporting the participation of end-users and other stakeholders in designing AI systems. PD methods involve engaging those affected by design outcomes, such as end-users, in making design decisions to develop the design outcomes (Simonsen & Robertson, 2013). In AI design, PD methods can allow end-users to collaborate with researchers to make decisions about how AI systems should be designed (Zytko et al., 2022) such as the types of training data used to train AI systems. Moreover, engaging end-users from the beginning of AI design can help ensure that its scope and purpose are aligned with users’ needs from the beginning and that suitable training data is employed from the start to remove possible machine biases.

However, practical constraints, such as lack of resources and time, can limit end-user participation in AI design, to only providing feedback on the user interface for example (Delgado et al., 2023). Additionally, official regulatory frameworks for AI, such as in the UK (Ada Lovelace Institute, 2023), EU (Council of the EU, 2024) and USA (The White House, 2022) mandate risk management procedures for AI rather than PD methods for understanding how and if an AI system should be used in a certain context.

In this paper, we propose a promising PD method for AI design to efficiently engage end-users at the start of development to define its scope and purpose from the beginning. Specifically, we discuss how a virtual world cafe method, based on the World Café method, was developed and modified as a PD method to identify design requirements for sounding sensing AI in the home.

To begin the paper, challenges of applying PD methods in AI design are discussed to demonstrate a need for guidance in this. The World Café is then presented as a suitable PD method for defining design requirements for AI design and sound sensing AI for the home is introduced as an example use-case. Subsequently, a description of the research methodology explains how a revised version of the virtual world café, based on the World Café method, was created as a PD method through review of previous virtual world café formats and used to explore sound sensing AI for the home with UK-based participants. A summary of how the method performed as a PD method for initiating AI design is presented, such as how the method informed the creation of design requirements for sound sensing AI for the home. The paper ends with a concluding discussion on the suitability of the virtual world café as a PD method for initiating AI design, as well as limitations of the method and how these limitations might guide future PD methods in designing AI more generally.

1.1. Challenges of applying PD methods in AI design

PD methods have been embraced by the human computer interaction (HCI) community to develop new technologies for several years (Muller & Druin, 2012). Whereas applying PD methods to AI design is relatively new as this can be challenging. Design decisions and their outcomes can be difficult to anticipate in AI design because AI systems can behave differently from the system initially being used over time as it learns from additional inputted data (Bratteteig & Verne, 2018).

However, if end-user participation in AI design occurs at the beginning, such as to direct model selection and original training data used, outputs are more likely to align with user needs and be unbiased even as they change over time. For example, the data used to train an AI model determines how it makes its calculations to create its outputs. An AI model originally trained on data defined by user needs therefore is less likely to be biased and produce results that will conflict with those user needs, even as it learns from additional inputted data.

Notably, applying PD methods can be time and resource intensive as they can require understanding and aligning the timescales and viewpoints of a variety of stakeholders such as end-users in the right context for creating collective objectives and solutions (Gaudio et al., 2017). Relatedly, organisational timelines and limited resources have been described by AI researchers and practitioners as restricting the depth and diversity of end-user participation in AI design, such as in Delgado et al.'s (2023) empirical study exploring the current state of “participatory AI”.

Furthermore, within the design research community many authors discuss the use of PD methods in AI design without describing specific PD methods in detail for defining AI design requirements (e.g., Auernhammer, 2020; Loi et al., 2018; Weller, 2019). More specifically, Weller (2019) asserts the importance of applying user research to mitigate against machine bias, Loi et al. (2018) discusses PD as an approach for AI design, Auernhammer (2020) summarises PD as an AI design approach for prioritising human needs, Van der Maden et al. (2023) present a framework for designing AI systems for community wellbeing and Sadek et al. (2023) presents recommendations for improving the use of PD methods when designing conversational agents.

Notably, Hossain and Ahmad(2021) present a model for an “agile participatory design” approach to AI design to overcome some of the challenges of applying PD methods in AI such as supporting diverse and continuous stakeholder engagement. The model emphasises defining objectives collaboratively with stakeholders, such as end-users, from the beginning of development—such as to subsequently use these to guide development and deployment to align the final AI design with stakeholder expectations—but does not suggest a specific PD method for achieving this.

Given the time and resource constraints of previous AI projects that have reportedly limited the depth and diversity of end-user participation in AI design (Delgado et al., 2023), a PD method that converges perspectives and ideas of many diverse stakeholders simultaneously into actionable objectives in a short timescale is desirable. The next section introduces the World Café as a suitable PD method for initiating AI design.

1.2. World café as a participatory design method

The World Café was originally formulated to help facilitate change in different organisations, such as large multinational corporations and government offices, to converge perspectives from various stakeholders into actionable objectives and solutions to complex problems (Brown & Isaacs, 2005). The main aim of the World Café is to bring together different people’s perspectives through quality conversations to generate new insights and inspire collective action from this (Schieffer et al., 2004).

The World Café is typically conducted in a café style environment—such as provision of tea and coffee refreshments and use of small tables covered with linen cloths with no more than five table hosts per table—where participants are divided into small groups, of up to five participants, to discuss the same question in three 20-minute rounds (Brown, 2002). At the end of each question round, all participants apart from a participant designated as the “table host” switches tables and the table host remains to summarise previous discussions and gently encourage other participants to take note of key ideas (Schieffer et al., 2004, p. 9).

No more than five participants per group is recommended to ensure enough time for personal interactions and no fewer than four participants to provide a broad enough range of perspectives (The World Café Community Foundation, 2015). An alternative question is usually posed if including another subsequent round of three 20-minute discussions. A “Harvest Session” follows these conversation rounds where everyone is invited to contribute their conclusions from discussions overall and formulate unified action plans (Brown, 2002). The World Café is guided by several core principles that can be summarised as follows (Brown & Isaacs, 2008):

1. setting the context (make the purpose of the meeting clear such as to learn from each other and present actionable findings)

2. create hospitable space (facilitate a café style environment and support free discussion)

3. exploring questions that matter (present intriguing questions that progressively deepen reflection over the course of several question rounds)

4. encourage everyone’s contribution (only one person talks at a time and everyone else fully concentrates on what is being said without interrupting)

5. connect diverse perspectives (as participants move between tables, they exchange personal reflections and draw links between varying viewpoints to create new ideas)

6. listen together and notice patterns (listening for novel concepts or deeper questions that emerge from sharing different viewpoints)

7. share collective discoveries (during the “Harvest Session” anyone in the room begins to share their insights, while others note what ideas from their own conversations link to this initial sharing)

The World Café supports a context for rapid cross-pollination of ideas from multiple viewpoints through the movement in the discussion rounds and sharing of insights in the “Harvest Session” (Brown & Isaacs, 2008). Open and equal discussion from a range of stakeholders is encouraged through the uninterrupted responses participants are encouraged to provide during discussion rounds while others listen. The café style environment also supports a relaxing context to encourage open conversation and free flow of ideas from all participants.

The World Café has been verified as a resource efficient method for inviting the participation of many different people to create, verify and refine ideas simultaneously (Löhr et al., 2020) as well as generating a large amount of conversational data illustrating participants’ viewpoints within a short timescale (Schiele et al., 2022). Furthermore, the World Café has been successfully applied as a PD method where employment of its principles, such as “encouraging everyone’s contribution” and “sharing collective discoveries”, were cited as allowing a diverse group of stakeholders, including local community members, to shape the design of a new community space (Thompson, 2015).

Considering the previously discussed challenges of applying PD to AI design, such as allowing for the time, resources, and stakeholder management required for PD methods within the constraints of AI projects, the World Café’s aptitude for inviting feedback from variety of perspectives within a fast timeframe suggest that it might be a suitable PD method for AI design. Furthermore, by encouraging stakeholders to identify agreed-upon actions in the Harvest Session, the World Café shows potential in supporting stakeholder investment and participation in AI projects from the beginning, supporting Hossain and Ahmad’s (2021) agile approach to PD in AI design.

1.3. Sound sensing AI for the home

Sound sensing AI refers to machine learning AI technology that can automatically sense and tag a range of everyday sounds (Virtanen et al., 2018). Sound sensing AI has been previously employed in the home, such as to support security by recognising unusual sounds such as glass breaking (Harini et al., 2024), and healthcare by monitoring activity (Do et al., 2022). However, sound sensing AI could also be used to categorise and tag current sounds with how they are subjectively perceived by people in different contexts as previous research has documented the effects of specific sounds on psychological wellbeing. For example, natural sounds can relieve anxiety and aid concentration (Ge et al., 2023), and mechanical noise can increase depression and stress (Zhou et al., 2020).. This information could help inform improvements to the sound environment for psychological wellbeing, such as through new legislation or development of additional technologies for achieving this.

However, at the time of this study (2021) there existed no research exploring sound sensing AI to improve psychological wellbeing in homes. Sound sensing AI for the home therefore presented a novel test-case for the World Café as a PD method for AI design. Moreover, exploring this topic required a method that would allow us to understand:

1. what sounds people routinely listened to in their homes to identify training data needs for future sound sensing AI

2. how these sounds affected their psychological wellbeing to determine the actions taken by technology solutions using sound sensing AI

3. what technology solutions using sound sensing AI would be acceptable in the home to determine design requirements

Given the ability of the World Café to gather the views and ideas of multiple individuals simultaneously using research-based questions in a short timeframe (Löhr et al., 2020), it presented a suitable PD method to answer points 1–3. For conciseness, we will now refer to psychological wellbeing in homes as home wellbeing for the remainder of the paper.

2. Methodology

We employed the World Café method as a PD method to identify design requirements for sound sensing AI for home wellbeing with UK-based participants. Because of national UK lockdown restrictions during the time of the study (Spring, 2021) where movement was restricted to the home, the World Café method was adapted into a virtual world café method to enable the entire study to be conducted remotely. We now present a review of several procedures and critical reflections of researchers’ who applied the World Café method virtually that enabled the identification of design requirements for running a virtual world café method successfully.

2.1. Review of previous virtual world café methods

Gilson’s (2015) virtual world café follows the face-to-face World Café closely. Participants are divided into small groups using virtual breakout rooms of a teleconferencing tool instead of tables to discuss a question posed for each question round. One participant per group acts as a table host who keeps track of key points and encourage contributions from other participants (Gilson, 2015, p. 239). At the end of a question round, participants are moved to a new breakout room apart from the table host who remains and summarises key points from the previous conversation to the new arriving participants. However, participants are invited to “to be as involved as they would like” and table hosts were found to be the most talkative participants in the subsequent analysis of resulting dialogues (Gilson, 2015, p. 161). Resultantly, Gilson (2015, p. 178) advises future facilitators to “take the time to train them [table hosts] for their roles” to ensure table hosts understand their role is to encourage everyone’s contribution, not just provide their own.

McKimm et al.'s (2020) virtual world café begins with a welcome from a host that includes advice on online etiquette and confidentiality assurance to support contributions in later discussions. Virtual breakout rooms are then used to facilitate small discussion groups. Participants remain in the same virtual breakout room for one 30-minute session with two experienced moderators guiding conversations before returning to the main room to report on insights. Accordingly, McKimm et al. (2020, p. 389) reflect that the use of small group discussions, and a positive tone set by the host at the beginning of session can support “psychological safety”, and in turn collective learning and collegiality amongst participants in subsequent conservations.

In Ozorio et al.'s (2020) virtual world café format, an average of four participants join one of four breakout rooms that each have a different but related question to discuss for 10 minutes before participants switch to another room. Each breakout room has a participant acting as a “host”, nominated by the group, who remains in the room to note highlights from conversations and report these back in the main room during the “harvest” session (Ozório et al., 2020, p. 239). Ozorio et al.'s (2020, p. 240) review of participant feedback states that most participants found the breakout room discussions to support input from everyone but some participants desired training for participation, such as knowing when to speak using the microphone and how to use the chosen application (Ozório et al., 2020, p. 241).

Kinney and Kinney (2024)’s virtual world café uses virtual breakouts rooms to host small group discussion rounds for two question rounds with a harvest session at the end of each. Two undergraduate students are trained to act as room hosts per breakout room: one facilitates one participant speaking at a time, and the other monitors the chat feature for additional comments while both take notes of discussions. A designated technology host is also used to set up breakout rooms, handle any technical issues for hosts and participants, and move participants between breakout rooms between question rounds. Kinney and Kinney (2024) “highly recommend identifying a technology host” as this allows those facilitating the café to focus on the process. Moreover, they agree that providing specific guidelines and training to rooms hosts in advance of the session is beneficial in supporting the smooth running of the small discussion rounds.

2.2. Our virtual world café method

Our review of previous virtual world café formats suggested the following design requirements to run a successful virtual world café:

1. Use virtual breakout rooms (in place of tables for small group discussions)

2. Designate at least one table host per break room (to moderate discussions, such as only one participant speaks at a time, and summarise insights during Harvest Session)

3. Nominate a host (to set a positive tone, lay the ground rules, and manage the main room)

4. Designate a technology host (for setting up breakout rooms, moving participants in and out of these, managing any technical issues)

5. Provide training and guidelines to all participants in advance of session (such as for the application being used and expectations for participation)

2.2.1. Piloting

We used the identified design requirements from our virtual world café review with some additional modifications to inform an initial 1.5-hour virtual world café. For example, we decided to place a time limit on how long each participant could speak in the breakout rooms to make it easier for the table hosts to moderate discussions and support equal contributions. Table hosts were also instructed to introduce themselves and their response to the question first at the beginning of the breakout room discussion round to provide an example to other participants of what was expected. Additionally, table hosts were instructed to select speakers by viewing the order of participants’ names as they appeared listed in the application panel view from top to bottom to randomise selection.

We also created an additional scribe role for each breakroom that was filled by a project representation to note highlights of discussions as we wanted the table hosts to focus on moderating the discussion rounds. These notes were captured live on a digital whiteboard that was visible to participants in each breakout room and participants were encouraged to comment if they felt any points were missed. At the end of each breakout room discussion, scribes saved and sent a copy of their notes as an image file to the table hosts to refer to when reporting on their breakout room discussion highlights at the start of the Harvest Session. We did not encourage participants to use the chat feature during the breakout room discussions as we felt this would distract them from listening to other participants responses. We also included a short feedback questionnaire (see Appendix A) at the end of the session to allow participants to comment on what went well and submit suggestions for improvement. Our initial virtual world café format is summarised as follows:

Preparation:

1. Roles such as for the host, technology host, scribes and table hosts allocated and provided with specific role guidelines before the session

2. All participants provided with written guidance and optional training for the application being used

Procedure:

3. 10-minute introduction explaining the research project background, world café etiquette and the question to be discussed to all participants in the main room (by the host)

4. Four to five participants divided into breakout rooms (by the technology host), with one participant acting as a table host and project member acting as a scribe per breakout room

5. First 20-minute breakout room session with table hosts inviting participants to introduce themselves and respond to the question posed uninterrupted for up to two minutes each

6. All participants apart from table hosts move to a different breakout room (by the technology host) (allowing 5 minutes for switching)

7. Second 20-minute breakout room session (same procedure as point 4)

8. 30-minute Harvest Session starting with table hosts summarising main conversation insights from each breakout room using notes created by scribes, followed by an open discussion by all

9. Remaining 5 minutes is used to fill in a feedback questionnaire

We ran a pilot of this initial virtual world café format on Zoom (version 5.6.1) with 20 academics from the host university to explore the question: “How can sound sensing AI be used to benefit society?”. We used Zoom to host the virtual world cafe as participants without Zoom installed could easily join the call by following a hyperlinked meeting invite.

Ten of 20 pilot participants completed the feedback questionnaire and included positive comments that can be summarised as the virtual world café was an appropriate method for meeting new people, supporting “[a] diverse range of perspectives on a topic” and “exploration [of ideas] and listening”. Seven suggestions for improvement were also received, which we used to inform changes when finalising our virtual world café format. Example extracts of pilot participant comments with recommendations for improvements are summarised with the corresponding changes made in Table 1.

Overall, pilot participants’ feedback on improvements, such as four of seven comments received, suggested that more time and additional progressive questions were needed to increase the depth of small group discussions. As a result, we increased the breakout room sessions to three with each focusing on different question that built on the previous. Notably “When people feel comfortable… they do their most creative thinking, speaking and listening” (The World Café Community Foundation, 2015, p. 2). To help create a hospitable space in a virtual space, we decided to put participants in the same breakout room for all breakout room sessions to allow more time for quality interactions with the same people and remove whiteboards to focus attention on the person speaking. However, we knew we would need to ensure that each group had an even distribution of participant characteristics that could signify varying viewpoints for the discussion topic to connect diverse perspectives.

To connect diverse perspectives between breakout room groups, mini-harvests would now take place at the end of each breakout room session where the scribes would feedback discussion highlights to all participants in the main room. Scribes were already tasked with making notes of the breakout room discussions, so it made logical sense for them to conduct the mini-harvests but participants would be encouraged to input additional comments if any points were missed.

Table hosts would still be asked to provide an overview of insights from all breakout room sessions at the beginning of the Harvest Session as we felt this encouraged participants to play an active role in listening together for patterns and insights. Scribes would also input a reminder of the question for discussion into the chat function at the beginning of each breakout room session to keep discussions relevant.

Lastly, the technology host reported operating at full capacity when running the pilot, so we decided to cap the maximum participant number for our virtual world café at 20 with no more than five in one of four breakout rooms.

The final structure and process of our virtual world café method includes stages: preparation, procedure, and data analysis and is discussed in detail in subsequent sections.

2.2.2. Preparation

Prior to the live virtual world café, participants are sent the questions for discussion during the session and guidelines for participation and using the teleconference tool being employed. Participants are also invited to an optional virtual training session on the teleconference tool. To support the running of a virtual world café, a designated host and technology host are appointed as well as one scribe and one table host per virtual breakout room being used. Guidelines on specific duties are provided to those covering each role prior to the event and are summarised in Table 2.

All roles apart from table host are covered by non-participants to allow participants to focus on critically reflecting on and responding to questions posed.

2.2.3. Procedure

Our virtual world cafe procedure consists of an intro, three Breakout Room Sessions followed by Mini-Harvest Sessions, and one Harvest Session, as depicted in Figure 1.

Figure 1.Example format and timings of virtual world café with three Breakout Room Sessions.

Three Breakout Room Sessions are used to allow for the discussion of three sequential questions that lead into the next, one per each Breakout Room Session, to facilitate deeper reflection on the topic as the event progresses. To support conversation results that will inform technology design outcomes based on user needs, questions are formulated to take a “cultural probe” approach (Gaver et al., 1999) such as:

Q1 encourages participants to reflect on and explain their current experiences (in Breakout Room Session 1)

Q2 invites discussion on related hopes and dreams such as how they would like to change their current experiences (in Breakout Room Session 2)

Q3 prompts new design ideas such as imaginary technology to make imagined improvements to experiences a reality (in Breakout Room Session 3). This flexible structure also allows questions to be tailored for exploring design requirements for different AI and other technology.

The host leads the introduction by introducing team members and their roles, highlighting the World Café principles and provides context for the questions to be discussed in the Breakout Room Sessions.

Subsequently, participants are divided into small groups of up to five for the Breakout Room Sessions including each group’s table host. Each group is placed into one of four designated virtual breakout rooms where they discuss their responses to a question that they previously reflected on prior to the session. The table host and other participants are divided evenly between breakout rooms in relation to relevant participant characteristics that could signify varying viewpoints for the discussion topic to support diverse discussion outcomes.

A Mini-Harvest Session follows each Breakout Room Session in which everyone re-joins the main room to hear discussion highlights from each Breakout Room group. The event ends with a Harvest Session where the table hosts outline the main themes from the three Breakout Room Sessions and an open discussion is had between everyone on conclusions overall. All virtual world cafe conversations are audio recorded and used to generate transcriptions for data analysis.

2.2.4. Data analysis

Data analysis involves analysing the transcriptions of discussions using a modified version of Corbin and Strauss’s (2015) grounded theory. Grounded theory is a data analysis strategy for uncovering new theories from qualitative data (Corbin & Strauss, 2015) such as transcripts, and is a promising strategy for discovering design requirements, as a new theory, from user data. Grounded theory includes stages: (1) opening coding, (2) axial coding, (3) selective coding.

Open coding of transcribed discussions involves manually highlighting and grouping text into discrete parts to create initial themes and sub-themes, referred to as codes. Axial coding involves reviewing the codes and their associated text to understand how they relate each other and organising them into categories (Corbin & Strauss, 2015). A category can be based on an existing code, or a new category can be created that contains multiple codes. Selective coding involves identifying a core category that connects all previous categories and codes together to create an overarching theory to explain findings (Corbin & Strauss, 2015). In this case, the design requirements form several core categories where rationales for each are provided by reviewing all other categories and codes.

2.3. Using our virtual world café method to explore sound sensing AI for home wellbeing

We adapted our virtual world café to specifically act as a PD method for exploring sound sensing AI for home wellbeing.

2.3.1. Preparation

Our virtual world café method and protocol for testing the method was initially reviewed and approved by the host university’s Ethics Committee. To support participant relatability when discussing sounds in the home and narrow the user group for the purposes of testing the method, we recruited participants who lived within a 10-mile radius of Guildford. Other recruitment criteria included:

• Adult (over 18 years of age)

• Have an interest in improving the sound environment

• Have access to an internet connected and video and microphone enabled computer, laptop, tablet, or smartphone

12 participants, eight females and six males aged between 24 and 60+, were recruited by advertising a call-for-participation to social media groups local to Guildford.

Research project team members were allocated roles for running the event, such as host, technology host person, and scribes for each breakout room. A week before the event, we also invited three participants to volunteer as table hosts for each breakout room and the host offered additional meetings to table hosts to go through role duties if any were unclear.

2.3.2. Procedure

Our introduction to the virtual world cafe explained the capabilities of sound sensing AI with examples to inspire later discussions. We used three Breakout Room Sessions to divide the participants into three groups of four. The three questions for discussion in each Breakout Room Session were as follows:

Q1. What sounds can you hear in your home and what sounds do you find annoying, neutral, or pleasant and why?

Q2. What would be your ideal listening experience here?

Q3. What imaginary new technology would allow you to achieve this?

Our participants were divided evenly into breakout room groups by age range to support different generational perspectives on topics discussed. At the end of the virtual world café, we asked participants to complete a short feedback questionnaire (see Appendix A) sharing their experience of the event, such as what they liked about it and how we might improve future events.

2.3.3. Data analysis

During open coding of conversational data, we started to identify themes that included pleasant, annoying, and neutral sounds, and sub-themes as specific sounds that were considered pleasant, annoying, or neutral as a well as variables that effected how sounds were perceived. During the axial coding phase, we reviewed these initial themes and sub-themes and created explanations for how they related to each other, such as why a sound was considered pleasant, annoying, or neutral, to finalise them into categories. Selective coding involved finalising design requirements for sound sensing AI as core categories by reviewing how they related to other categories, such as sounds considered annoying, pleasant, or neutral and variables effecting their perception.

3. Results

Over the course of its 2 hours and 20-minute duration, our virtual world café generated 47 pages of transcribed conversational data and 12 completed feedback questionnaires. Analysis of these results allowed us to (1) understand participant engagement during the event, (2) identify design requirements for sound sensing AI and (3) gather feedback on how participants experienced the overall format. The following sub-sections will now elaborate on each of these points.

3.1. Participant engagement during event

All recruited participants participated in the virtual world café for its full duration.

Conversational data resulting from Breakout Room Sessions demonstrated each participant individually reflecting on how pleasant, annoying, or neutral home sounds could have positive or negative impacts on wellbeing at home and how changes to the sound environment through future technology could improve this. Review of the data showed over half of participants contributing to the Harvest Session. Three participants acting as table hosts summarised the breakout room discussions at the beginning of the session and four additional participants added further reflections on themes discussed during the remaining open discussion.

3.2. Findings from data analysis

Analysis of conversational data of Breakout Room Session 1—where participants reflected on sounds they liked and didn’t like, and why—led to the identification and categorisation of common pleasant, neutral, and annoying sounds that all 12 participants agreed were experienced in the home with explanations. Table 3 lists examples of these sounds with accompanying rationales.

For example, natural sounds were identified as pleasant sounds from participant statements such as “I do just like to hear soothing, yeah natural noises that just… that kind of fade into the background but fundamentally the effect they’re having on me is quite just a relaxing effect…”.

Subsequently, data analysis of Breakout Room Session 2 data—which included participants’ reflections on how they would like to change their sound environment and why—revealed several variables that could influence whether a sound was perceived as pleasant, annoying, or neutral and rationales for each. For example, having control over sounds at home by removing annoying sounds and introducing pleasant sounds depending on their current activity (e.g. relaxing, sleeping, or working) featured in every participant’s description of how they would optimise their listening experiences. All 12 participants also described how their personal preferences of certain sounds determined how they wanted to tailor their listening experiences such as four preferred complete silence for relaxing while eight preferred some background sound.

When describing their desire to control sounds at home, over half of participants remarked on how their home routines could determine what sounds they wished to hear or not hear at different times of day. Lastly, eight participants described how the social significance of sounds, such as whether they signified anti-social behaviour (e.g. shouting) or social cohesion (e.g. laughing and background chatter) could determine whether they wanted to remove, keep, or not introduce associated sounds. Table 4 lists these variables for sound perception with explanations and example participant comments that informed them.

Data analysis of Breakout Room Session 3—where participants suggested imaginary technology to improve sounds in their homes—then identified sound-based technology features that were desirable to participants. It was also found that each technology feature could be related to each variable identified from the Breakout Room Session 2 discussions (see Table 4) and could be grouped under each accordingly. Design requirements for sound sensing AI for home wellbeing could then be identified by considering what design requirement was inferred by the desired technology features and the associated variables for sound perception.

Specifically, all participants proposed a variety of technology features to control what sounds they heard, demonstrating a design requirement for customisation. For example, nine participants wanted to remove unpleasant sounds such as mechanical sounds, and five wished to mask these and emphasise pleasant sounds such as bird song. Seven participants desired technology that adjusted to personal preferences by learning to recognise and disrupt annoying sounds or introduce pleasant sounds, suggesting the design requirement intelligent. Six participants also discussed the social significance of proposed technology features. This included the importance of not disconnecting the listener from social communication or introducing sounds that disturbed others, illustrating a design requirement for the technology to be pro-social.

Finally, all participants indicated that they wanted a technology that would seamlessly integrate with their home routines, suggesting the design requirement integrable and unintrusive. For example, four desired an unnoticeable wearable device that they could easily carry with them and eight preferred discrete technology built into the home infrastructure that would automatically activate when needed. Table 5 lists the five design requirements that were arrived at with the corresponding variable, technologies features and example participants responses that led to the identification of the technology features.

Data analysis of the Harvest Session, such as the table host summaries of key takeaways from each Breakout Room Session and additional comments offered by other participants further confirmed previous data analysis findings. Additionally, Harvest Session data summarised the main priorities for participants in supporting home wellbeing through sound sensing AI. This included the importance of access to natural sounds, especially bird song, to promote relaxation and removal of mechanical sounds such as heavy machinery through technology for removing the noise or informing legal restrictions on time and place usage.

3.3. Participant questionnaire feedback of event

Feedback questionnaire results showed that participants enjoyed the event. Each participant rated it as either somewhat enjoyable (three), very enjoyable (five) or extremely enjoyable (four). Participants’ positive comments included engaging and well organised (three), and supported diverse perspectives (three), exploration of ideas (four), and space for personal reflection (two). Participants also indicated that they were adequately supported to participate by reporting either having enough (two) or plenty (ten) of support and information prior and during to the event.

Half of participants remarked that no improvements to the format where necessary and there was no consensus on suggested improvements from remaining participants. Individual improvement suggestions included mixing up the breakout rooms, providing a countdown timer in breakout rooms, distinguishing the questions more to prevent overlap of discussions, extending the session to run over two days to allow more time for reflection, and including an expert’s view on sound sensing AI.

4. Concluding discussion

This paper presents a modified virtual world café method as a PD method for AI design, using sound sensing AI for home wellbeing as a test-case. Our virtual world café method was strategically used as a PD method for this purpose due to the traditional World Café’s method ability to quickly combine varying perspectives from multiple stakeholders, such as end-users, simultaneously into collective ideas while adhering to the restrictions of the COVID-19 pandemic.

4.1. Supporting participant engagement and reflection

Our virtual world café format provided an enjoyable and engaging event for participants to explore how everyday sounds in the home affected their wellbeing and how technology might improve these experiences. Discussions resulting from the questions (i.e. 1, 2, 3) allowed participants to reflect on how pleasant, annoying, or neutral home sounds could have positive or negative impacts on home wellbeing and how changes to the sound environment through future technology might improve this. The Harvest Session provided a space for participants to summarise discussion highlights from the Breakout Room Sessions and for others to add additional reflections if they wished to finalise themes.

Suggested improvements from participants also implied their engagement and interest in the event. There was a demonstrated keenness to: interact with new people (i.e. by mixing up the breakout rooms), talk for as long as possible (i.e. by including a countdown timer in each breakout room), provide unique responses to each question (i.e. by distinguishing the questions more), extend the event to accommodate further reflection (i.e. by running it over two days), and to learn more about the topic (i.e. including an expert opinion on the subject during the event).

4.2. Generating data leading to significant findings in a short timescale

Our virtual world café shows promise as an efficient PD method for generating ample data with different stakeholders, such as users, to identify design requirements for AI design as well as a resource efficient method for inviting many different people to contribute ideas simultaneously. In our presented test-case that explored designing sound sensing AI for home wellbeing, the method’s use of three sequential questions rounds (i.e. Breakout Room Session 1, 2, 3) and one concluding open discussion (i.e. Harvest Session) generated plenty of conversational data from 12 participants in a short timeframe to identify design requirements. Analysis of this data led to the identification of different sounds and their impacts on home wellbeing, variables for sound perception in this context and five design requirements for associated technology.

These findings, such as participant identified pleasant, annoying and neutral sounds, can be used to inform the datasets needed to train future sound sensing AI for home wellbeing. Additionally, findings, such as design requirements and suggested sound-based technology features, can be used to guide how sound sensing AI should be designed to support home wellbeing, including what kinds of technology it should be combined with, such as sound removal systems.

4.3. Limitations of our virtual world cafe

Notably, our virtual world café format does not include a follow-up process to validate the findings, such as design requirements, from applying the method such as a feedback mechanism to review the identified design requirements with participants. Considering Arnstein’s Ladder of Citizen Participation (see Arnstein, 1969), we acknowledge that our virtual world café as a PD method could be applied to support consultation rather than partnership if World Café principles are not adequately implemented and follow-up participant engagement is not sought to validate findings.

Application of additional PD methods are also needed to conceptualise and test example concepts with end-users. Our virtual world café also does not address how end-users can provide continuous feedback in the training, building, and validating of resulting AI concepts, and in the case of sound sensing AI for home wellbeing, how improvements to psychological wellbeing are measured.

Moreover, our virtual world café method is only accessible to those who have access to appropriate internet connected devices with sufficient digital literacy to use them. Lastly, as an online method, it lacks opportunities for serendipitous social interactions where participants can move freely around.

4.4. Advantages of our virtual world cafe

Our virtual world café method offers a starting point for collaborative AI design with end-users in a short timeframe, especially when face-to-face PD methods are not possible. Participants do not require any prior knowledge of AI to take part in our virtual world café and the structure of the prompting questions supports transferability to other AI and technology development.

The prompting questions used in our virtual world café can also be modified following a cultural probe approach (discussed in Section 2.2.2) to explore design requirements for other forms of AI and technology in future research. For example, research investigating AI companions would include three prompting questions exploring (1) current experiences of social interactions, (2) ideal social interaction experiences, and (3) imaginary AI companions for supporting ideal social interactions.

Despite the restriction of 20 participants, the method has the potential to be run multiple times, on different days with different participants to gather diverse insights from various stakeholders in future research. Lastly, as a virtual method, our virtual world cafe overcomes participant mobility issues and can include a diverse range of participants from both national and international locations.

5. Future work

Our virtual world café only explored sound sensing AI for home wellbeing from the perspectives of a small UK residential group from the Guildford area. Further research is needed to validate the identified design requirements by reviewing, testing and verifying identified design requirements with research participants and exploring sound sensing AI for home wellbeing from different user-group perspectives. Additional research is also needed to understand if the resulting design requirements are specific to sound sensing AI for home wellbeing, and if different design requirements will emerge when exploring other types of AI from a user perspective using our virtual world café.

Moreover, substantial further research is necessary to identify and develop suitable follow-up PD methods to our virtual world café that allow users to provide continuous feedback in the training, building, and validating of resulting AI concepts.

6. Conclusions

This paper presented a modified virtual world café as a suitable PD method for identifying design requirements for AI design in collaboration with participants as potential end-users. Sound sensing AI for home wellbeing was presented as the test-case to demonstrate the viability of our virtual world café method as a PD method for AI design. Results showed that our virtual world café method supported engagement from all participants, provided valuable findings in a short timescale for identifying design requirements as well as an enjoyable experience for all those involved. With limitations acknowledged, our virtual world café method presents a promising PD method for supporting AI design at the beginning of development. We hope that this paper inspires future research for designing AI using PD methods as well as future PD methods for AI design.

Acknowledgements

We thank all the participants involved in this research for their time, creativity, and insights. This research was funded by EPSRC under grant number EP/T019751/1 at the Centre for Vision, Speech and Signal Processing (CVSSP), University of Surrey.

Declaration of interest statement

On behalf of all authors, the corresponding author states that there is no conflict of interest.