1. Introduction
The United Nations Convention on the Rights of the Child (UNCRC) obligates the countries who ratify it to give an opportunity for children and young people to offer their views, be listened to and taken seriously, as well as incorporate their views on matters that affect them (Article 12). In line with this, there is recognition of the need to properly include children and young people (CYP) in research about them, acknowledging their expertise and enhancing their status within research (Bradbury-Jones & Taylor, 2015). Participatory research with CYP has resulted in numerous approaches being developed, emphasising CYP as partners (Montreuil et al., 2021). Such approaches are not without challenges, such as power imbalances between adults and children, and ensuring that CYP’s contributions are not overlooked or included tokenistically (Lundy, 2007; Mannay, 2016).
We report how we went about co-designing a situational judgement test (SJT) with CYP alongside adult research partners. This is noteworthy as SJTs, which measure a person’s decision-making skills, are often designed by and with adult subject experts even when CYP are the intended test users. In our case, the SJTs we sought to develop formed part of Project Fortitude, a child-centred action research project that aimed to develop CYP’s legal capability via game-based learning[1]. Within this project, the SJT was intended to evaluate Law Yeah![2], a digital legal learning game, also co-produced with CYP, to specifically assess its effectiveness in developing CYP’s decision-making skills in everyday legal or rights-based situations. Our approach brought together CYP’s lived experiences and views alongside the contributions of other stakeholders, namely adult experts and a team of digital game developers. The contributions of these adults were functional in the case of the game developers (i.e. they were under contract to produce the final game and digital assessments to our specifications). The remaining adults comprised the project team (who had expertise drawn from Law, Developmental Psychology, Social Science more broadly and Game Design), and teachers (who would be key in distributing and implementing the resources once developed and held expertise in relation to working in educational contexts and with CYP). This paper explains to how these tests can be designed and used with CYP, particularly as this is the first study that we are aware of which developed SJTs with CYP, viewing them as equal expert contributors.
In this paper, we set out our understanding of children as experts and research partners before detailing the standard approach to designing SJTs and the challenges faced when designing SJTs for CYP. We then detail how we accessed CYP’s expertise and enabled their contributions to the design of our SJTs. The final section focuses on the challenges that arose during the design of the SJT, namely balancing the ‘competing’ views, and how we navigated this aspect of co-design.
1.1. Children as Experts
There is a need to recognise the knowledge that CYP hold as experts on their own lives and experiences. CYP report ‘disappointment and frustration’ at being unheard (Race & Frost, 2022, p. 6). Adults frequently dismiss CYP’s input ‘either because they are held incompetent in making judgements or because they are thought to be unreliable witnesses about their own lives’ (Qvortrup, 1994, in A. Clark & Statham, 2005, p. 46) and they are content with other adults speaking on behalf of CYP (NSPCC, 2024). We see examples of this even in cases where there are legal requirements to include CYP in decisions about their lives, such as in the case of children who need specialist educational provision (e.g. Bloom et al., 2020).
Within a research context, participatory methods are often viewed as way to enable the contributions of those historically silenced (Mason & Danby, 2011), but there are challenges which can limit or inhibit CYP’s contributions (Bradbury-Jones & Taylor, 2015). One of the most significant obstacles to inclusion of CYP experiences are the power imbalances that exist in both the research process and CYP’s daily lives (Mannay, 2016). Including CYP formally as research partners in the design process can help to create more equitable relationships between adults and CYP, enabling them to be actively included throughout the research process, from conception to data analysis and dissemination (Mannay, 2016).
However, other factors in the co-design process can also hinder, override, or exclude CYP contributions altogether. These include the competing roles, expertise and social systems that influence child-adult collaborations (Svensson et al., 2007). Methods to address these challenges include creating a safe environment in which CYP can express themselves and avoiding activities that ‘ask children to tell a story or take an action that is not part of their own history’ (UNICEF, 2024; cf. NSPCC, 2024). Lundy (2007) highlights the importance of providing a safe space and adopting methods that facilitate CYP contributions, ensuring that their views are communicated effectively, taken seriously, and acted upon. Additionally, acknowledging barriers and engaging in critical reflection on social hierarchies present within CYP’s lives, whilst not being able to rectify the issues, can help re-centre their experiences (Kiili et al., 2024; Mannay, 2016).
In recognising CYP as experts, it is also important to acknowledge that their expertise may not always be expressed through direct methods, such as speech, writing or the creation of artefacts. Silence, hesitation, uncertainty, and even misconceptions can function as meaningful forms of knowledge in their own right. In other words, the unsaid is often as significant as the said, and silence can operate as a mode of expression, resistance, or relational positioning rather than as a simple absence of voice (Hanna, 2022; Spyrou, 2016). Recognising and being attentive to quieter demonstrations of knowledge is therefore essential to understanding how CYP make sense of their experiences and how they participate as experts.
1.2. Adult-Led Design of Situational Judgement Tests
SJTs measure a person’s decision-making skills by presenting scenarios grounded in everyday and realistic events, requiring the test-taker to select a response from a set of options that best reflects how they would react or behave in that scenario (Weekley & Ployhart, 2005; Lievens & Motowidlo, 2016). Their grounding in real-life enables the tests to be relevant to test-takers (Gessner & Klimoski, 2005), and SJTs are particularly good for nuanced situations: they are flexible and can easily be constructed and tailored to specific contexts (Fritzche et al., 2005). Typically, SJTs are used in training and development within employment settings, enabling different stakeholders to establish what skills their employees need to develop (Fritzche et al., 2005). Within these contexts, SJTs are most frequently used to assess employees’ adaptive knowledge in specific situations (Fritzche et al., 2005). However, research into SJTs has also demonstrated that they can be effective as learning tools, aiding education and training in specific areas where students require additional learning or support in skill development (Cox & Barron, 2015).
Common SJT design methods include theoretical and expert-led approaches. Theoretical approaches are ‘top-down’ with the tests structured around pre-determined theoretical positions (Ployhart & Ehrhart, 2003). This method has been criticised as test-takers can often infer what the designer intends to measure and may adjust their answers accordingly (Ployhart & Ehrhart, 2003). This is less a social desirability problem per se, but more an issue of the abstract, idealised nature of theory-driven scenarios, which, due to not being as grounded within ‘reality’, can make the ‘expected’ response more obvious. In contrast, ‘expert-led’ approaches require researchers to work directly with test-takers to create SJTs that are grounded in experiences and use language and situations that feel familiar and meaningful. When SJTs are relatable and comprehensible, test-takers are more likely to draw upon their own memories, judgements, and typical behaviour rather than attempting to infer what the designers want (Ployhart & Ehrhart, 2003; Wolcott et al., 2020). This aligns with the four cognitive stages of SJT response: comprehension, retrieval, judgement, and response selection, where relatable scenarios support more authentic decision-making (Wolcott et al., 2020). Combining theoretical and expert-led approaches can therefore provide a balance, ensuring that SJTs meet the research aims whilst also being relatable to test-takers (Ployhart & Ehrhart, 2003).
SJT validity is thus closely tied to test design. SJTs do not always have a single ‘correct’ answer; instead, the ‘best’ response(s) depends on the construct being measured and for what purpose, allowing for a true assessment of a person’s skills that identify where additional learning is needed (Ployhart & Ehrhart, 2003). Therefore, specific and focussed SJTs are easier to validate as multidimensionality is reduced resulting in less room for test-takers to interpret the scenario differently to how designers intended (Guenole et al., 2017). Additionally, relatable scenarios and language ensure that test-takers select answers that accurately reflect their behaviour in that situation (Ployhart & Ehrhart, 2003; Wolcott et al., 2020).
Research on SJTs that measure CYP’s skills is limited to three previous studies (Franssens et al., 2022; Murano et al., 2021; Wang et al., 2009). These studies took a hybrid approach to SJT design, but CYP were not included as experts. Rather, only adult views and expectations underpinned their design. Whilst these studies concluded that SJTs are good for capturing CYP’s ‘intuitive judgements’ (Wang et al., 2009, p. 122), they also revealed challenges when designing suitable SJTs for CYP. These included communication and content issues, where scenarios and the response options were too complex or unrelatable resulting in CYP struggling to interpret and answer questions. The studies highlight that CYP would have benefited from more age-appropriate language, less ambiguous and simpler questions, and scenarios that reflected their lives (Franssens et al., 2022; Murano et al., 2021). Murano (2021) had to redesign them and Franssens (2022) recommended varying response options to reflect CYP’s everyday lives, and/or requiring participants to keep a daily diary so researchers could assess real-world examples. These challenges reflect the wider literature on SJT design and the need for context-rich and relatable scenarios and clear language (Guenole et al., 2017).
1.3. SJTs in Project Fortitude
Project Fortitude aims to improve CYP’s legal capability – knowledge, skills, and confidence - when dealing with legal and/or rights-based situations that CYP may face in their everyday lives. To achieve this, we developed a children’s legal capability framework centred around objectives that address each of these discrete areas (Watkins et al., 2025) and Law Yeah!'s game design is constructed around this framework.
Law Yeah! is a scenario-based digital game aimed at CYP aged 7 to 15 years. It features 21 everyday scenarios across seven levels, each centred on a different legal or children’s rights-based problem that CYP may face. In each scenario, players are presented with a decision-making challenge based around help-seeking. Their chosen response powers up a ‘lawometer’, which also provides feedback on the effectiveness of their decision and encourages reflection on alternative actions. Some scenarios also require players to escalate the issue further to seek effective help. For instance, level 2 of the game presents CYP with a vignette where they are shortchanged whilst shopping. CYP are asked ‘what will you do?’ and are presented with several response options (Figure 1). If a player selects, ‘Call the police’, the ‘lawometer’ only powers partway and the accompanying feedback explains why this is not the most effective action (Figure 2). CYP can then try again, learn more about the relevant law or UNCRC articles, or move on to another scenario. Through this process, the game teaches core legal capability skills in context: recognising a problem, identifying appropriate sources of support, and taking proportionate action.
SJTs were central to evaluating the game within the project. Their purpose was to measure how effectively the game developed CYP’s decision-making skills, as reflected in the game’s ‘what would you do?’ challenges. The SJTs were therefore designed to measure only the baseline skill objectives from our legal capability framework: ‘I am able to recognise this situation is a problem’, ‘I am able to transfer my learning about rights and law to new contexts’, ‘I am able to identify appropriate sources of support’, and ‘I am able to take appropriate action’ (Watkins et al., 2025, p. 7).
Focussing exclusively on baseline legal skills was intentional. These skills are designed to be attainable for all CYP, regardless of age or learning ability. We argue that a CYP being able to recognise that a problem is a problem and know that they have the right to speak up about things that affect them are the core skills to being legally capable (Watkins et al., 2025). Additionally, it was our aim that the SJTs would not present a single ‘correct’ response but instead present a range of response options that varied from least to most effective. What constitutes an effective decision may not always be what is grounded in legal instruments; in many everyday situations, the most appropriate action may be seeking help from a trusted adult. We wanted the SJTs to reflect this, particularly as not all legal or rights-based problems end up going through the legal system (Watkins et al., 2025).
As noted above, adult perceptions and expectations of CYP do not always align with their lived realities. Whilst the previous studies that use SJTs with CYP argued that these challenges did not affect test validity, they also demonstrated the need for a more inclusive approach to design. We argue that a participatory approach offers a means to address the content and communication challenges, making the assessments more relatable and understandable whilst importantly honouring and centring CYP’s contributions to assessment tools that are developed specifically to capture their understanding and performance in a given context.
2. Methods
2.1. Approaching Recruitment and Consent
We partnered with CYP aged between 7 and 15-years-old in demographically diverse primary and secondary school settings in England. This included working within inner-city schools including those situated within economically deprived areas, rural schools, private fee-paying schools, and with CYP whose English is not their first language or who had special education needs and disabilities (SEND). Table 1 provides an overview of the CYP involved. Who participated, at what stage of the project, and how that varied over the course of the project and are detailed in the relevant sections that follow. All our work with CYP detailed here received ethics approval from the University of Leicester or University of Sheffield ethics committees (the initial mapping work described later occurred when the work was based at University of Leicester). Informed consent from the Headteacher and parents/guardians was obtained in advance. At the request of one participating school and with additional ethical approval, the storyboarding and diamond ranking activities were run on an opt-out basis; parents/guardians were notified of our activities in advance and were given the opportunity to withdraw their child from those sessions.
Prior to working with CYP, we visited their schools to speak with potential participants. This allowed them to get to know us, learn about the project, and provide them with the opportunity to ask us any questions ahead of deciding whether to participate. We were fully transparent with CYP during these visits, including that we were developing a legal learning game, and we explained that their insights would directly shape its content and design. We emphasised that we wanted to work with them as experts and collaborators, rather than evaluating them. The activities outlined in the sections below encouraged creativity and were framed to support the elements (e.g., places, situations, relationships, decisions) that were already a part of their everyday lives, ensuring that the work we did together would connect naturally to their interests and experiences. This helped CYP understand the relevance of the project and supported their engagement as experts.
In addition, during all our visits we adopted Mandell’s (1988) ‘least-adult role’, which involves presenting ourselves in ways that reduce social hierarchies, like those typically found in school settings between adults and CYP. Practically, this meant breaking down barriers and building rapport. This included, for example, researchers also adopting the same fruit and vegetable names that were allocated to CYP as pseudonyms or dressing in casual clothing. In UK school environments where school uniforms are commonplace, where school staff may wear formal attire that reflects their authority, and where CYP address adults through hierarchical titles such as ‘Miss’ or ‘Sir’, these small acts helped us to soften the barriers which may otherwise have impacted how CYP interacted with us. This resulted in a more relaxed and interactional space in which CYP could speak freely, challenge ideas, and contribute their expertise. Additionally, the research team obtained CYP’s assent before each session through the use of a specially developed and illustrated ‘assent mat’ (Mills et al., 2026), designed to maximize their understanding of what their rights were as research participants through a series of illustrated statements that they engaged with. CYP confirmed that they understood their rights and that they were free to withdraw from any activity if they wished.
2.2. Accessing CYP’s Expertise
Within Project Fortitude we used a range of methods to involve CYP and maximise their roles as experts within our SJT design. We primarily drew upon on four methods: mapping and storytelling, diamond ranking, storyboarding, and piloting.
2.2.1. Mapping and storytelling
We began by inviting CYP to work in groups, then individually, to map the places they go, the things they see, and things that happen there. To do this we invited CYP from four primary schools and two secondary schools to engage in this mapping activity with members of the research team, and this resulted in maps from 123 CYP (see Figure 3).
Individual maps were then storied by the CYP who created them to other members of their group in a ‘gallery’ exercise, where they had opportunity to ask and answer questions, and to a member of the research team in a 1-1 interview, which was audio-recorded (see Figure 4).
The maps and transcripts were indexed, recording locations (e.g. home, shop, school, park), characters (e.g. family member, school staff, other child), and activities (e.g. transport, sport, play). Notable scenarios were also recorded, such as incidents involving the police, broken toys or equipment, and buying items from a shop.
A preliminary analysis treated maps and stories as a single dataset. However, this was considered oversimplistic, since the places and scenarios discussed by CYP could be quite different. Therefore, we explored the relationship between locations, characters, activities and incidents separately for data extracted from maps and stories.
The aim was to establish how common different combinations of location, character, activity and incident were. We wanted to understand what types of experience were common, with a view to building settings and scenarios to explore legal capability that would resonate with and engage CYP. Although the maps were produced by a diverse group of CYP and resulted in maps and stories which differed in detail, our focus on places, people, and everyday situations, enabled us to find the common structures within this diversity: home, school, shops, parks, transport, and interactions with peers and adults. By ‘experiences’ we refer to the kinds of situations that CYP described occurring in these settings, whether it was buying items, navigating school rules, resolving disagreements, or interacting with adults in positions of authority. These experiences were not necessarily presented by CYP as ‘legal’ or ‘rights-based’ experiences (or even as problems), but as everyday moments within their lives in which they navigate decisions, relationships, and expectations. This allowed us to identify recurring and relatable situations that were common enough to inform later stages of the project without assuming all CYP share identical experiences.
2.2.2. Scenario creation
The next step was to share these findings with CYP, and to engage them in co-creating and developing scenarios for potential inclusion in both the game and the SJT that would assess any learning from it. As the game needed to be accurate in relation to the law it was necessary for the research team to present some hypothetical law-related scenarios for discussion and further development with the CYP, in addition to the ones the CYP told us they had direct experience of. However, we took care to ensure that the scenarios that we suggested were still situated in contexts which resonated with the CYP and could be linked back to their stories.
Members of the research team took the socio-physical domains identified (home, park, school, being online, eating out, shops, sports and leisure, streets, using transport, and religion) from the first phase of work and drafted approximately 10 short scenarios for each one. These were further refined through discussion among the wider team. Finally, a simple illustration was created for each scenario by a member of the research team. (see Figure 5)
Some of the situations identified as potentially important for inclusion within the SJT and the game touched on sensitive topics, including adverse childhood experiences (ACEs), such as divorce, abuse, or neglect. It was agreed that more sensitive scenarios would be worded in the second person (‘your friend…’) rather than the first person (‘you…’) in recognition that in we could not know what CYP working with us may have personally experienced, and so some CYP may be more sensitive to the wording of those scenarios than others.
2.2.3. Diamond-ranking
To explore which scenarios resonated with CYP within the domains already identified, participants in a primary school (n = 316) and two secondary schools (n = 22 and 30) were invited to work in small groups to rank the scenarios using a diamond ranking method. Diamond ranking or ‘Diamond 9’ is an established method for eliciting opinions and beliefs (Niemi et al., 2015) and for encouraging discussion, thinking and reflection (J. Clark, 2012).
It was explained to participants that the purpose of the exercise was to inform the game design, so scenarios which they considered important or interesting for CYP to learn about should be ranked highest. They were invited to discuss the scenarios, stick them to the relevant part of the diamond (see Figure 6), and then briefly explain or annotate the reasoning behind their ranking using blank paper speech bubbles provided. CYP could also swap out a scenario for one they themselves created.
Scenarios which had been developed based on the ACEs literature were deliberately excluded from the diamond-ranking exercise, because we were aware that this could upset some participants and there was limited opportunity for the research team to monitor and respond to this appropriately given the scale of the activity. These scenarios were considered by CYP in a later phase of the development work instead (see section 2.2.4 and 2.2.5) where appropriate monitoring and support could be provided if required.
The diamond ranking activity prompted high levels of engagement from the CYP and provided some insights into issues that they considered important and why. However, there was not a clear consensus across CYP regarding which scenarios that they felt were most important. This further underscored the individualised nature of children’s experiences. To address this challenge, responses from the secondary and primary school groups were analysed separately and the secondary school groups were invited to choose from a reduced pool of their top-ranked scenarios and scenarios they introduced during the exercise and create a storyboard. A more complex approach was taken to the analysis of the diamond-sorting data from primary schools, with a frequency analysis informing which scenarios were taken forward to storyboarding.
2.2.4. Storyboarding
This activity centred on gaining CYP’s insight into everyday legal or rights-based scenarios. We worked with 316 CYP in one primary school (283 CYP across 11 classes) and two secondary schools (33 CYP across two Year 9 groups). Our primary school participants included children with special educational needs and disabilities (SEND) who usually received specialist learning in a designated SEND class, and who were integrated within the ‘mainstream’ classes during our visits. Both secondary school groups (aged 13 to 14) were of mixed ability, and one group comprised CYP where English was an additional language.
CYP were asked to create a realistic story using a scenario of their choice from the ‘scenario bank’ (discussed in section five, below). Stories were captured on A3 storyboarding sheets (Figure 7) which included space for CYP to decide ‘what happens next’ and prompts were provided as a guide (e.g., ‘what happened?’, ‘did you/they speak to anyone?’). CYP could develop two endings which would help us understand their perceived consequences of a situation. To support the different abilities and interests of participants, CYP were encouraged to storyboard in any way they wished, and we collected a mixture of more traditionally written narratives as well as illustrated stories, comics, and other artwork.
We received 288 storyboards from primary school students and 43 storyboards from secondary school students. A descriptive analysis was conducted. This involved transcribing the storyboards and describing any visual data. Attention was paid to the language and style of writing CYP used as well as the wider story itself, specifically the narrative events, the socio-physical domains, and what actions (if any) were taken to resolve the situation. The approach was reflexive: when analysing, we were mindful of our own interpretations, positions, and approaches to reading the data.
As shown in Figure 7, the storyboards were created on a template beginning with their chosen scenario and ending with both fair and unfair outcomes. The storyboarding was useful in understanding the language used by CYP to describe legal issues. These storyboarded accounts of how the CYP understood each scenario and potential responses to it informed both the design of the game, and the SJTs we needed to incorporate into the game to measure the development of their understanding of legal situations. A significant aspect of this was the inclusion of emotional reactions in the range of response options linked to scenarios; as CYP’s responses to the scenarios were emotionally contextualised and we felt it was important to reflect this in how some of the responses were framed.
2.2.5. Piloting SJTs
From the storyboarding exercise we identified a set of situations and potential responses. We piloted these draft SJTs with CYP at different primary and secondary schools. Participants included primary and secondary school Child and Young Person Advisory Boards (CYPAB) formed during Project Fortitude and a secondary school group that we were working with us on the project to develop game content. This additional group comprised of CYP who were elected to participate as part of their school’s enrichment curriculum. Their inclusion was a direct result of our secondary-aged CYPAB, who expressed concern that they may not be representative and advised that we should include another group of similarly aged CYP.
Eight CYP from our primary school CYPAB (Year 3, aged 7 to 8), seven from our secondary school CYPAB (multiple year groups, aged 13 to 16), and nine from our additional secondary school group (Years 8 and 9, aged 12 to 14) participated in this phase. A range of different legal and rights-base scenarios were piloted, and secondary school sessions were delivered using TurningPoint voting software. Our primary school CYPAB used pen and paper. Consequently, this group were able to alter their answers and where this occurred, researchers made a note of both their original and revised answers. To ensure anonymity and avoid peer pressure in this group, we asked that they keep their answers ‘secret’ and CYP took it upon themselves to shield their sheets from their peers. Where TurningPoint was used, all answers were gathered anonymously and immediately presented in a bar chart for the group to view. During the session, CYP knew their expertise was being sought. They were briefed on SJTs, how they are designed, and their purpose. During the piloting and post-pilot, CYP were given a de-brief for each question and invited to give further feedback and ideas.
3. Reflecting on Outcomes: How CYP’s Expertise Informed Our Design
CYP’s expertise shaped the design of the SJT items in three areas: scenario topics for the SJT stems, SJT relatability, and communication style.
3.1. Scenario Topics
SJTs are often scenario-based and designed to reflect real-life situations (Guenole et al., 2017). Research into scenario-based learning suggests that scenarios can capture imaginations and allow CYP to explore complex or sensitive issues within a safe environment (Barter & Renold, 2000). Within legal education, scenario-based learning supports critical approaches to law, providing a space to explore everyday legal problems (Baron, 1991) and, with CYP, using scenarios is an engaging method to explore CYP’s understandings of law within their everyday lives (Watkins et al., 2018).
As outlined above, ten socio-physical domains were established from the mapping exercise (namely, eating out, home, online, park and outdoors, beliefs, school, shopping, sports and leisure, streets, transport) and this was supplemented by the literature on adverse childhood experiences (ACEs). The first two authors drafted a ‘bank’ of 98 legal and rights-based scenarios which were encapsulated within a sentence and illustrated by the first author. CYP reduced these scenarios to fifty via the diamond-ranking exercise, during which they could also choose to substitute scenarios presented to them with ones they created. Where different scenarios addressed the same legal problem scenarios were merged, leaving a total of 30 scenarios.
The storyboarding and piloting activities enabled CYP to narrow the scenarios further to create suitable SJT situations. For instance, during piloting, CYP highlighted the scenarios that resonated with them and their lives. These scenarios were mostly grounded within criminal law, such as theft, criminal damage, and secondary liability. In the storyboarding activity, we noted how criminal legal responses were often brought into stories, even when the legal issue was civil. Piloting supported this, e.g., 75% of our primary school CYPAB opted to ‘phone the police’ when short-changed in a shop. These findings align with other research that examines children’s understanding of the law where it is generally noted that CYP’s legal knowledge is often shaped by misunderstandings and misconceptions of the law and legal process (Crawford & Bull, 2006; Peterson-Badali et al., 1997). This is significant as it provides us with an opportunity to develop both CYP’s legal knowledge and skills regarding misconceptions such as these, equipping CYP with an effective solution that will likely result in a successful resolution.
CYP also advised on the details that made these issues relatable. For instance, in piloting, CYP explained that the scenario on criminal damage and secondary liability (throwing stones at cars) was relatable. However, they commented that the response option ‘I throw stones too’ was unappealing due to the obvious risk of damage which makes taking part yourself unlikely. Instead, they offered an alternative scenario: egging[3] houses. They explained that this was a common occurrence, and one which is perceived amongst their peers as a harmless act that also had a ‘danger feeling’ which makes ‘you join in too’ a more tempting response option.
During piloting, CYP informed us which scenarios were unrelatable. These were mostly scenarios where the issue happened to someone more specific (as opposed to a more general scenario). For instance, CYP explained that scenarios on disability discrimination were ‘off-putting’ (Secondary CYPAB). In explaining why, CYP constructed a ‘us’ and ‘them’ dichotomy, situating the problems faced by ‘the other’ as unrelatable to their lives. ACEs presented a similar issue. In the storyboarding activity, we observed that many CYP had difficulty in constructing stories for some of our ACEs (e.g., neglect, abuse). CYP often went ‘off-track’, producing elaborate stories that brought in additional and sometimes unconnected issues and some CYP chose not to engage with the ACEs, replacing them back in the ‘bank’ and selecting another.
3.2. Relatability
The feedback from CYP on relatability aligns with SJT design literature and studies that use SJTs with CYP (e.g., Murano et al., 2021). For instance, CYP highlighted the need for the scenarios to be context-rich. During piloting, CYP explained that many scenarios were ‘left to the imagination’ (Secondary School CYPAB) as they were reflecting upon their current lives. Additional detail, like a shop being local, would help place themselves ‘in’ the SJT and ground it within their reality.
CYP also supported the design of relatable responses to scenarios. The storyboarding activity provided a starting point. Across storyboards, scenarios, and age groups, CYP often developed stories that explored their experiences. In line with the literature, all age groups showed an awareness that they had relevant experience, but recognised they struggled sharing it due to their societal status and ‘not being believed’. This affected whether they sought help for a problem or solved it themselves (agency was raised as an issue in 246 out of the 288 storyboards discussed, and not being believed was raised in 162 of them). Primary-aged CYP mostly sought help from a familiar adult: a parent, a teacher, or the police, even in situations where these sources of help would not be able to provide support. Frequently, these stories involved CYP seeking help from these trusted adults only to be met with disbelief, punishment (e.g., detention, violence, and imprisonment), or no support. Conversely, secondary age CYP preferred self-help: their storyboards showed a degree of distrust for adults and seeking help was often presented as a last resort. Doing nothing was also a popular response. In some storyboards, the scenario was presented as ‘acceptable’, a non-issue; in others, CYPs created stories where characters simply did not know how to respond, instead exploring the continuation of the scenario.
One of the more explicit and common elements within storyboards was violence. Violence was often an immediate reaction when the CYP or characters were not being listened to when they sought help. Whilst the storyboards presented violence as an inappropriate response that triggered further consequences, it was often portrayed within storyboards as a viable solution to CYP not being listened to. During piloting, CYP also showed a preference for similar forms of ‘self-help’ such as hitting back or running away (e.g., from police).
3.3. Communication
Similarly to Franssens (2022) and Murano (2021), our work with CYP revealed that the language needs to be direct, understandable, and the issue explicit. It is within the area of communication that CYP provided the most expertise. CYP showed some awareness of legal and rights-based words. In the storyboards, words associated with criminal law and justice were sometimes used, like ‘custody’, ‘arrest’, ‘victims’, and ‘evidence’. Whilst CYP showed unfamiliarity with how the criminal justice system functioned, these words were mostly used correctly. CYP also used ‘sue’/‘suing’ frequently; a civil law term. CYP linked suing to reparations, showing an awareness of its meaning and the broad legal outcome; although, their reparations were often unrealistic (e.g., a free holiday for being fed unhealthy food at school).
In the storyboards, CYP identified specific support systems such as ChildLine (a helpline for CYP experiencing abuse or neglect) and the government. However, the piloting exercise revealed that we cannot take this for granted. For instance, our primary school CYPAB lacked awareness of ChildLine; one CYP said they thought ChildLine was ‘a line of children’ and another thought it was a place for ‘naughty’ children. ChildLine also provoked conversation amongst our secondary school group, with one CYP being critical of including such services. They commented that for CYP who are experiencing serious issues, such as neglect or abuse, contacting ChildLine means the child enters ‘the system’:
‘You will have to go into the system if you call ChildLine – I don’t think you would call them or a trusted adult. A family member you know, maybe. Foster parents might be worse than your actual parents’.
However, the group agreed that terms like ‘trusted adult’ rather than specifying a particular adult were more suitable. These are important contributions as it allows us to include and build awareness of sources of help within the game and the SJTs.
CYP also highlighted language that was unclear. For instance, both CYPABs commented that if an action requires picking up a phone to contact someone, we should avoid saying ‘I phone’. The primary school CYPAB said it was a confusing homophone that sounded like the product (iPhone) and the secondary school CYPAB said it was not a phrase they used. Both groups suggested using ‘I call’ or ‘I ring’, noting these terms are used interchangeably. On point of view within the storyboarding activity, we noted that CYP preferred to construct stories using short sentences written in the first person.
Finally, CYP expertise contributed to the visual communication of the situations portrayed. Research highlights that visual communication can make SJTs more engaging to test-takers (Bardach et al., 2021; Heininger et al., 2021). During piloting, SJTs were illustrated with gender and racially neutral ‘blue people’ to avoid perpetuating stereotypes (Figure 8). Feedback was mixed. Overall, CYP agreed that visual aids were useful, commenting that illustrations helped to give context to a scenario and clarified where there could be ‘two meanings of words’ (Primary School CYPAB). CYP also explained that illustrations needed to link to the scenario as abstract representations would not help them understand questions: our primary school CYPAB suggested using comic strips for complex SJTs. On visual appearance, our primary school CYPAB commented that they preferred to see illustrations of people who represented themselves. Conversely, our secondary school CYPAB emphasised the importance of keeping a person’s ‘identity ambiguous’, commenting that blue was good as a skin tone as it could represent anyone. Our additional secondary school group were unbothered. They saw the benefits of the blue people, but using realistic illustrations would not affect how they answer the question: one CYP commented ‘even photos of real people isn’t realistic as it’s still a photo’.
4. Reflecting on Process: Balancing ‘Competing’ Expertise
Involving CYP as experts involves balancing competing views and interests. As noted above, taking CYP’s views seriously and acting upon them creates an environment that facilitates CYP’s participation (Lundy, 2007). Additionally, showing an awareness of, and reflecting on, the power imbalances that exist both within the research process and in CYP’s lives can avoid undermining their contributions (Kiili et al., 2024; Mannay, 2016). Due to the role of the SJTs within the project, there have been boundaries to CYP’s expertise. It was necessary to strike a balance between CYP’s knowledge, our legal knowledge, and the game’s parameters to ensure that the SJTs were effective. This section discusses how we balanced competing views and expertise and implemented this within the design of the final SJTs (Appendix 1). ‘Balancing’ concerned four areas: meeting the project aims, safeguarding CYP, incorporating correct legal actions, and design issues.
Based on CYP’s feedback, we directly incorporated a number of their ideas. For instance, scenarios were rewritten to present more of a narrative. CYP’s feedback on the SJT language supports the research into SJT design, which recommends using the first (i.e., ‘I go’) or second person (‘you go’) (Ployhart & Ehrhart, 2003). This is particularly necessary as CYP highlighted that they struggled to situate themselves within some of our proposed scenarios as they could not ‘connect’. Writing the SJTs in the first person was a way to address this and the SJTs were re-written to ensure that they were all in the first person. Efforts were also made to ensure that the vocabulary reflected CYP’s preferred style, and their ideas were directly applied to SJTs wherever possible. Further, whilst knowledge of specific legal terms was not necessary for the SJTs, we were able to ascertain what legal terms were familiar to CYP, which was useful for the design of the game itself. Similarly, because of CYP feedback, we were able to avoid using language which confused or was unclear to CYP.
As noted above, the scenarios that resonated the most with CYP were grounded within criminal law. CYP reflected upon our scenarios and provided suggestions as to how to improve them to construct relatable and effective SJTs, such as replacing throwing stones with egging houses as the seemingly insignificant act makes the ‘joining in’ response option more realistic and tempting. We incorporated these suggestions in the design of the final SJTs. However, we did not structure the interventions solely around criminal legal issues. We do not deny that teaching criminal legal issues to CYP is important. However, the game’s purpose was to build CYP’s legal capability across different legal and rights-based areas that feature within CYP’s daily lives, and our SJTs needed to assess understanding across all these contexts.
CYP’s preference for criminal legal responses is perhaps unsurprising: previous research into children’s understanding of law shows they have an awareness of criminal law and justice, although this is often limited and guided by misconceptions (Peterson-Badali et al., 1997; Watkins et al., 2018). Misconceptions were also present in criminal legal scenarios that CYP highlighted as important. An example of this is police stop and search. In CYPs’ storyboards and during piloting, police powers appeared to be a concern and CYP displayed a lack of awareness of policing and police powers. The issue is also important from a legal education perspective: research highlights that certain groups in society may be disproportionately targeted for stop and search and that, generally, people are unaware of their rights (Miller et al., 2020). Discussing this scenario with both the CYPABs revealed that they were confused by the legal issue, believing that police only target ‘bad’ people (Primary School CYPAB). In practice, under strict guidelines, within England the police can stop and search a person under the Police and Criminal Evidence Act 1984. There is no minimum age that a person can be stopped and searched, and section 60 of the Criminal Justice and Public Order Act 1994 sets out that in certain situations the police can stop and search someone at any time for any reason. Misconceptions are important sources of CYP knowledge. Mannay (2016) argues that, to understand expertise and facilitate participation, the unsaid is often just as important as the said (p.57). Here, CYP’s apparent lack of knowledge and misconceptions is a form of knowledge. By setting legal issues out carefully, we could use these ‘knowledge gaps’ to develop CYP’s legal capability across various scenarios in the game and assess it via the SJTs.
Similarly, we contend that issues that only affect specific groups (like disability discrimination) are important issues that all CYP should develop legal capability on, although we acknowledge that the SJTs need to be relatable to the majority of test-takers. As such, we took on board CYP’s feedback and decided not to include legal issues that were targeted towards specific groups within the SJTs. However, due to the importance of these scenarios and our desire to develop CYP’s legal capability of these issues, they feature throughout the game.
A similar approach was taken with regards to ACEs. As aforementioned, CYP lead complex lives and social pressures, stigmas, and hierarchies exist not only between CYP and adults, but also between CYP themselves (Mannay, 2016). Reflecting on CYP’s difficulty with constructing stories around ACEs, and with their description of ACEs as ‘unsettling’ during piloting, we recognise that these social pressures, such as CYP feeling uncomfortable in engaging with situations openly in front of their peers due to social stigma, could be contributing to their lack of engagement.
This presented an ethical issue as well as a potential safeguarding one. Whilst ACEs were included within the game, we recognised that these were difficult and unsettling topics which may present issues when included in SJTs. The SJTs would be undertaken by all CYP participating in the game’s evaluation before they got to play the game to provide a baseline assessment. In the game, discussion of ACEs was limited to advanced levels and phrased to be suggestive in order to be suitable for our entire age range. For instance, our scenario on sexual assault (‘One of your classmates tries to touch you in a way that makes you feel uncomfortable’) allows for the issue to be explored gently; younger children may not necessarily pick up on the sexual subtext but will nevertheless learn how to deal with issues surrounding non-consensual physical contact. Older CYP could recognise the issue and learn more on the topic and access appropriate sources of help through the game.
To minimise such issues, only two ACEs – bullying and cyber-bullying – were included in the SJTs. We chose these for three reasons. Firstly, from our work with CYP, both issues are important and commonplace. Secondly, these scenarios allow us to test CYP’s recognition of sources of help in two different but familiar environments which can be transferred to different issues that can occur in these spaces. Finally, whilst the effects of bullying and cyberbullying may be unsettling, we could present the issues in a style that minimised any potential discomfort.
Additionally, we needed to ensure that the SJTs presented CYP with the correct legal response to a scenario. We recognised the various actions CYP may take when responding to a problem as well as the reasons that underline them, and we sought to replicate this as much as possible to ensure relatability. As mentioned earlier, CYP found actions that may not be appropriate relevant and tempting responses, such as phoning the police when shortchanged, or actions that invoke their agency, such as violence. Whilst these actions may be appropriate and natural responses to problems in some instances, from a legal standpoint, these responses are not always suitable. We sought to find a balance and reflect this within the response options and scoring, rewarding positive actions regardless of whether it was the best legal action to take, whilst addressing the inappropriate actions.
Based upon this, we devised four types of response options, with each option having a score that reflects its legal appropriateness:
-
‘Inappropriate’: score of minus one.
-
‘Do nothing’: score of zero.
-
‘Positive action’: score of plus one.
-
‘Appropriate action’: score of plus two.
This response option structure allowed us to recognise and incorporate inappropriate responses or proactive ‘self-help’ which may not be the best but nevertheless CYP have identified as being relatable and tempting. For instance, in the bullying scenario, we incorporated a favoured and relatable response option - pushing or hitting the bully back - and its score reflects its effectiveness as a solution (a score of minus 2): pushing a bully back could result in the situation becoming worse or may create further legal problems, such as assault. Similarly, we incorporated common legal misconceptions in our response options and these were also scored appropriately to reflect their legal effectiveness (again minus 2).
Similarly, as noted above, storyboards also revealed that, at times, CYP simply did not know how to respond to a situation or did not view a problem as problematic. This aligns with research on legal capability (Pleasence & Balmer, 2014) and presents another example of CYP’s seeming lack of knowledge being a form of knowledge in itself. Not knowing how to respond or not responding at all is a valid and realistic response. The ‘do nothing’ response option is designed to capture this. During piloting, CYP responded positively to its inclusion; our secondary school CYPAB commented that several proposed SJT scenarios were so commonplace in their lives that they did not view the situation as problematic, so ‘do nothing’ was an accurate reflection of their response.
The game developers also set boundaries that influenced the SJTs. For instance, they limited the number of scenarios and learning outcomes we could include, which influenced what scenarios were turned into SJTs. The game has 21 scenarios split across seven levels. The first three levels are designed to be suitable for CYP across our entire age range. From level four, the scenarios increased in difficulty and the advanced levels (six to seven) are targeted at older CYP or those whose legal capability has developed alongside the game. We decided to have a total of 12 SJTs that reflected the range of these game levels. Eight SJTs were scenarios that featured within the game and four tested the transfer of skills learned in the game to new scenarios.
Finally, whilst not the focus of this paper, the game’s visual design also affected the SJTs. Although CYP’s feedback on how people should be depicted was considered and we opted to use neutral illustrations of people to ensure inclusivity, we could not explore the use of comics or other forms of visual communication as suggested by CYP. The SJTs’ final illustrations are in the same style as those used in the game, and for the eight questions that represent the game levels, the same illustrations were used.
5. Conclusion
This paper has discussed how CYP are knowledgeable experts who can positively contribute to assessment design. It showed that CYP’s expertise is not limited to their worlds: ‘knowledge-gaps’ and silence (e.g., ‘doing nothing’) are also knowledge. By including CYP as experts, we have been able to address design challenges faced by other researchers who have sought to measure CYP’s skills, resulting in the design of SJTs that are arguably more relevant and understandable. As such, we are the first project to our knowledge to successfully incorporate CYP’s expertise in SJT development.
This paper also recognised the challenges of working with CYP as equitable partners. Navigating competing voices and interests impacted the extent to which we were able to act upon CYP’s expertise. Balancing has limited the role of CYP in some areas, but by outwardly acknowledging and reflecting on the underlying motivations, we have tried to address these competing influences without diminishing CYP’s voices and role within SJT design.
Acknowledgements
We wish to thank the children and young people, schools, teachers, and other school staff who supported and participated in this project. The European Research Council (ERC) funded this research under the European Union’s Horizon 2020 research and innovation programme (Grant agreement No. 818457).