Co-designing Interactive Music Performances with Children
This work resulted in a best paper award at ICIDS (International Conference on Digital Narratives) in 2019.
Live interactions have the potential to meaningfully engage audiences during musical performances, and modern technologies promise unique ways to facilitate these interactions. Through cooperative inquiry (CI) with Kidsteam, a co-design group at the University of Maryland, I investigated how audiences might want to interact with live music performances, including design considerations and opportunities. Findings from these sessions also formed a Spectrum of Audience Interactivity in live musical performances, outlining ways to encourage interactivity in music performances from the child perspective.
CI Session Structure. Sessions began with a discussion of the day’s design topic and goals. The team then divided into 3 small groups, each comprised of 2-3 children and 1-2 adults, and worked in these small intergenerational groups to design a technology that addressed the session’s goals. At the end of each session, small groups presented their technology design ideas to the entire design team while an adult design partner wrote the Big Ideas on a whiteboard and performed a rapid thematic analysis. The entire team then discussed and refined the themes and unique ideas that arose.
Data and Analysis. Across the three design sessions, at least one adult in each small group took observational notes. Session data included these notes, photographed session artifacts, audio recorded debriefs, and video recordings of presentations. Session themes (Big Ideas) were iterated through reviews of these data during discussions with the research team.
SESSION 1: Interactions with a Pianist
To provide initial scope, in this session the team was asked to consider a performance that included a single performer and instrument: How can an audience change what is being played during a piano performance? Small groups used a 3D prototyping technique called Bags of Stuff to build low fidelity prototypes of technologies that would interact with a live piano performance using art supplies.
SESSION 2: Interactions with Any Performer
As with the previous session, the design team was asked to envision ways an audience could interact with a performance as it is happening. However, in this design session no constraints were placed on the type of performance being attended. This session used the Big Props technique (Walsh, 2013) which incorporates large stage props (in this case, toy guitars, harps, etc.) in addition to art supplies to develop prototypes that focus on interactions \ref{fig:Session2}. Small groups were instructed to design as many ways for audiences to interact with music performances as they could.
SESSION 3: Degrees of Interactivity
In design sessions 1 and 2 observed that the design teams proposed a range of individual to group and passive to active interactions with performers. Inspired by these findings, the third design session began by asking small groups to review their designs to develop a spectrum of audience participation. Each group was given a list of previous ideas and asked to arrange them in order of least to most interactive (Figure 4). In the second half of the session, children individually voted on their favorite type of participation from the spectrum, and the team worked together to develop interactions to support the becoming performers category, which received the most votes.
Overaching Themes
Tangible Experiences: Groups primarily designed tangible, largely wearable, technologies such as interactive hats, hand sensors, and palm pushbuttons that could interface with the musical performance. Groups transformed both music and feedback into experiences that could be manipulated with tangible technologies. For instance, one group suggested creating tangible “sound chips,” discrete bits of music that audiences could append to performance melodies. Another group suggested using a wearable forehead sensor to gauge audience engagement from head nodding and shaking.
Addressing Personal and Group Preferences: Groups made it clear that individual preferences for the experience of the performance must be considered, and designed technologies that encouraged personalized experiences. For instance, one group designed hats and earbuds that allowed them to add accompaniment to the piano performance that only they could hear. When making changes to the performance as a whole, aggregate feedback was generally sought to ensure everybody's desires would be addressed. For instance, children designed a button that changed the music based on audience members' overall mood.
Multimodal Interactions: In study 2, groups created multimodal interactions with which to engage in the performance. For instance, groups proposed using hand gestures to change sound effects, throwing tokens onstage to get better seats, clapping physical blocks to control volume, and waving LED pompoms to conduct the musicians. Groups also suggested using tactile feedback, throwing colored paintballs at the stage, or using paper planes to send messages to each other.
Multisensory and 4D Experiences: All groups designed multisensory experiences into performances. For instance, one group wanted to control wind gusts to lift a bride's veil, another wanted to tickle a performer with remote controlled feathers, and a third wanted to graffiti a performer's outfit with a cellphone. Groups also wanted to influence their own multisensory experience; one group created a game played with performers where they could fall into ice water, and another had performers influence the type of food audience members had available during the show.
Rules for Democratization and Fairness: The team strongly considered issues of fairness and democratization in their designs for interactive participation--either offering everyone a chance to participate, or including everyone and allowing individuals to opt out. In the democratized experience, the group proposed using a video camera that allowed audience members to play air guitar with the onstage performers from their seat. Similarly, in the fair experience, children sat in a line and took turns going onstage. While it was recognized that this dynamic created peer pressure on audience members, the team suggested this might keep audiences more engaged because they would be “constantly on [their] toes.”
Distinct Spaces: Throughout the many designs there were clear boundaries between the audience and performance spaces. Audiences were either interacting from their seat, or distinctly transitioning from audience member to performer by going on stage. For instance, the group suggested that audience members could find out that they'd been selected by having their seat buzz, then go to a different room to become the performer, or could disappear into the group and reappear on stage. Interestingly, the group also emphasized a boundary between their role as an audience member or a performer; rather than participating throughout the performance, groups wanted audience members to enjoy the performance passively at times.
A Spectrum of Audience Interactivity
Findings from these CI sessions informed a Spectrum of Audience Interactivity in musical performances. The children’s arrangements of ideas in session 3 were used to develop an initial spectrum. Session data on children’s designs (e.g., artifact photos, notes, Big Ideas) were coded into this spectrum, resulting in refined level descriptions and the addition of a new level that distinguished between influencing and augmenting performances. The final spectrum represents a preliminary view into how children envision audience participation in live music performances on a scale of increasing interactivity.
Based on these co-design sessions, I conducted an in-depth literature review across theater, theme parks, and games to validate the spectrum across theater, theme park, and game domains. The spectrum can be used by researchers and designers to consider interaction on a high-level during the design process, and to discuss and compare the interaction of different types of transmedia audience experiences.
