Designing A Music VR Experience: Background and Design Principles

Musical VR Prototype (In Development)

Designing a VR music experience

Background and Design Principles

Over the past six months I’ve been one of the new talent fellows with the South West Creative Technology Network (SWCTN), where we’ve been tasked with exploring ideas and questions on immersion and immersive media. In this series of blog posts, I’ll be discussing how my thoughts around music interaction in immersive technology developed, how I designed and developed my own VR experience, and how I plan to take the project forward. In this post, I’ll be talking about my introduction and research into VR/AR experiences, the design principles I used in my music VR experience, before summarising the research question that forms the focus of this project.

Gestural music with Mi.Mu gloves

I’m currently a PhD student working on digital musical instruments, which is about making musical instruments and facilitating musical expression with digital technology. Specifically, I work with the Mi.Mu Gloves, a pair of gloves that detect your hands’ orientation and postures in space. A piece of dedicated software allows users to connect, or map, this movement data to musical MIDI messages. This process of mapping has become a novel aspect of musical expression for Glove musicians, with artists like Imogen Heap, Chagall and Kris Halpin demonstrating the gloves’ ability to enable musicians to express themselves. My research explores this expressive mapping design process.

Chagall performing with the Mi.Mu Gloves at Ableton Live Loop in 2017 (Video from Ableton)

While I would consider myself a creative technologist, the world of AR/VR and the concepts and practices associated with immersive media were new to me. What initially drew me to AR/VR tech was what I thought they could do for the Mi.Mu Gloves. I had ideas of using it to provide real-time feedback to musicians about the status of their gloves while they were onstage. Over the course of the fellowship, this idea morphed into making an interactive musical experience for VR.  

My first real experience in a VR headset was playing SWCTN fellow Katie Goode’s game Unseen Diplomacy during our first workshop. To begin exploring VR technology on my own, I splashed out on an Oculus Rift and started having a go on some VR content: Epic Games’ Robo Recall, where you ‘recall’ malfunctioning killer robots with a myriad of guns and weaponry; Budget Cuts, a stealth game where you must sneak through the office building you work in to avoid the murderous HR department; and the ultraviolent gladiatorial combat simulator Blades and Sorcery.

Embodied Interaction in VR

While playing these games, I started to focus on the gestural interactions, and how the games were designed to make you use your body and gestures to play the game. For example, Unseen Diplomacy had me crawling around on the floor through vents, ducking beneath lasers and furiously whacking a keyboard to deactivate a techno-virus, while Robo Recall had me reaching over my back to grab shotguns, and grabbing robots to tear them literally limb from limb.

When playing these games, I started thinking about Embodied Interaction –  a human-computer interaction philosophy based on the ideas of Embodied Cognition, which argues that our understanding of ourselves and the world are shaped by our body and physical actions within it. Embodied Interaction uses this philosophy to inform our interactions with computers, by using interactions that are based on those we are used to in our experience in the world. The classical example of Embodied Interaction comes from German philosopher Martin Heidegger, who describes our interaction with a hammer (think back to your last DIY session). When you are using a hammer, you become unaware of the hammer in your hand, you are thinking about hitting the nail. You act through the hammer on the world, and the hammer becomes a part of you (you embody the hammer). If the hammer breaks, you are no longer able to embody it to perform the task you were performing, and you become aware of it, and the hammer becomes a part of the world again.

Another example is your computer keyboard. When you first learned to type, you had to consciously think about the location of each key. Over time, you learn the layout of the keyboard, and you start to embody and act through the keyboard, and you shift your focus to the words you are typing.

Being able to embody a tool is a process of learning and practice. If you’re from an English-speaking country, chances are you have a QWERTY keyboard. If you used a completely different keyboard layout like Dvorak, your embodied experience with QWERTY would be near useless, and you would have to relearn typing. However, embodied experience is transferable, if you changed to a QWERTZ keyboard, where most of the letters remain in the same locations, you would be able to transfer much of your QWERTY embodied experience.

This transferability of embodied experience can be used in digital musical instrument design, where a new instrument is based on the interactions of an existing instrument while also providing musicians with new ways to control sound. A good example of this is the ROLI Seaboard, which allows pianists to transfer their existing embodied piano expertise while adding additional expressive control based on their finger’s location and movements on the piano key.

I like my piano with extra wobble. ROLI’s Seaboard Rise (Video from ROLI)

What got me thinking about Embodied Interaction was the closeness of mapping between the actions carried out by your avatar in the game and the actions you perform with your body. In controller-based game interaction, the actions of the player’s avatar are mapped abstractly to buttons and joysticks. In a game like Dark Souls, hand-to-hand combat is abstracted into a precise timing and rhythm game (or in my case, a panicked button mash). Meanwhile, in Blades and Sorcery, you’re required to perform the physical actions of hand-to-hand combat, and you begin to embody the interaction, and have thought processes like “I’ll block his sword with my shield and then stab him in the head with my spear”. This bodily closeness of mapping, and embodied interaction through performing these physical actions, had me thinking and feeling more immersed in the content. While this differs from the keyboard example in that most people don’t have first-hand expertise in wielding a spear, you do have cultural understanding and knowledge about how to use spears through films, games and the like. This is the same reason why non-musicians can appreciate professional musical performances, as they have cultural experience about how a instrument works and is played.

In summary, the closeness of mapping between a user’s physical actions and the actions of the user’s avatar within immersive content supports an embodied interaction, and leads to more immersive experiences. Next, I looked for how embodied interactions are be supported in music-based VR experiences.

I began by looking at some of the existing music content for VR. SoundStage, now open source, is an experience that allows you to build up synthesisers, loops and tracks using physical representations of digital signal processing elements like loops and filters. Electronauts is a music app that allows you to remix popular tracks in a sci-fi Daft Punkian spaceship environment. What struck me about these experiences was how similar they were to traditional desktop Digital Audio Workstations (DAWs) like Logic Pro and Ableton Live. In both SoundStage and Electronauts, you interact by pressing buttons and moving sliders, which are common User Interaction (UI) elements found in DAWs. While you could there was a closeness of mapping between my actions and the software’s response, these actions mirrored mouse and keyboard interactions, and failed to make use of the gestural capabilities that I had experienced in the VR games. I ended up thinking: why bother doing this in VR? What does this bring to the table that traditional DAWs don’t?

This lead to my first question: How can we design musical interactions in VR that allows for embodied interactions, while making use of the gestural capabilities of VR technology?

Design principles for VR music interaction

To address this I began, in best academic practice, by looking for what someone else has come up with on the subject, which led me to find a set of design principles for making VR musical instruments by researchers at the Multisensory Experience Lab at Aalborg University Copenhagen. These principles have been drawn from both the digital musical instrument field that I’m part of as well as the VR and immersive technology field that I’m being introduced to through the fellowship, and felt like an appropriate set of guidelines.

Design for feedback and mapping

Designers should not only consider sound and visuals, but haptic (sense of touch) and tactile feedback should also be included, which are important for heightening the instrument’s playability.

Reduce latency

Responsiveness and low-latency is an important goal for musical interactions, and can also help to minimise cybersickness.

Prevent cybersickness

Designers must be aware of and avoid inducing motion sickness in its users, mainly caused by sensory conflict.

Make use of existing skills

Designers can leverage the existing instrumental experience of its users, which helps users to quickly grasp and understand the new experience. However, rather than copying existing instruments, VR instruments instead should draw inspiration from existing instruments and adapt them to suit VR.

Consider both natural and ‘magical’ interaction

Designers should make use of natural interactions that ground the experience in the user’s understanding of the world (such as objects abiding by the laws of physics) to aid in familiarity and usability, but should also make use of interactions that leverage VR’s ability to overcome physical limitations.

Create a sense of presence

The sense of ‘being there’ in an immersive VR experience is created through the senses (touch, vision, sound etc.) supported by the system. As such, a VR experience should be designed with the limitations of the system in mind, so that users do not have to rely on unsupported senses.

Represent a player’s body

As well as heightening the sense of presence through virtual body ownership, representing the player’s body in VR provides an important channel of visual feedback.

Make the experience social

An important part of any musical interaction is its social aspect, where many relationships exist: performer / audience, performer / performer, performer / composer, etc. It is important that VR musical instruments allow for such relationships to exist and flourish.

Final research question

I’ve been keeping these principles in mind as I develop my own music VR experience, but the design principles that were most interesting to me were those around leveraging existing skills and the idea of using natural and ‘magical’ interactions. The idea of leveraging existing musical expertise is one that’s growing in popularity in the digital musical instrument world, and ties back into embodied interaction. The ROLI Seaboard, for example, leverages the existing musical expertise of pianists while augmenting it to give musicians additional controls for expression.

The ideas around natural and magical interactions fit with my thinking. Using natural interactions in instrument design is about using the types of interactions we have with traditional musical instruments, drawn from our embodied experiences: hitting a drum makes a sharp, percussive sound; bowing a string makes a long, harmonic sound. If a user were to encounter a drum in a VR environment, they would expect to be able to hit it and hear a similar sound to what they would be used to with a real drum. Magical interactions meanwhile, are about using the abilities of the new technology to create interactions that are impossible in the real world. In our drum example, a magical interaction could be that a user can stretch and distort the drum into impossible shapes in real time. This conceptualisation helped me to finalise the question that I wanted to explore:

How can we design immersive musical experiences that blend natural and magical interactions, make use of the gestural affordances of VR technology and support embodied interaction?

In my next post, I’m going to be talking about how I began to answer this question, by discussing how I designed and developed my own VR music experience. In the meantime, if you’d like to talk about these ideas or if you have any questions or comments find me on twitter @dombrown94.