Generating music through movement
Experimental interaction project
Designer & engineer
Context
How could a dancer create music with their movement?
Professional dancers can spend hours trying to find the right song for their choreography. Instead, dancer were finding it easier to just make their own music and many have been learning how to produce music. When I learned about this, I started to wonder… what if a dancer's choreography could create the music?
Research
First, how could I track a dancer's movement?
PoseNet uses computer vision to detect human figures in images/videos so that we can determine where a body point may be.
PoseNet run on TensorFlow.js, which means anyone (like me) can run this technology from a web browser.
Next, what are live music production tools I can take inspiration from?
I saw that Midi boards were a simple tool for live beat creation.
In this board, each button triggers an uploaded sound. Musicians will hit the buttons in different patterns to create live music.
Design call out: This midi board leverages a grid which makes buttons easy to recall
Design
Combining PoseNet & MidiBoards
To capture the dancer's movement, I set up a Kinect. I then used JavaScript to take the Kinect video feed and place it on a html canvas of the dancer’s body.
Next, I implemented PoseNet to track a body part on the dancer
On the HTML canvas, I created a set of buttons on a grid similar to a Midi board.
I set up simple code to detect if that body part “hit” one of the created buttons on the screen, then the music will play.
I used Ableton Live and sample beats, tying them to different buttons.
Iterate
Optimizing for user control
After set up, came iteration time. In real time, I worked with the dancer to figure out where buttons should be placed.
The grid layout was easy for the dancer to recall.
We experimented with different body parts, but landed on her wrist to be the best.
Test
Live performance
To test, I recruited a dancer to complete a live performance with the tool.
I was looking to see if the dancer was able to successfully use the tool and most importantly, how the audience would react?
After the performance, it was clear the audience found it extremely compelling. Many not even realizing the music was being generated live. Yet, the best part was each performance was unique - revealing how much more control the dancer had her on process.