A gesture can leave a shape behind. It can also become a score.
For Footwerk, I worked with Alex Murray-Leslie of Chicks on Speed and Penn State's Biomechanics Lab to turn recorded movement into visualizations and sound. The collaboration brought the precision of a biomechanics laboratory into a performance practice built around experimentation with bodies, technology, and music.
The work has been exhibited at the Australian Pavilion at the 56th Venice Biennale International Art Exhibition and at the ArtScience Museum in Marina Bay Sands, Singapore.
The body as source material
We recorded Murray-Leslie's performances with the Biomechanics Lab's motion capture system. The recordings describe movement as a sequence of spatial coordinates: joints and limbs changing position, frame by frame. Gestures that pass quickly in a live performance become material we can examine, translate, and compose with.
That shift opens up a different way of working with choreography. A movement has a visible path, a duration, and a relationship to the movements around it. Those properties can carry through into another medium. The challenge is deciding which relationships to preserve and what a new representation allows you to perceive.
Our work followed two paths from the same recordings: one visual, one sonic.
Drawing with movement
The visualizations were rendered in the lab's software environment. They give movement a form that lasts beyond the instant of the gesture: coordinates become paths, and paths accumulate into a record of the performance. The images make the structure of a movement available to look at, including details that are difficult to isolate while watching the body itself.
We also planned to extend the visual work into dynamic 3D pieces in Processing and physical objects derived from the captured movement. A gesture could eventually become something you could walk around or hold.
Composing with movement
For the sonifications, we mapped distinct movements to sound profiles composed in SuperCollider. The gestures supplied the data that shaped the sonic texture and rhythm of the piece.
The mapping is a compositional decision. It establishes how something the body does becomes something the listener hears. Working from the same movement data as the visualizations connects the two outputs: a performance can be followed through its visible traces and its sonic consequences.
Footwerk gave us a way to carry the specificity of a recorded gesture across media. The biomechanics system captured its detail; visualization and synthesis gave that detail another life in performance.
