An image appears. The system offers several possible interpretations, each with a different probability. Its reading stays visibly unsettled. In THE_OPER&, that uncertainty became part of the performance.
I was the machine learning collaborator and researcher on this algorithmic opera, developed at Duke University and performed over five nights in January 2018. The collaboration brought together media artist Bill Seaman, composer John Supko, director Jim Findlay, performers from the Lorelei Ensemble, and visual artist Keith Scretch.
My contribution focused on how a machine's reading of an image could become a theatrical event: something the audience could watch, interpret, and question. I developed the image classification system that helped give the piece its appearance of exploratory thought.
A world that builds and breaks itself
THE_OPER& used opera to speculate about future intelligence, apocalypse, renewal, and survival. A computer system held a collection of video, sound, and poetic text fragments. During each performance, those materials formed a world within the room, then gave way to destruction and disorder before rebuilding. The cycle repeated.
The system had a voice. It narrated the action while a chorus responded to the changing environment. The score moved from minimal, ambient passages into complex, industrial textures, following the world's rise and collapse.
Within that larger work, machine learning provided another way for the system to seem present. Images were being read as they appeared. The audience could see interpretations arriving alongside the visual material, adding a layer of apparent deliberation to a performance already concerned with the behavior of an imagined intelligence.
Giving the classifier something to perform
My first approach was to fine-tune VGG16 on the images used in the piece. The model classified each image, and its category label appeared over the projection in real time.
It worked accurately. It also closed down the image very quickly. A boat received the label “boat”; a mountain received “mountain.” The label settled the interpretation before much could happen around it. That was a useful technical result, but a limited dramatic one.
We moved toward a more open presentation. I used the raw VGG16 model and exposed its classification probabilities across a wide range of categories. An image could now appear alongside several competing readings. The audience saw a distribution of possibilities, with probabilities shifting and competing for attention.
That change became the core of my contribution. A classifier already contains multiple possible interpretations; presenting them makes a very different experience from displaying only its most confident answer. The competing labels created room for association, surprise, and speculation about what the system might be doing.
The piece invited the impression of a system learning and reasoning over time. The mechanism was image classification. Holding those two levels together was central to the work: a recognizable computational process could suggest something much less settled when placed inside an opera.
Three computers, one performance
The wider performance system coordinated visuals, sound, and lighting across three computers using Open Sound Control (OSC). One computer ran Isadora for visual output and Keras for machine learning. Another ran MaxMSP for the compositions. The third controlled lighting.
OSC messages carried events, data, and logic between the platforms. The system was designed so that one computer could initiate the piece and the full two-hour performance could proceed without technicians intervening. My machine learning component sat within that larger collaborative architecture.
Leaving the interpretation open
THE_OPER& made the presentation of a model's output an artistic material. The significant change came from deciding what the audience should be able to see: the unresolved possibilities within a prediction.
Those probabilities acquired another meaning beside live voices, projected worlds, and a system narrating its own apparent consciousness. They offered the audience something to read back into. That exchange between computation and interpretation was the part of the opera I wanted my work to open up.
