Opera of the Future

"Death and the Powers" is a groundbreaking opera that brings a variety of technological, conceptual, and aesthetic innovations to the theatrical world. Created by Tod Machover (composer), Diane Paulus (director), and Alex McDowell (production designer), the opera uses the techniques of tomorrow to address age-old human concerns of life and legacy. The unique performance environment, including autonomous robots, expressive scenery, new Hyperinstruments, and human actors, will blur the line between animate and inanimate. The opera will open in Monte-Carlo in fall 2010 and then tour worldwide.

Early in the opera "Death and the Powers," the main character, Simon Powers, is subsumed into a technological environment of his own creation. The theatrical set comes alive through robotic, visual, and sonic elements that allow the actor to extend his range and influence across the stage in unique and dynamic ways. This environment must compellingly assume the behavior and expression of the absent Simon. In order to distill the essence of this character, we recover performance parameters in real time from physiological sensors, voice, and vision systems. These gesture and performance parameters are then mapped to a visual language that allows the off-stage actor to express emotion and interact with others on stage. Additionally, we use these gestural performance parameters for vocal manipulation. Our approach takes a new direction in augmented performance by employing a non-representational abstraction of a human presence that fully translates a character into an environment.

We are developing the first multimodal interface for hand rehabilitation following stroke. EMG forearm sensors read attempted finger presses in disordered limbs, and serve as an input to an expressive feedback interface. Auditory, visual, and tactile cues are presented to support rehabilitation of the representation of finger movements across sensory domains. The multisensory feedback is embedded in a rich task, situated between piano learning and expressive music performance. A user of this system will rehabilitate finger movement while developing an expressive music performance. Imagine a complete shift in the form and function of rehabilitation, towards something empowering, where individuals strive in tandem with tailored interfaces, mapped to push them forward at each step, and as part of fundamentally enriching expressive tasks. Our rehabilitative health care environments can sculpt our minds, while changing our lives, if we invent the right tools.

Many performance artists and interaction designers use human gestures to drive, manipulate, or generate digital media. However, the existing systems for developing mappings between incoming data streams and output media have extremely low-level concepts of “gesture,” forcing the user to focus on the particulars of input sensor or video data, rather than on meaningful and expressive gestures. We are developing a new framework for gestural control of media in performance, allowing users to easily create clear, intuitive, and comprehensible mappings by using high-level descriptions of gestures and of gestural qualities. This system currently is realized in an initial proof-of-concept tool for gestural media manipulation, mapping a simple gestural vocabulary to parameters of interactive visual applications.

The Hyperinstrument project creates expanded musical instruments and uses technology to give extra power and finesse to virtuosic performers. They were designed to augment a wide range of traditional musical instruments and have been used by some of the world's foremost performers (Yo-Yo Ma, the Los Angeles Philharmonic, Peter Gabriel, and Penn & Teller). Research focuses on designing computer systems that measure and interpret human expression and feeling, exploring appropriate modalities and content of interactive art and entertainment environments, and building sophisticated interactive musical instruments for non-professional musicians, students, music lovers, and the general public. Recent projects involve both new hyperinstruments for children and amateurs, and high-end hyperinstruments capable of expanding and transforming a symphony orchestra or an entire opera stage.

Hyperscore is an application to introduce children and non-musicians to musical composition and creativity in an intuitive and dynamic way. The "narrative" of a composition is expressed as a line-gesture, and the texture and shape of this line are analyzed to derive a pattern of tension-release, simplicity-complexity, and variable harmonization. The child creates or selects individual musical fragments in the form of chords or melodic motives, and layers them onto the narrative-line with expressive brushstokes. The Hyperscore system automatically realizes a full composition from a graphical representation, allowing individuals with no musical training to create professional pieces. Currently, Hyperscore uses a mouse-based interface; the final version will support freehand drawing, and integration with the Music Shapers and Beatbugs to provide a rich array of tactile tools for manipulation of the graphical score.

Mei-Mei is a six-legged walking robot that is being developed as a set piece for "Death and the Powers." Its design was inspired by the Theo Jansen mechanism in which a rotation movement is converted to a walking pattern through various four-bar linkages. With six legs driven by two separate motors, this unusual robot can move forward, backward, and even turn around with differential steering control. Mei-Mei, as a combination of artistic and engineering efforts, is a poetic moving sculpture that will coordinate with narrative and musical materials in the opera.

The scientific community is making marked progress in the area of Alzheimer's disease (AD) treatment: memory-related pharmaceuticals are available, the neurobiology of AD is fairly well understood, and the genetic underpinnings of the disease continue to be unraveled. However, despite these advances, it has been shown that individuals often present the symptoms of AD years before they seek a diagnosis. The barrier to treatment is the lack of structure with which to obtain a diagnosis or even predict the onset of disease in a stigmatized environment. With technology, we can build clinically valid assessment into the tools we use every day—the tools we care about. We are developing music tools to detect cognitive performance in the memory domains at risk of decline in the earliest stages of AD. These tools are mobile, longitudinal, and the patient is the first point of feedback.

Our work in Music, Mind, and Health continues to push forward, showing the technologies and perspectives required to build on the transformative nature of music to drive specific neurological, physical, emotional, and psychological change in the clinical setting and for the general public. A radically new "Personal Instrument" is currently being used by Dan Ellsey, a quadraplegic individual, who controls this interface to sculpt an expressive performance of music in real time. A three-month study of Ellsey's expressive behavior—its potential as well as its limits—resulted in an interface tailored just for him, enabling him to access expressive performance despite his physical disability. This new line of work highlights principles for future instruments and applications, where the impact is in the marriage of the interface and uniqueness of the person. In this way, we are pursuing new design philosophies, technologies, and collaborations within the scientific community, public performance, and clinical research.

Robots and performers make beautiful music together. The opera "Death and the Powers" will feature a chorus of seven-foot tall, autonomous, polymorphic robots which at various times function as characters, set pieces, and lighting elements. Using state-of-the-art control electronics and a novel real-time performance recovery protocol, a total of 16 individually addressable BobBots will reflect on, participate in, and illuminate the action onstage.

We are studying and implementing a swarm of 36 holonomic-drive robots for use in an upcoming robotic opera. Each robot will eventually comprise roughly eight degrees of freedom, and will follow a centralized control, allowing swarm behaviors as well as pre-scripted paths. In May, a test platform of three 3-DOF robots will be dancing, controlled by an animation with music composed by Tod Machover.

Personal Opera is a radically innovative creative environment that enables anyone to create musical masterpieces sharing one’s deepest thoughts, feelings, and memories. Based on our design of, and experience with, such projects as Hyperscore and the Brain Opera, we are developing a totally new environment to allow the incorporation of personal stories, images, and both original and well-loved music and sounds. This development is based on two guiding principles: first, that active music creation yields far more powerful benefits than passive listening; and second, that increasing customization of the musical experience is both desirable and possible, as evidenced in our group’s development of Personal Instruments (see Music, Mind, and Health) and Personal Music. Personal Opera goes a step further, using music as the medium for assembling and conveying our own individual legacies, representing a new form of archiving, easy to use and powerful to experience.

Skellig is an opera with music by Tod Machover and a libretto based on the best-selling novel for young people by David Almond. It premiered in the UK in November 2008. Besides blending acoustics and electronics, natural noise, and soaring melodies, Skellig also presents several live performance breakthroughs. A non-professional teenage chorus is used throughout, blended seamlessly with high-level professionals; this chorus is guided by an interactive "sonic score" that provides auditory cues, textures to imitate, and electronic reinforcement for the entire 100-minute show. In addition, specially designed "ambisonics" were developed to allow sound to emanate from the stage and engulf the audience in all dimensions, the first time such a technique has been used in a full-scale theatrical setting.

The Chandelier is a large-scale robotic musical instrument that is being developed for "Death and the Powers." Its 48 strings can be actuated both through powerful electromagnets, and tactilely (plucked like a harp or bowed like a cello). With the strings driven by electromagnets, the tactile player can also repeatedly damp strings or create overtones by carefully touching the strings' anti-nodes, creating a new intimacy between players, who play not just the same instrument, but the same strings. The Chandelier is composed of many systems—logic for control of music and lighting, networked servers, and playable interfaces—all built around an elegant, articulated skeletal structure which allows changes to the length, angle, and tensions of the strings. We are currently experimenting with playing it through new types of interfaces to take advantage of its unusual tuning and sonorities.

Toy Symphony combines children, virtuosic soloists, composers, and symphony orchestras around the world to alter radically how children are introduced to music, as well as to redefine the relationship between professional musicians and young people. A complete set of Music Toys will be distributed to children in each host city (including Berlin, Dublin, Glasgow, Manchester/London, and Tokyo), where children will be mentored to create their own sounds and compositions for toys and traditional instruments. A pedagogy for using these Music Toys to teach and to instill a love for musical creativity will also be developed. Final concerts will be presented in each host city including children's compositions and specially commissioned works by young composers, to be performed by children, soloists, and orchestra, playing Music Toys, Hyperinstruments, and traditional instruments.

The Vocal Augmentation and Manipulation Prosthesis (VAMP) is a gesture-based, wearable controller for live-time vocal performance. This controller allows a singer to capture and manipulate single notes that she sings, using a gestural vocabulary developed from that of choral conducting. By drawing from a familiar gestural vocabulary, this controller and the associated mappings can be more intuitive to both performer and audience. This instrument will be used in the upcoming opera Death and the Powers by the character of Nicholas.