Opera of the Future
Extending expression, learning, and health through innovations in musical composition, performance, and participation.
The Opera of the Future group (also known as Hyperinstruments) explores concepts and techniques to help advance the future of musical composition, performance, learning, and expression. Through the design of new interfaces for both professional virtuosi and amateur music-lovers, the development of new techniques for interpreting and mapping expressive gesture, and the application of these technologies to innovative compositions and experiences, we seek to enhance music as a performance art, and to develop its transformative power as counterpoint to our everyday lives. The scope of our research includes musical instrument design, concepts for new performance spaces, interactive touring and permanent installations, and "music toys." It ranges from extensions of traditional forms to radical departures, such as the Brain Opera, Toy Symphony and Death and the Powers.

Research Projects

  • City Symphonies: Massive Musical Collaboration

    Tod Machover, Akito Van Troyer, Benjamin Bloomberg, Charles Holbrow, David Nunez, Simone Ovsey, Sarah Platte, Peter Alexander Torpey and Garrett Parrish

    Thus far, the impact of crowdsourced and interactive music projects has been limited: the public typically contributes only a small part of the final musical result, and is often disconnected from the artist leading the project. We believe that a new “musical ecology” is needed for true creative collaboration between experts and amateurs, benefiting both. Toward this goal, we have been creating “city symphonies,” each collaboratively composed with an entire city. We have designed the infrastructure needed to bring together an unprecedented number of people, including a variety of web-based music composition applications, a social media framework, and real-world community-building activities. This process establishes a new model for creating complex collaborations between experts and everyone else. Over the past two years, we premiered city symphonies in Toronto, Edinburgh, and Perth, and we are now developing a symphony with Lucerne, Switzerland.

  • Death and the Powers: Global Interactive Simulcast

    Tod Machover, Peter Torpey, Ben Bloomberg, Elena Jessop, Charles Holbrow, Simone Ovsey, Garrett Parrish, Justin Martinez, and Kevin Nattinger.

    The live global interactive simulcast of the final February 2014 performance of "Death and the Powers" in Dallas made innovative use of satellite broadcast and Internet technologies to expand the boundaries of second-screen experience and interactivity during a live remote performance. In the opera, Simon Powers uploads his mind, memories, and emotions into The System, represented onstage through reactive robotic, visual, and sonic elements. Remote audiences, via simulcast, were treated as part of The System alongside Powers and the operabots. Audiences had an omniscient view of the action of the opera, as presented through the augmented, multi-camera video and surround sound. Multimedia content delivered to mobile devices, through the Powers Live app, privileged remote audiences with perspectives from within The System. Mobile devices also allowed audiences to influence The System by affecting the illumination of the Winspear Opera House’s Moody Foundation Chandelier.

  • Death and the Powers: Redefining Opera

    Tod Machover, Ben Bloomberg, Peter Torpey, Elena Jessop, Bob Hsiung, Akito van Troyer
    "Death and the Powers" is a groundbreaking opera that brings a variety of technological, conceptual, and aesthetic innovations to the theatrical world. Created by Tod Machover (composer), Diane Paulus (director), and Alex McDowell (production designer), the opera uses the techniques of tomorrow to address age-old human concerns of life and legacy. The unique performance environment, including autonomous robots, expressive scenery, new Hyperinstruments, and human actors, blurs the line between animate and inanimate. The opera premiered in Monte Carlo in fall 2010, with additional performances in Boston and Chicago in 2011 and a new production with a global, interactive simulcast in Dallas in February 2014.
  • Disembodied Performance

    Tod Machover, Peter Torpey and Elena Jessop
    Early in the opera "Death and the Powers," the main character, Simon Powers, is subsumed into a technological environment of his own creation. The set comes alive through robotic, visual, and sonic elements that allow the actor to extend his range and influence across the stage in unique and dynamic ways. This environment assumes the behavior and expression of the absent Simon; to distill the essence of this character, we recover performance parameters in real time from physiological sensors, voice, and vision systems. Gesture and performance parameters are then mapped to a visual language that allows the off-stage actor to express emotion and interact with others on stage. To accomplish this, we developed a suite of innovative analysis, mapping, and rendering software systems.
  • Figments

    Peter Torpey

    Figments is a theatrical performance that tells a story inspired by a variety of source texts, including Dante Alighieri's prosimetrum La Vita Nuova. Framed by a woman's accidental discovery of the compelling journals of the Dante-archetype, three inner vignettes reveal the timeless tribulations of the memoir's author(s). Figments was created using Media Scores, a framework in development to facilitate the composition of Gesamtkunstwerk using parametric score-like visual notation. The Media Score for Figments is realized in this production through the performance of actors, light, visuals, and the generation of musical accompaniment in response to the expressive qualities represented in the score. The score served as a reference during the creation and design of the piece, a guide during rehearsals, and as show control for the final production.

  • Gestural Media Framework

    Tod Machover and Elena Jessop
    We are all equipped with two extremely expressive instruments for performance: the body and the voice. By using computer systems to sense and analyze human movement and voices, artists can take advantage of technology to augment the body's communicative powers. However, the sophistication, emotional content, and variety of expression possible through original physical channels is often not captured by the technologies used for analyzing them, and thus cannot be intuitively transferred from body to digital media. To address these issues, we are developing systems that use machine learning to map continuous input data, whether of gesture or voice, to a space of expressive, qualitative parameters. We are also developing a new framework for expressive performance augmentation, allowing users to create clear, intuitive, and comprehensible mappings by using high-level qualitative movement descriptions, rather than low-level descriptions of sensor data streams.
  • Hyperinstruments

    Tod Machover
    The Hyperinstruments project creates expanded musical instruments and uses technology to give extra power and finesse to virtuosic performers. They were designed to augment a wide range of traditional musical instruments and have been used by some of the world's foremost performers (Yo-Yo Ma, the Los Angeles Philharmonic, Peter Gabriel, and Penn & Teller). Research focuses on designing computer systems that measure and interpret human expression and feeling, exploring appropriate modalities and content of interactive art and entertainment environments, and building sophisticated interactive musical instruments for non-professional musicians, students, music lovers, and the general public. Recent projects involve both new hyperinstruments for children and amateurs, and high-end hyperinstruments capable of expanding and transforming a symphony orchestra or an entire opera stage.
  • Hyperproduction: Advanced Production Systems

    Tod Machover and Benjamin Bloomberg

    Hyperproduction is a conceptual framework and a software toolkit which allows producers to specify a descriptive computational model and consequently an abstract state for a live experience through traditional operating paradigms such as mixing audio or operation of lighting, sound, and video systems. The hyperproduction system is able to interpret this universal state and automatically utilize additional production systems, allowing for a small number of producers to cohesively guide the attention and perspective of an audience using many or very complex production systems simultaneously. We focus on exploring the relationship of conventional production systems and techniques to computational abstract models of live performance, with attention and perspective as the cornerstones of this exploration.

  • Hyperscore

    Tod Machover
    Hyperscore is an application to introduce children and non-musicians to musical composition and creativity in an intuitive and dynamic way. The "narrative" of a composition is expressed as a line-gesture, and the texture and shape of this line are analyzed to derive a pattern of tension-release, simplicity-complexity, and variable harmonization. The child creates or selects individual musical fragments in the form of chords or melodic motives, and layers them onto the narrative-line with expressive brushstokes. The Hyperscore system automatically realizes a full composition from a graphical representation. Currently, Hyperscore uses a mouse-based interface; the final version will support freehand drawing, and integration with the Music Shapers and Beatbugs to provide a rich array of tactile tools for manipulation of the graphical score.
  • Media Scores

    Tod Machover and Peter Torpey

    Media Scores extends the concept of a musical score to other modalities, facilitating the process of authoring and performing multimedia compositions and providing a medium through which to realize a modern-day Gesamtkunstwerk. The web-based Media Scores environment and related show control systems leverages research into multimodal representation and encoding of expressive intent. Using such a tool, the composer will be able to shape an artistic work that may be performed through a variety of media and modalities. Media Scores offer the potential for authoring content considering live performance data as well as audience participation and interaction. This paradigm bridges the extremes of the continuum from composition to performance, allowing for improvisation. The Media Score also provides a common point of reference in collaborative productions as well as the infrastructure for real-time control of technologies used during live performance.

  • Personal Opera

    Tod Machover and Peter Torpey
    Personal Opera is a radically innovative creative environment that enables anyone to create musical masterpieces sharing personal thoughts, feelings, and memories. Based on our design of, and experience with, such projects as Hyperscore and the Brain Opera, we are developing a totally new environment to allow the incorporation of personal stories, images, and both original and well-loved music and sounds. Personal Opera builds on our guiding principle that active music creation yields far more powerful benefits than passive listening. Using music as the through-line for assembling and conveying our own individual legacies, Personal Opera represents a new form of expressive archiving; easy to use and powerful to experience. In partnership with the Royal Opera House in London, we have begun conducting Personal Opera workshops specifically targeting seniors to help them tell their own meaningful stories through music, text, visuals, and acting.
  • Powers Sensor Chair

    Elena Jessop and Tod Machover

    The Powers Sensor Chair gives visitors a special glimpse into Tod Machover’s robotic opera "Death and the Powers," by providing a new way to explore the sonic world of the opera. A solo participant sitting in a chair discovers that when she moves her hands and arms, the air in front of her becomes an instrument. With a small, delicate gesture, a sharp and energetic thrust of her hand, or a smooth caress of the space around her, she can use her expressive movement and gesture to play with and sculpt a rich sound environment drawn from the opera, including vocal outbursts and murmurs and the sounds of the show’s special Hyperinstruments. This installation explores the body as a subtle and powerful instrument, providing continuous control of continuous expression, and incorporates Elena Jessop’s high-level analysis frameworks for recognition and extension of expressive movement.

  • Remote Theatrical Immersion: Extending "Sleep No More"

    Tod Machover, Punchdrunk, Akito Van Troyer, Ben Bloomberg, Gershon Dublon, Jason Haas, Elena Jessop, Brian Mayton, Eyal Shahar, Jie Qi, Nicholas Joliat, and Peter Torpey

    We have collaborated with London-based theater group Punchdrunk to create an online platform connected to their NYC show, Sleep No More. In the live show, masked audience members explore and interact with a rich environment, discovering their own narrative pathways. We have developed an online companion world to this real-life experience, through which online participants partner with live audience members to explore the interactive, immersive show together. Pushing the current capabilities of web standards and wireless communications technologies, the system delivers personalized multimedia content, allowing each online participant to have a unique experience co-created in real time by his own actions and those of his onsite partner. This project explores original ways of fostering meaningful relationships between online and onsite audience members, enhancing the experiences of both through the affordances that exist only at the intersection of the real and the virtual worlds.

  • Using the Voice As a Tool for Self-Reflection

    Tod Machover and Rebecca Kleinberger

    Our voice is an important part of our individuality. From the voices of others, we understand a wealth of non-linguistic information, such as identity, social-cultural clues, and emotional state. But the relationship we have with our own voice is less obvious. We don’t hear it the way others do, and our brain treats it differently from any other sound. Yet its sonority is deeply connected with how we are perceived by society and how we see ourselves–body and mind. This project is composed of software, devices, installations, and thoughts used to challenge us to gain new insights on our voices. To increase self-awareness we propose different ways to extend, project, and visualize the voice. We show how our voices sometimes escape our control, and we explore the consequences in terms of self-reflection, cognitive processes, therapy, affective features visualization, and communication improvement.

  • Vocal Vibrations: Expressive Performance for Body-Mind Wellbeing

    Special Interest group(s): 
    Tod Machover, Charles Holbrow, Elena Jessop, Rebecca Kleinberger, Le Laboratoire, and the Dalai Lama Center at MIT

    Vocal Vibrations explores the relationships between human physiology and the vibrations of the voice. The voice is an instrument everyone possesses—it is incredibly individual, expressive, and intimately linked to the physical form. In collaboration with Le Laboratoire in Paris and the Dalai Lama Center at MIT, we are examining the hypothesis that the singing voice can influence mental and physical health through physicochemical phenomena. We are developing a series of multimedia experiences, from solo “meditations” to group “singing circles,” that explore possible emotional, cognitive, and physical transformations, all in a context of immersive music. In March 2014, we premiered a vocal art installation in Paris where a public “Chapel” space encourages careful and meditative listening, and a private "Cocoon” environment guides an individual to explore his or her voice and its vibrations, augmented by tactile and acoustic stimuli.