Research Group Projects and Descriptions

The Opera of the Future group (also known as Hyperinstruments) explores concepts and techniques to help advance the future of musical composition, performance, learning, and expression. Through the design of new interfaces for both professional virtuosi and amateur music-lovers, the development of new techniques for interpreting and mapping expressive gesture, and the application of these technologies to innovative compositions and experiences, we seek to enhance music as a performance art, and to develop its transformative power as counterpoint to our everyday lives. The scope of our research includes musical instrument design, concepts for new performance spaces, interactive touring and permanent installations, and "music toys." It ranges from extensions of traditional forms to radical departures, such as the Brain Opera and Toy Symphony.

In the tradition of Brain Opera and Toy Symphony, Death and the Powers is a groundbreaking opera project that aims to innovate in a variety of technological, conceptual, and aesthetic ways. To dramatize the physical environment or "System" coming "alive," Machover, in association with Cynthia Breazeal and production designer Alex McDowell, will create the world’s first robotic stage, as well as numerous autonomous robots that compel the audience to suspend disbelief about the line between animate and inanimate. In addition, an ensemble of physical objects—like The Chandelier—will create a new kind of Hyperorchestra: technologically empowered but richly palpable. In addition, users will be able to create their own version of The System, a new kind of legacy or personal opera. The opera will premiere in Monte-Carlo in fall 2009 and then tour worldwide.

The scientific community is making marked progress in the area of Alzheimer's disease (AD) treatment: memory-related pharmaceuticals are currently on the market, the neurobiology of AD is fairly well understood, and the genetic underpinnings of the disease continue to be unraveled. However, despite these advances it has been shown that individuals often present the symptoms of AD for years before they seek a diagnosis. The barrier to treatment is the lack of structure with which to obtain a diagnosis or even predict the onset of disease. With technology, we can build clinically valid assessment into the tools we use everyday—the tools we care about. Together with the Alzheimer's Association, the Intel Corporation, and McLean Geriatric Hospital, we are developing music tools that can detect cognitive performance in the memory domains at risk of decline in the earliest stages of AD.

By coupling damped semi-acoustic physical objects with virtual percussion acoustics, we have developed highly controllable hybrid digital/acoustic percussion instruments that can respond not just to the velocity of the hit, but also to how they are struck. Scraping, stirring with brushes, and very subtle sounds are all possible, greatly extending the richness of digital percussion systems.

Alumni Contributor(s): Roberto M. Aimi

The Hyperinstrument project creates expanded musical instruments and uses technology to give extra power and finesse to virtuosic performers. They were designed to augment a wide range of traditional musical instruments and have been used by some of the world's foremost performers (Yo-Yo Ma, the Los Angeles Philharmonic, Peter Gabriel, and Penn & Teller). Research focuses on designing computer systems that measure and interpret human expression and feeling, exploring appropriate modalities and content of interactive art and entertainment environments, and building sophisticated interactive musical instruments for non-professional musicians, students, music lovers, and the general public. Recent projects involve both new hyperinstruments for children and amateurs, and high-end hyperinstruments capable of expanding and transforming a symphony orchestra or an entire opera stage.

Alumni Contributor(s): Roberto M. Aimi, Mary Farbood, Ed Hammond, Tristan Jehan, Margaret Orth, Dan Overholt, Egon Pasztor, Joshua Strickon, Gili Weinberg and Diana Young

Hyperscore is an application to introduce children and non-musicians to musical composition and creativity in an intuitive and dynamic way. The "narrative" of a composition is expressed as a line-gesture, and the texture and shape of this line are analyzed to derive a pattern of tension-release, simplicity-complexity, and variable harmonization. The child creates or selects individual musical fragments in the form of chords or melodic motives, and layers them onto the narrative-line with expressive brushstokes. The Hyperscore system automatically realizes a full composition from a graphical representation, allowing individuals with no musical training to create professional pieces. Currently, Hyperscore uses a mouse-based interface; the final version will support freehand drawing, and integration with the Music Shapers and Beatbugs to provide a rich array of tactile tools for manipulation of the graphical score.

Alumni Contributor(s): Mary Farbood, Ed Hammond, Tristan Jehan, Margaret Orth, Dan Overholt, Egon Pasztor, Joshua Strickon, Gili Weinberg and Diana Young

There have been few studies of bowing data from real players, in part due to the difficulty in capturing this information. We have designed an interface to measure bowing parameters produced by real players, while maintaining the portability and playability of a traditional violin bow. This interface, the Hyperbow, consists of a carbon-fiber violin bow with a custom wireless sensing system (with accelerometers, gyroscopes, electric field position and force sensors). The Hyperbow is partnered with a Yamaha Silent Violin SV-200 also augmented with gesture sensors. This playable measurement system is the core component of an experimental setup used to investigate the bowing parameters produced by real violinists. In these investigations, the parameters are recorded with the sound produced during performances of different bowing techniques (détaché, martelé, spiccato). These data can then be analyzed to help understand the various strategies employed by violinists to achieve similar goals in sound production.

Alumni Contributor(s): Diana Young

The Hyperinstruments group is exploring new ways to visualize music both at the song-level and the collection-level. At song-level: SoundSieve is a music visualizer that takes the intrinsic qualities of a musical piece—pitch, time, and timbre—and makes their patterns readily apparent in a visual manner. For example, you can quickly pick out repeating themes, chords, and complexity from the pictures and video. At collection-level: MusicBox focuses on the problem of navigating a large body of music. It aims to help you find music you like, both inside your own collection (to match a particular mood, for example), or from a body of entirely new music. MusicBox visualizes your music collection in space, giving each track a location based on how similar it sounds to other tracks. This new manner of navigation stands in stark contrast to traditional, text-dependent media players like iTunes and Windows Media Player.

Our work in Music, Mind, and Health continues to push forward, showing the technologies and perspectives required to build on the transformative nature of music to drive specific neurological, physical, emotional, and psychological change in the clinical setting and for the general public. A radically new "Personal Instrument" is currently being used by Dan Ellsey, a quadraplegic individual, who controls this interface to sculpt an expressive performance of music in real time. A three-month study of Ellsey's expressive behavior—its potential as well as its limits—resulted in an interface tailored just for him, enabling him to access expressive performance despite his physical disability. This new line of work highlights principles for future instruments and applications, where the impact is in the marriage of the interface and uniqueness of the person. In this way, we are pursuing new design philosophies, technologies, and collaborations within the scientific community, public performance, and clinical research.

MusicBox focuses on the problem of navigating a large body of music. It aims to help you find music you like, both inside your own collection (to match a particular mood, for example), or from a body of entirely new music. MusicBox visualizes your music collection in space, giving each track a location based on how similar it sounds to other tracks. This new manner of navigation stands in stark contrast to traditional, text-dependent media players like iTunes and Windows Media Player.

We are studying and implementing a swarm of 36 holonomic-drive robots for use in an upcoming robotic opera. Each robot will eventually comprise roughly eight degrees of freedom, and will follow a centralized control, allowing swarm behaviors as well as pre-scripted paths. In May, a test platform of three 3-DOF robots will be dancing, controlled by an animation with music composed by Tod Machover.

Personal Opera is a radically innovative creative environment that enables anyone to create musical masterpieces sharing one’s deepest thoughts, feelings, and memories. Based on our design of, and experience with, such projects as Hyperscore and the Brain Opera, we are developing a totally new environment to allow the incorporation of personal stories, images, and both original and well-loved music and sounds. This development is based on two guiding principles: first, that active music creation yields far more powerful benefits than passive listening; and second, that increasing customization of the musical experience is both desirable and possible, as evidenced in our group’s development of Personal Instruments (see Music, Mind, and Health) and Personal Music. Personal Opera goes a step further, using music as the medium for assembling and conveying our own individual legacies, representing a new form of archiving, easy to use and powerful to experience.

SoundSieve is an exciting new way to look at the structure of music, designed to create a visual “fingerprint” that can provide the listener with instant information about any given piece. SoundSieve aims to counter the trend towards overly simplistic, flashy music visualization programs by creating a real-time visualization that is not only informative, but enlightening as to the underlying musical structure of the piece. From any MP3 file, SoundSieve uses simple, intuitive mapping to create a picture and video playback that highlight key audio features. In addition, SoundSieve allows the user to actively examine the piece: to zoom in on portions of interest, adjust playback, and add informative overlays. SoundSieve has recently been used to create customized animations that highlight particular features in a piece (e.g., Tod Machover’s "The System" for the opera “Death and the Powers”).

The Chandelier is a large-scale robotic musical instrument with a dual life: it is both a tool for making expressive textural sounds and a platform for expression through robotic motion. Though originally conceived as a set piece for the opera Death and the Powers, it is being tested in other contexts as well, including public installation. The Chandelier represents a new breed of string instrument, with novel actuators such as electromagnets, rosin wheels, and a mechanism called a weedwacker, as well as traditional actuators like hammers and picks. Its movement and associated light production reacts to its sound, becoming yet another actor in the complex drama.

Toy Symphony combines children, virtuosic soloists, composers, and symphony orchestras around the world to alter radically how children are introduced to music, as well as to redefine the relationship between professional musicians and young people. A complete set of Music Toys will be distributed to children in each host city (including Berlin, Dublin, Glasgow, Manchester/London, and Tokyo), where children will be mentored to create their own sounds and compositions for toys and traditional instruments. A pedagogy for using these Music Toys to teach and to instill a love for musical creativity will also be developed. Final concerts will be presented in each host city including children's compositions and specially commissioned works by young composers, to be performed by children, soloists, and orchestra, playing Music Toys, Hyperinstruments, and traditional instruments.

Alumni Contributor(s): Roberto M. Aimi, Mary Farbood, Tristan Jehan, Ariane Martins, Laird Nolan, Gili Weinberg and Diana Young

The Trainer Piano actively moves your fingers as you learn how to play. Employing active magnetic force below the keys of a working piano, the Trainer Piano provides the user with kinesthetic input that augments their normal motor learning process. By providing a "feel" for what a user is supposed to play, the Trainer Piano minimizes the amount of time necessary to learn new motor patterns and acquire new motor skills. The core idea behind this project lies in the hypothesis that computer-controlled force can be used to teach students how to play an instrument at a faster and more efficient rate than would occur in an unaided learning environment.