Research Projects
A Toronto Symphony: Massive Musical Collaboration
Tod Machover and Peter Alexander TorpeyThe results of existing crowd-sourced and interactive music are limited so far, with the public being only a small part of a final musical result, and often disconnected from the artist leading the project. We believe that a new “musical ecology” is needed for true creative collaboration between experts and amateurs that benefits both. For this purpose, we are creating a new work for symphony orchestra in collaboration with the entire city of Toronto. Called “A Toronto Symphony,” the work–commissioned by the Toronto Symphony Orchestra–will be premiered in March 2013. We are designing the necessary infrastructure, creative tools based on Hyperscore, social media framework, and real-world community-building activities to bring together an unprecedented number of people from diverse ages, experiences, and musical backgrounds to create this new work. We also will establish a model for creating complex creative collaborations between experts and everyone else.
Advanced Audio Systems for Live Performance
Tod Machover and Ben BloombergThis project explores the contribution of advanced audio systems to live performance, their design and construction, and their integration into the theatrical design process. We look specifically at innovative input and control systems for shaping the analysis and processing of live performance; and at large-scale output systems which provide a meaningful virtual abstraction to DSP in order to create flexible audio systems that can both adapt to many environments and achieve a consistent and precise sound field for large audiences.
Coda
Eyal Shahar and Stephen BresnickCoda is a collaborative musical-knowledge database in which articles are accessed by selecting graphic entities in a musical score. Coda is meant to serve as the center for a community of learners sharing music-theory knowledge and musical ideas through musical pieces with personal meaning.
CogNotes: Cognitive Assessment in Social Media-Enabled Creativity Tools
Tod Machover and Adam BoulangerWith CogNotes, new music composition tools become platforms for cognitive assessment. CogNotes users engage their memory as they emerge as composers and participants in their own health process. Together with partners the Lincoln Park Performing Arts School and the Yamaha Corporation, a group of seniors are undertaking an extensive multi-month music composition workshop built around Tod Machover's Hyperscore program. The program is outfitted with cognitive measures sensitive to the earliest transition to Alzheimer's Disease, validated as part of post-doc Adam Boulanger's research with Harvard Medical School and the Alzheimer's Association. Disease assessment can be part of your everyday, creative, and rewarding life. You can be the manager of your own health information as part of the activities you love.
Dance, Emotion, and Expression
Tod Machover and Janice WangDance is an expressive activity that combines music and movement. We are interested in both encouraging people to dance, and in measuring the emotional experience of musical expression through movement. The Dance Remixer is a program that transforms any piece of music into something that people can dance to. The program remixes music to add customizable rhythmic elements–for instance, those typical to Latin dance music. The Dance Remixer is a first step toward personalizing how people interact with music, by giving the user the ability to modify its function and emotional content. We are also interested in quantitatively measuring the emotional experience of dancing. What makes dancing enjoyable? How is our enjoyment of music reflected in dance? Specifically, we examine the psychological factors behind Argentine Tango, an improvisational dance that prioritizes the interpretation of music.
Death and the Powers: Redefining Opera
"Death and the Powers" is a groundbreaking opera that brings a variety of technological, conceptual, and aesthetic innovations to the theatrical world. Created by Tod Machover (composer), Diane Paulus (director), and Alex McDowell (production designer), the opera uses the techniques of tomorrow to address age-old human concerns of life and legacy. The unique performance environment, including autonomous robots, expressive scenery, new Hyperinstruments, and human actors, blurs the line between animate and inanimate. The opera premiered in Monte-Carlo in fall 2010, with additional performances in Boston and Chicago in 2011 and continuing engagements worldwide.Tod Machover, Ben Bloomberg, Peter Torpey, Elena Jessop, Bob Hsiung, Michael Miller, Akito van Troyer, and Eyal ShaharDisembodied Performance
Early in the opera "Death and the Powers," the main character Simon Powers is subsumed into a technological environment of his own creation. The set comes alive through robotic, visual, and sonic elements that allow the actor to extend his range and influence across the stage in unique and dynamic ways. This environment must assume the behavior and expression of the absent Simon; to distill the essence of this character, we recover performance parameters in real time from physiological sensors, voice, and vision systems. Gesture and performance parameters are then mapped to a visual language that allows the off-stage actor to express emotion and interact with others on stage. To accomplish this, we developed a suite of innovative analysis, mapping, and rendering software systems. Our approach takes a new direction in augmented performance, employing a non-representational abstraction of a human presence that fully translates a character into an environment.Tod Machover, Peter Torpey and Elena JessopDrumTop
Tod Machover and Akito Oshiro van TroyerThis project aims to transform everyday objects into percussive musical instruments, encouraging people to rediscover their surroundings through musical interactions with the objects around them. DrumTop is a drum machine made up of eight transducers. Placing objects on top of the transducers triggers a "hit," causing sounds to come out from the objects themselves. In addition, users can program drum patterns by pushing on a transducer, and the weight of an object can be measured to control the strength of a “hit.”
Gestural Media Framework
We are all equipped with two extremely expressive instruments for performance: the body and the voice. By using computer systems to sense and analyze human movement and voices, artists can take advantage of technology to augment the body's communicative powers. However, the sophistication, emotional content, and variety of expression possible through the original physical channels is often not captured by or addressed in the technologies used for analyzing them, and thus cannot be transferred from body to digital media. To address these issues, we are developing systems that use machine learning to map continuous input data, whether of gesture or voice or biological/physical states, to a space of expressive, qualitative parameters. We are also developing a new framework for expressive performance augmentation, allowing users to easily create clear, intuitive, and comprehensible mappings by using high-level qualitative movement descriptions, rather than low-level descriptions of sensor data streams.Tod Machover and Elena JessopHyperinstruments
The Hyperinstrument project creates expanded musical instruments and uses technology to give extra power and finesse to virtuosic performers. They were designed to augment a wide range of traditional musical instruments and have been used by some of the world's foremost performers (Yo-Yo Ma, the Los Angeles Philharmonic, Peter Gabriel, and Penn & Teller). Research focuses on designing computer systems that measure and interpret human expression and feeling, exploring appropriate modalities and content of interactive art and entertainment environments, and building sophisticated interactive musical instruments for non-professional musicians, students, music lovers, and the general public. Recent projects involve both new hyperinstruments for children and amateurs, and high-end hyperinstruments capable of expanding and transforming a symphony orchestra or an entire opera stage.Tod MachoverHyperscore
Hyperscore is an application to introduce children and non-musicians to musical composition and creativity in an intuitive and dynamic way. The "narrative" of a composition is expressed as a line-gesture, and the texture and shape of this line are analyzed to derive a pattern of tension-release, simplicity-complexity, and variable harmonization. The child creates or selects individual musical fragments in the form of chords or melodic motives, and layers them onto the narrative-line with expressive brushstokes. The Hyperscore system automatically realizes a full composition from a graphical representation, allowing individuals with no musical training to create professional pieces. Currently, Hyperscore uses a mouse-based interface; the final version will support freehand drawing, and integration with the Music Shapers and Beatbugs to provide a rich array of tactile tools for manipulation of the graphical score.Mary Farbood and Tod MachoverMedia Scores
Tod Machover and Peter TorpeyMedia Scores extends the concept of a musical score to other modalities to facilitate the process of authoring and performing multimedia compositions, providing a medium through which to realize a modern-day Gesamtkunstwerk. Through research into the representation and the encoding of expressive intent, systems for composing with media scores are being developed. Using such a tool, the composer will be able to shape an artistic work that may be performed through human and technological means in a variety of media and modalities. Media scores offer the potential for authoring content considering live performance data and the potential for audience participation and interaction. This paradigm bridges the extremes of the continuum from composition to performance, allowing for improvisatory compositional acts at performance time. The media score also provides a common point of reference in collaborative productions as well as the infrastructure for real-time control of technologies used during live performance.
New Experiences in Music and Food
Tod Machover and Janice WangFood offers a rich multi-modal experience that can deeply affect emotion and memory. We're interested in exploring the artistic and expressive potential of food beyond mere nourishment, as a means of creating memorable experiences that involve multiple senses. For instance, music can change our eating experience by altering our emotions during the meal, or by evoking a specific time and place. In other words, we can use music to set expectations, which can be met or violated for expressive effect. An example of complementary pairing is British chef Heston Blumenthal’s Sound of the Sea dish served with an iPod. In addition, food and music are both art forms that are heavily loaded with personal meaning. By utilizing the power of these mediums, we can create highly individualized sensory experiences.
Personal Opera
Personal Opera is a radically innovative creative environment that enables anyone to create musical masterpieces sharing one’s deepest thoughts, feelings, and memories. Based on our design of, and experience with, such projects as Hyperscore and the Brain Opera, we are developing a totally new environment to allow the incorporation of personal stories, images, and both original and well-loved music and sounds. Personal Opera builds on our guiding principle that active music creation yields far more powerful benefits than passive listening. Using music as the through-line for assembling and conveying our own individual legacies, Personal Opera represents a new form of expressive archiving; easy to use and powerful to experience. In partnership with the Royal Opera House in London, we have begun conducting Personal Opera workshops specifically targeting seniors to help them tell their own meaningful stories through music, text, visuals, and acting.Tod Machover and Peter TorpeyRemote Theatrical Immersion: Extending "Sleep No More"
Tod Machover, Punchdrunk, Akito Van Troyer, Ben Bloomberg, Gershon Dublon, Jason Haas, Elena Jessop, Eyal Shahar, Jie Qi, Nicholas Joliat, and Peter TorpeyWe are collaborating with London-based theater group Punchdrunk to create an online platform connected to their NYC show, Sleep No More. In the live show, masked audience members explore and interact with a rich environment, discovering their own narrative pathways. We are developing an online companion world to this real-life experience, through which online participants partner with live audience members to explore the interactive, immersive show together. Pushing the current capabilities of web standards and wireless communications technologies, the system will deliver personalized multimedia content allowing each online participant to have a unique experience co-created in real time by his own actions and those of his onsite partner. This project explores original ways of fostering meaningful relationships between online and onsite audience members, enhancing the experiences of both through the affordances that exist only at the intersection of the real and the virtual worlds.
SoundStrand
Tod Machover and Eyal ShaharSoundStrand is a music composition toy. It comprises a set of building blocks, each containing a musical motif. The blocks can be connected to each other to create a musical theme. They can also be manipulated with three degrees of freedom: elongation changes the rhythmic distribution of the notes; bending changes the direction of the melody; and twisting changes the harmonic context.
Vocal Vibrations: Expressive Performance for Body-Mind Wellbeing
Tod Machover and Elena JessopThe voice and the body are instruments that everyone possesses–they are incredibly individual, infinitely expressive, and intimately linked to one's own physical form. As such, they can affect us on both emotional and anatomical levels. Building on our recent work on capturing operatic performances for Death and the Powers, we are now creating the next generation of tools for analyzing and enhancing expressive vocal and physical performance. Using sophisticated voice, breath, and movement analysis techniques, we seek to develop compelling artistic experiences that encourage the use of the body and singing voice in ways that are physiologically and emotionally beneficial.
