Music, Mind and Machine

Standard loudspeakers transmit sound which necessarily spreads very quickly, and the control of sound projection and position is only about as flexible as where you can hang a loudspeaker. The Audio Spotlight is a device that will project sound much like a spotlight projects light: shining it at a listener allows only that person to hear it, while shining it at a surface causes the sound to appear to originate from there, creating something of a "virtual loudspeaker." Beamsteering will allow the sound to move, enabling the user dynamically to place sound—exactly, and only, where it is desired.

Musicpainter is a networked, graphical composing environment that encourages sharing and collaboration within the composing process. It provides a social environment where users can gather and learn from each other. The approach is based on sharing and managing music creation in small and large scales. At the small scale, users are encouraged to begin composing by conceiving small musical ideas, such as melodic or rhythmic fragments, all of which are collected and made available to all users as a shared composing resource. The collection provides a dynamic source of composing material that is inspiring and reusable. At the large scale, users can access full compositions that are shared as open projects. Users can listen to and change any piece if they want. The system generates an attribution list on the edited piece and thus allows users to trace how a piece evolves in the environment.

Musicscape is a two-dimensional, spatial music navigation interface designed for browsing large-scale sound archives. It simulates a 2-D sound field by applying Head-Related Transfer Function (HRTF) and enables users to virtually walk around the sound space with a computer mouse.

Sound designers, audio professionals, and musicians often spend time and energy looking for the right sound for a particular piece of music or sonic environment. Current sound synthesizers either contain numerous sound presets that are laborious to parse, or batteries of parameters to tweak without straightforward connections to one's intuitive expectation. We propose a sound retrieval and modification engine based on everyday words like "bright," warm," and "fat." The perceptual sound synthesis engine is informed by a survey of musicians and listeners worldwide and can also be customized. This system allows dynamic tagging of sound material from online libraries, and "sound sculpting" based on common verbal descriptors instead of obscure numerical parameters.

A live music event where musicians are located in different places is possible through the Internet but highly constrained by network latency. This is especially true with rhythmic music that requires tight synchrony, or in situations where musicians are separated by long distances. To overcome time delays we propose an intelligent system that listens to the audio input at one end and synthesizes a predicted audio output at the other end without having to wait for packets to travel across the network. This system enables various types of real-time online musical collaboration.

How many decisions does it take before you hear a desired piece of music on your iPod? First, you are asked to pick a genre, then an artist, then an album, and finally a song. The more songs you own, the tougher the choices are. To resolve the issues, we turn the modern music player into an old analog radio tuner, the Radio-ish Media Player. No LCDs, no favorite channels, all you have is a knob that will help you surf through channel after channel accompanied by synthesized noise. Radio-ish is our attempt to revive the lost art of channel surfing in the old analog radio tuner. Let music find you: your ears will tell you if the music is right. This project is not only a retrospective design, but also our reflection on lost simplicity in the process of digitalization.

Building on well-developed data visualization techniques, audio is used to enhance understanding by improving the brain's ability to parse information in the scientific process. Sonification (aka auditory display) is the use of non-speech audio to convey information. In collaboration with the Media Lab, NASA is interested in using the human auditory system's powers of organizing and deconstructing sound for the purposes of scientific research and exploratory data analysis. Faced with increasingly voluminous and complex multimodal data and computations, researchers are seeking novel ways to optimize the conversion of information into knowledge. Several sonifications are under study, including interactive tours through the solar system, spectral data from Mars, sensor data from the Media Lab, new augmentations of the traditional orrery, and enhanced exploration of the Mandelbrot set. The eventual goal is to generalize these new sonification techniques for the display, exploration, and analysis of any large dataset.

We have created a dataset of sounds by killer whales, which were individually identified visually as they were recorded is being assembled. Speaker identification techniques (such as a Gaussian Mixture Model (GMM), using cepstral coefficients as features) will be applied to determine if individual marine mammals can be identified by their vocalizations alone. If a sufficient number of examples of a single call type can be found, a Hidden Markov Model will be used and compared to the GMM results. The ability to identify marine mammals from their vocalizations alone, in addition to the theoretical interest for production mechanisms, would be extremely valuable in tracking these mammals from remote locations where visual information is not present.