Tod Machover and Elena Jessop
We are all equipped with two extremely expressive instruments for performance: the body and the voice. By using computer systems to sense and analyze human movement and voices, artists can take advantage of technology to augment the body's communicative powers. However, the sophistication, emotional content, and variety of expression possible through the original physical channels is often not captured by or addressed in the technologies used for analyzing them, and thus cannot be transferred from body to digital media. To address these issues, we are developing systems that use machine learning to map continuous input data, whether of gesture or voice or biological/physical states, to a space of expressive, qualitative parameters. We are also developing a new framework for expressive performance augmentation, allowing users to easily create clear, intuitive, and comprehensible mappings by using high-level qualitative movement descriptions, rather than low-level descriptions of sensor data streams.