Advances in building technology and sensor networks offer a chance to imagine new forms of personalized and efficient utility control. One such area is lighting control. With the aid of sensor networks, these new control systems not only offer lower energy consumption, but also enable new ways to specify and augment lighting. It is our belief that dynamic lighting controlled by a single user, or even an entire office floor, is the frontier of future intelligent and adaptive systems.

We developed a music control surface which enables integration between any musical instruments via a versatile, customizable, and inexpensive user interface. This sensate surface allows capacitive sensor electrodes and connections between electronics components to be printed onto a large roll of flexible substrate unrestricted in length. The high dynamic range capacitive sensing electrodes can not only infer touch, but near-range, non-contact gestural nuance in a music performance. With this sensate surface, users can “cut” out their desired shapes, “paste” the number of inputs, and customize their controller interfaces, which can then send signals wirelessly to effects or software synthesizers. We seek to find a solution for integrating the form factor of traditional music controllers seamlessly on top of one’s instrument while adding expressiveness to performance by sensing and incorporating movements and gestures to manipulate the musical output.

We are working with sponsor Schneider Electric to deploying a dense, low-power wireless sensor network aimed at environmental monitoring for smart energy profiling. This distributed sensor system measures temperature, humidity, and 3D airflow, and transmits this information through a wireless Zigbee protocol. These sensing units are currently deployed in the lower atrium of E14. The data is being used to inform CFD models of airflow in buildings, explore and retrieve valuable information regarding the efficiency of commercial building HVAC systems, energy efficiency of different building materials, and lighting choices in novel architectural designs.

Homes and offices are being filled with sensor networks to answer specific queries and solve pre-determined problems, but no comprehensive visualization tools exist for fusing these disparate data to examine relationships across spaces and sensing modalities. DoppelLab is a cross-reality virtual environment that represents the multimodal sensor data produced by a building and its inhabitants. Our system encompasses a set of tools for parsing, databasing, visualizing, and sonifying these data; by organizing data by the space from which they originate, DoppelLab provides a platform to make both broad and specific queries about the activities, systems, and relationships in a complex, sensor-rich environment.

Expressive musical re-performance is about enabling a person to experience the creative aspects of a playing a favorite song regardless of technical expertise. This is done by providing users with computer-linked electronic instruments that distills the instruments' interface but still allows them to provide expressive gesture. The next note in an audio source is triggered on the instrument, with the computer providing correctly pitched audio and mapping the expressive content onto it. Thus, the physicality of the instrument remains, but requires far less technique. We are implementing an expressive re-performance system using commercially available, expressive electronic musical instruments and an actual recording as the basis for deriving audio. Performers will be able to select a voice within the recording and re-perform the song with the targeted line subject to their own creative and expressive impulse.

The FreeD is a hand-held, digitally controlled, milling device that is guided and monitored by a computer while still preserving the craftsperson's freedom to sculpt and carve. The computer will intervene only when the milling bit approaches the planned model. Its interaction is either by slowing down the spindle speed or by drawing back the shaft; the rest of the time it allows complete freedom, letting the user to manipulate and shape the work in any creative way.

The Gesture Recognition Toolkit (GRT) is a cross-platform, open-source, c++ machine-learning library that has been specifically designed for real-time gesture recognition. The GRT has been created as a general-purpose tool for allowing programmers with little or no machine-learning experience to develop their own machine-learning based recognition systems, through just a few lines of code. Further, the GRT is designed to enable machine-learning experts to precisely customize their own recognition systems, and easily incorporate their own algorithms within the GRT framework. In addition to facilitating developers to quickly create their own gesture-recognition systems, the machine-learning algorithms at the core of the GRT have been designed to be rapidly trained with a limited number of training examples for each gesture. The GRT therefore allows a more diverse group of users to easily integrate gesture recognition into their own projects.

The tongue is known to have an extremely dense sensing resolution, as well as an extraordinary degree of neuroplasticity, the ability to adapt to and internalize new input. Research has shown that electro-tactile tongue displays paired with cameras can be used as vision prosthetics for the blind or visually impaired; users quickly learn to read and navigate through natural environments, and many describe the signals as an innate sense. However, existing displays are expensive and difficult to adapt. Tongueduino is an inexpensive, vinyl-cut tongue display designed to interface with many types of sensors besides cameras. Connected to a magnetometer, for example, the system provides a user with an internal sense of direction, like a migratory bird. Plugged into weighted piezo whiskers, a user can sense orientation, wind, and the lightest touch. Through tongueduino, we hope to bring electro-tactile sensory substitution beyond vision replacement, towards open-ended sensory augmentation.

Extending the Living Observatory installation, we have instrumented the roots of several trees outside of E15 with vibratory transducers that excite the trees with live streaming sound from a forest near Plymouth, MA. Walking though the trees just outside the Lab, you won't notice anything, but press your ear up against one of them and you'll feel vibrations and hear sound from a tree 60 miles away. Visit at any time from dawn till dusk and again after midnight; if you’re lucky you might just catch an April storm, a flock of birds, or an army of frogs.

Light enables our visual perception. It is the most common medium for displaying digital information. Light regulates our circadian rhythms, affects productivity and social interaction, and makes people feel safe. Yet despite the significance of light in structuring human relationships with their environments on all these levels, we communicate very little with our artificial lighting systems. Occupancy, ambient illuminance, intensity, and color preferences are the only input signals currently provided to these systems. With advanced sensing technology, we can establish better communication to our devices. This effort is often described as context-awareness. Context has typically been divided into properties such as location, identity, affective state, and activity. Using wearable and infrastructure sensors, we are interested in detecting these properties and using them to control lighting. The Mindful Photons Project aims to close the loop and allow our light sources to “see" us.

We are developing an opt-in camera network, in which users carrying wearable tags are visible to the network and everyone else is invisible. Existing systems for configurable dynamic privacy in this context are opt-out and catch-all; users desiring privacy carry pre-registered tags that disable sensing and networked media services for everyone in the room. To address these issues, we separate video into layers of flexible sprites representing each person in the field of view, and transmit video of only those who opt-in. Our system can also define groups of users who can be dialed in and out of the video stream dynamically. For cross-reality applications, these dynamic layers achieve a new level of video granularity, allowing users and groups to uncover correspondences between their activities across spaces.

Sensor networks permeate our built and natural environments, but our means for interfacing to the resultant data streams have not evolved much beyond HCI and information visualization. Researchers have long experimented with wearable sensors and actuators on the body as assistive devices. A user’s neuroplasticity can, under certain conditions, transcend sensory substitution to enable perceptual-level cognition of “extrasensory” stimuli delivered through existing sensory channels. But there remains a huge gap between data and human sensory experience. We are exploring the space between sensor networks and human augmentation, in which distributed sensors become sensory prostheses. In contrast, user interfaces are substantially unincorporated by the body—our relationship to them never fully pre-attentive. Attention and proprioception are key, not only to moderate and direct stimuli, but also to enable users to move through the world naturally, attending to the sensory modalities relevant to their specific contexts.

Increasing numbers of networked appliances are bringing about new opportunities for control and automation. At the same time, an increase in multifunctional appliances is creating a complex and often frustrating environment for the end-user. Motivated by these opportunities and challenges, we are exploring the potential for sensor fusion to increase usability and improve user experience while retaining the user in the control loop. We have developed a novel, camera-less, multi-sensor solution for intuitive gesture-based indoor lighting control, called RElight. Using a wireless handheld device, the user simply points at a light fixture to select it and rotates his hand to continuously configure the dimming level. Pointing is a universal gesture that communicates one’s interest in or attention to an object. Advanced machine learning algorithms allow rapid training of gestures and continuous control that supplements gesture classification.

Current sports-medicine practices for understanding the motion of athletes while engaged in their sport of choice are limited to camera-based marker tracking systems that generally lack the fidelity and sampling rates necessary to make medically usable measurements; they also typically require a structured, stable "studio" environment, and need considerable time to set up and calibrate. The data from our system provides the ability to understand the forces and torques that an athlete's joints and body segments undergo during activity. It also allows for precise biomechanical modeling of an athlete's motion. The application of sensor fusion techniques is essential for optimal extraction of kinetic and kinematic information. Also, it provides an alternative measurement method that can be used in out-of-lab scenarios.

We are developing a system for inferring safety context on construction sites by fusing data from wearable devices, distributed sensing infrastructure, and video. Wearable sensors stream real-time levels of dangerous gases, dust, noise, light quality, precise altitude, and motion to base stations that synchronize the mobile devices, monitor the environment, and capture video. Context mined from these data is used to highlight salient elements in the video stream for monitoring and decision support in a control room. We tested our system in a initial user study on a construction site, instrumenting a small number of steel workers and collecting data. A recently completed hardware revision will be followed by further user testing and interface development.

This project is a system of compact, wearable, wireless sensor nodes, equipped with full six-degree-of-freedom inertial measurement units and node-to-node capacitive proximity sensing. A high-bandwidth, channel-shared RF protocol has been developed to acquire data from many (e.g., 25) of these sensors at 100 Hz full-state update rates, and software is being developed to fuse this data into a compact set of descriptive parameters in real time. A base station and central computer clock the network and process received data. We aim to capture and analyze the physical movements of multiple people in real time, using unobtrusive sensors worn on the body. Applications abound in biomotion analysis, sports medicine, health monitoring, interactive exercise, immersive gaming, and interactive dance ensemble performance.