Advances in building technology and sensor networks offer a chance to imagine new forms of personalized and efficient utility control. One such area is lighting control. With the aid of sensor networks, these new control systems not only offer lower energy consumption, but also enable new ways to specify and augment lighting. It is our belief that dynamic lighting controlled by a single user, or even an entire office floor, is the frontier of future intelligent and adaptive systems.

We developed a music control surface which enables integration between any musical instruments via a versatile, customizable, and inexpensive user interface. This sensate surface allows capacitive sensor electrodes and connections between electronics components to be printed onto a large roll of flexible substrate unrestricted in length. The high dynamic range capacitive sensing electrodes can not only infer touch, but near-range, non-contact gestural nuance in a music performance. With this sensate surface, users can “cut” out their desired shapes, “paste” the number of inputs, and customize their controller interfaces, which can then send signals wirelessly to effects or software synthesizers. We seek to find a solution for integrating the form factor of traditional music controllers seamlessly on top of one’s instrument while adding expressiveness to performance by sensing and incorporating movements and gestures to manipulate the musical output.

We are working with sponsor Schneider Electric to deploy a dense, low-power wireless sensor network aimed at environmental monitoring for smart energy profiling. This distributed sensor system measures temperature, humidity, and 3D airflow, and transmits this information through a wireless Zigbee protocol. These sensing units are currently deployed in the lower atrium of E14. The data is being used to inform CFD models of airflow in buildings, explore and retrieve valuable information regarding the efficiency of commercial building HVAC systems, energy efficiency of different building materials, and lighting choices in novel architectural designs.

In DoppelLab, we are developing tools that intuitively and scalably represent the rich, multimodal sensor data produced by a building and its inhabitants. Our aims transcend the traditional graphical display, in terms of the richness of data conveyed and the immersiveness of the user experience. To this end, we have incorporated 3D spatialized data sonification into the DoppelLab application, as well as in standalone installations. Currently, we virtually spatialize streams of audio recorded by nodes throughout the physical space. By reversing and shuffling short audio segments, we distill the sound to its ambient essence while protecting occupant privacy. In addition to the sampled audio, our work includes abstract data sonification that conveys multimodal sensor data.

At present, luminous efficacy and cost remain the greatest barriers to broad adoption of LED lighting. However, it is anticipated that within several years, these challenges will be overcome. While we may think our basic lighting needs have been met, this technology offers many more opportunities than just energy efficiency: this research attempts to alter our expectations for lighting and cast aside our assumptions about control and performance. We will introduce new, low-cost sensing modalities that are attuned to human factors such as user context, circadian rhythms, or productivity, and integrate these data with atypical environmental factors to move beyond traditional lux measurements. To research and study these themes, we are focusing on the development of superior color-rendering systems, new power topologies for LED control, and low-cost multimodal sensor networks to monitor the lighting network as well as the environment.

The Gesture Recognition Toolkit (GRT) is a cross-platform, open-source, c++ machine-learning library that has been specifically designed for real-time gesture recognition. The GRT has been created as a general-purpose tool for allowing programmers with little or no machine-learning experience to develop their own machine-learning based recognition systems, through just a few lines of code. Further, the GRT is designed to enable machine-learning experts to precisely customize their own recognition systems, and easily incorporate their own algorithms within the GRT framework. In addition to facilitating developers to quickly create their own gesture-recognition systems, the machine-learning algorithms at the core of the GRT have been designed to be rapidly trained with just a few examples examples for each gesture. The GRT therefore allows a more diverse group of users to easily integrate gesture recognition into their own projects.

We want to help people in nations where electric power is scarce to sell power to their neighbors. We’re designing a piece of prototype hardware that plugs into a diesel generator or other power source, distributes the power to multiple outlets, monitors how much power is used, and uses mobile payments to charge the customer for the power consumed.

In the Living Observatory installation at the Other Festival, we invite participants into a transductive encounter with a wetland environment in flux. Our installation brings sights, smells, sounds, and a bit of mud from a peat bog undergoing restoration near Plymouth, MA to the MIT Media Lab. As part of the Living Observatory initiative, we are developing sensor networks that document ecological processes and allow people to experience the data at different spatial and temporal scales. Small, distributed sensor devices capture climate and other environmental data, while others stream audio from high in the trees and underwater. Visit at any time from dawn till dusk and again after midnight, and check the weather report on our website (tidmarsh.media.mit.edu) for highlights; if you’re lucky you might just catch an April storm.

Living Observatory is an initiative for documenting and interpreting ecological change that will allow people, individually and collectively, to better understand relationships between ecological processes, human lifestyle choices, and climate change adaptation. As part of this initiative, we are developing sensor networks that document ecological processes and allow people to experience the data at different spatial and temporal scales. Low-power sensor nodes capture climate and other data at a high spatiotemporal resolution, while others stream audio. Sensors on trees measure transpiration and other cycles, while fiber-optic cables in streams capture high resolution temperature data. At the same time, we are developing tools that allow people to explore this data, both remotely and onsite. The remote interface allows for immersive 3D exploration of the terrain, while visitors to the site will be able to access data from the network around them directly from wearable devices.

Sensor networks permeate our built and natural environments, but our means for interfacing to the resultant data streams have not evolved much beyond HCI and information visualization. Researchers have long experimented with wearable sensors and actuators on the body as assistive devices. A user’s neuroplasticity can, under certain conditions, transcend sensory substitution to enable perceptual-level cognition of “extrasensory” stimuli delivered through existing sensory channels. But there remains a huge gap between data and human sensory experience. We are exploring the space between sensor networks and human augmentation, in which distributed sensors become sensory prostheses. In contrast, user interfaces are substantially unincorporated by the body—our relationship to them never fully pre-attentive. Attention and proprioception are key, not only to moderate and direct stimuli, but also to enable users to move through the world naturally, attending to the sensory modalities relevant to their specific contexts.

Increasing numbers of networked appliances are bringing about new opportunities for control and automation. At the same time, an increase in multifunctional appliances is creating a complex and often frustrating environment for the end-user. Motivated by these opportunities and challenges, we are exploring the potential for sensor fusion to increase usability and improve user experience while retaining the user in the control loop. We have developed a novel, camera-less, multi-sensor solution for intuitive gesture-based indoor lighting control, called RElight. Using a wireless handheld device, the user simply points at a light fixture to select it and rotates his hand to continuously configure the dimming level. Pointing is a universal gesture that communicates one’s interest in or attention to an object. Advanced machine learning algorithms allow rapid training of gestures and continuous control that supplements gesture classification.

The SEAT-E provides free access to renewable energy to charge smart
phones and small electronic devices in cities. This brings cities one step
closer to fulfilling a key UN goal: sustainable energy access for
all. The seats are off-grid and entirely autonomous. Fully integrated solar
panels store energy in Li-ion batteries and can be accessed through
weatherproof USB ports. The batteries also power lighting and sensing. Each seat has an ID and forms part of the SEAT-E network. The seats gather location-based data on air quality. Cities typically measure air quality only at one or two locations. However, the levels vary significantly depending on traffic and other factors, and as a result, both policymakers and citizens are often uninformed. Public engagement with this sensor data has the potential to create a platform for real dialogue between cities and their citizens about the air we share.

Sticky Circuits is toolkit for creating electronics using circuit board stickers. Circuit stickers are created by printing traces on flexible substrates and adding conductive adhesive. These lightweight, flexible, and sticky circuit boards allow us to begin sticking interactivity onto new spaces and interfaces such as clothing, instruments, buildings, and even our bodies.

The virtual messenger system acts as a portal to subtly communicate messages and pass information between the digital, virtual, and physical worlds, using the Media Lab’s Glass Infrastructure system. Users who opt into the system are tracked throughout the Media Lab by a multimodal sensor network. If a participating user approaches any of the Lab’s Glass Infrastructure displays they are met by their virtual personal assistant (VPA), who exists in Dopplelab’s virtual representation of the current physical space. Each VPA acts as a mediator to pass on any messages or important information from the digital world to the user in the physical world. Participating users can interact with their VPA through a small subset of hand gestures, allowing the user to read any pending messages or notices, or inform their virtual avatar not to bother them until later.