Research

SixthSense and some of its applications (clockwise): taking photographs, watching news video, checking the time, drawing, using a map, and recognizing gestures.

SixthSense is a wearable, gestural interface that augments our physical world with digital information, and lets us use natural hand gestures to interact with that information.

SixthSense brings intangible, digital information into the tangible world, and allows us to interact with this information via natural hand gestures. SixthSense frees information from its confines, seamlessly integrating it with reality, thus making the entire world your computer.

The SixthSense prototype comprises a pocket projector, mirror, and camera worn in a pendant-like mobile device. Both the projector and the camera are connected to a mobile computing device in the user’s pocket. The system projects information onto the surfaces and physical objects around us, making any surface into a digital interface; the camera recognizes and tracks both the user's hand gestures and physical objects using computer-vision-based techniques. SixthSense uses simple computer-vision techniques to process the video-stream data captured by the camera and follows the locations of colored markers on the user’s fingertips (which are used for visual tracking). In addition, the software interprets the data into gestures to use for interacting with the projected application interfaces.

The current SixthSense prototype supports several types of gesture-based interactions, demonstrating the usefulness, viability, and flexibility of the system. The current prototype system costs approximately $350 to build.