Sanchez Lengeling, Thomas. "Panoptic"

Abstract

 Imagine traveling to a destination and, once you are there, hearing only car horns or seeing only a green sky. This project is an augmented reality experience in which you see and hear your current city while you are in another location through an immersive helmet. The multimodal experience includes augmented sounds and images that are generated by a semantic segmentation of your visual field. The segmentation produces overlaid sonic textures and superimposed visuals, modeling a newly perceived reality. The system presents cases where it is possible to imagine another place deeply and to become accustomed to a new location before being there physically. Moreover, one of the purposes of this thesis is to study the immersion of your current location but with distinct sounds and visuals from another location, for example, seeing colors and hearing sounds from Mexico City as if you were in Boston. The panoptic system aims to enhance human cognition and perception by using artificial intelligence through an immersive augmented reality device. Another goal is to augment your vision by knowing and understanding your surroundings. By using a Deep Learning model, the user’s view is a “diminished” or an “augmented” reality, in which it is possible to manipulate and control physical reality in real time reality. In addition, the purpose is to offer a helpful, immersive, subjective — rather than a detached, observational – experience.