Bishop: Understanding Complex Spatial Language

The Bishop project explores the subtleties of human language when talking about spatial scenes. In particular, we investigate the various descriptive strategies human speakers employ in talking about objects in relation to other objects. These strategies include ordering objects, visually grouping them, describing their spatial relation or even referring back to objects that used to be in the scene. Furthermore, human subjects frequently perform combinations of these strategies, for example "the green one to the left of the three purple ones." We are building a computational system that replicates both the individual phenomena and their compositional behavior. As a result, this system understands relatively complex expressions referring to a scene of objects and can indicate the object being described. This work has direct applications for understanding for natural-language user interfaces, especially in augmenting direct-manipulation interfaces with intelligent speech control. A good example is speech interfaces for GPS map devices in cars where users speak about objects on the map.