Peter Gorniak, Deb Roy

Abstract

We propose the use of perceived affordances to computationally model linguistic concepts in situated language use. Perceived affordances are mental structures that capture the relationship between the language user and the embedding situation. We employ computational models of human perceived affordances to understand situated language: language that dynamically depends on the current physical environment and the goals and plans of communication partners. To support this theory of situated language understanding, we describe an implemented system that understands verbal commands situated in a virtual gaming environment. The implementation uses probabilistic hierarchical plan recognition to generate perceived affordances. We have evaluated the system on its ability to correctly interpret free-form spontaneous verbal commands recorded from unrehearsed game play between human players, and find that it is able to “step into the shoes” of human players and correctly respond to a broad range of verbal commands in which linguistic meaning depends on social and physical context.