Ripley: A Conversational Helping Hand

We have constructed a 7 degree-of-freedom robot, Ripley, to investigate connections between natural language semantics, perception, and action. Our goal is to enable Ripley to perform collaborative manipulation tasks mediated by natural spoken dialogue. Key issues include representation and learning of spatial language, object and temporal reference, and physical actions / verbs. Furthermore, a "Grounded Situation Model" representation has been designed for Ripley, as well as associated processes, and a cognitive architecture was implemented through numerous intercommunicating modules. 

Ripley has been designed to support interactive learning from a human teacher. Ripley's "mouth" enables manipulation of mid-sized objects. The robot is sensor-rich, including senses of touch, stereo color vision, stereo audition, gravity, and proprioception. Based on series elastic actuators, each joint includes both absolute position and force sensors. As a result, Ripley can be physically manipulated while it is active, enabling natural hands-on training of motor skills.


Video clips

Ripley imagines and remembers [high resolution (12M) | low resolution (440K)]

Ripley tracks faces [high resolution (6M) | low resolution (280K)]

Ripley grasping objects [high resolution (201M) | low resolution (23M)]

Ripley changing perspectives [.mov, 17M)]

Training HMM model to pick up [.mov, 844K)]

HMM model generating pick up [.mov, 791K)]


Related Papers:

Kai-yuh Hsiao and Deb Roy. (2005). A Habit System for an Interactive Robot. To appear in AAAI Fall Symposium 2005: From Reactive to Anticipatory Cognitive Embodied Systems.pdf (981K)

Deb Roy. (2005). Semiotic Schemas: A Framework for Grounding Language in Action and Perception. Artificial Intelligence, 167(1-2):170-205.pdf (1 MB)

Nikolaos Mavridis and Deb Roy. Grounded Situation Models for Robots: Where words and percepts meet. IROS2006. pdf (598K)

Deb Roy, Kai-Yuh Hsiao, and Nikolaos Mavridis. Mental Imagery for  a Conversational Robot. IEEE Transactions on Systems, Man, and Cybernetics. pdf (467K) 

Deb Roy, Kai-yuh Hsiao, Nikolaos Mavridis, and Peter Gorniak. Ripley, Hand Me the Cup: Sensorimotor Representations for Grounding Word Meaning. Submitted to the Int. Conf. of Automatic Speech Recognition and Understanding pdf (125K)

Deb Roy, Kai-yuh Hsiao, Nikolaos Mavridis. Conversational Robots: Building Blocks for Grounding Word Meanings. In Proceedings of the HLT-NAACL03 Workshop on Learning Word Meaning from Non-Linguistic Data. pdf (364K)

Kai-yuh Hsiao, Nikolaos Mavridis, Deb Roy. Coupling Perception and Simulation: Steps Towards Conversational Robotics. To appear in the proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems. pdf (206K)