Jaques, N., McDuff, D., Kim, Y. K., and Picard, R. W. "Understanding and Predicting Bonding in Conversations Using Thin Slices of Facial Expressions and Body Language," In Proceedings of Intelligent Virtual Agents, California, USA, September 2016.

Abstract

This paper investigates how an intelligent agent could be designed to both predict whether it is bonding with its user, and convey appropriate facial expression and body language responses to foster bonding. Video and Kinect recordings are collected from a series of naturalistic conversations, and a reliable measure of bonding is adapted and verified. A qualitative and quantitative analysis is conducted to determine the non-verbal cues that characterize both high and low bonding conversations. We then train a deep neural network classifier using one minute segments of facial expression and body language data, and show that it is able to accurately predict bonding in novel conversations.